DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of Group I, including claims 132-148 in the reply filed on 15 July 2026 is acknowledged.
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 132-135, 137, and 139-148 are rejected under 35 U.S.C. 103 as being unpatentable over Lyons et al. (US 6970591), herein Lyons.
Regarding claim 132, Lyons discloses a computer-implemented method for generating a data set for computer vision operations, the method comprising:
detecting features in a first image frame associated with a camera having a first pose (see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations; see Lyons col. 8, ln. 55-60, where the image data of a first image frame in the sequence is processed to identify features in the image for tracking);
evaluating features of an additional image frame of a first plurality of image frames that excludes the first image frame, each additional image frame associated with a camera having a respective additional pose (see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations; see Lyons col. 8, ln. 63-67 where the tracking is performed of the identified feature points for the next frame in the input sequence);
selecting at least one second frame from the first plurality of image frames based on the evaluated features of the additional frame satisfying a first selection criteria of a threshold number of feature matches with the first image frame (see Lyons col. 15, ln. 5-30, where a current frame is selected as a key frame based on whether the number of tracked features which are in both the current frame and the first keyframe is greater than a threshold);
evaluating features of an additional image frame of a second plurality of image frames excluding the first image frame and first plurality of image frames, the at least one additional image frame of the second plurality of image frames having a new respective pose (see Lyons col. 16, ln. 50 – col. 17, ln. 5, where having set up a triple of keyframes, a next keyframe in the sequence after frame 3 of the preceding triple is considered; see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations);
selecting at least one keyframe from the second plurality of image frames based on the at least one keyframe satisfying a second selection criteria of a threshold number of trifocal features matches with the first frame and the selected at least one second frame (see Lyons col. 16, ln. 50 – col. 17, ln. 5, where the new key frame is determine whether the number of tracked features which appear in both the current keyframe and frame 3 of the preceding triple is less than a threshold, and when it is determined that the number of tracked features in the current keyframe and frame 3 of the preceding triple is less than the threshold, then the current keyframe is set as frame 1 of a new triple); and
compiling a keyframe set comprising the first image frame, the at least one second image frame, and the at least one keyframe (see Lyons col. 17, ln. 40-50, where a respective "set" of keyframes is created from each triple, a set being a group of keyframes in which the camera projection for each frame in the set is defined relative to another frame in the set).
Although Lyons does not explicitly disclose that the image frames are associated with a camera having respective poses, and that the at least one keyframe from the second plurality of image frames satisfying a second selection criteria of a threshold number of trifocal features matches with the first frame, Lyons does disclose that the image frames are recorded by a moving camera of a scene at different locations (see Lyons col. 8, ln. 10-20), thus suggesting that the corresponding image frames are associated with the corresponding camera poses at the different locations; and while Lyons does not explicitly disclose that the at least one keyframe from the second plurality of image frames satisfying a second selection criteria of a threshold number of trifocal features matches with the first frame, Lyons does disclose that the current keyframe is set as frame 1 of a new triple when the number of tracked features in the current keyframe and frame 3 of the preceding triple is less than the threshold (see Lyons col. 16, ln. 50 – col. 17, ln. 5), and that the frame determined as the frame 3 of the preceding triple has a number of tracked features which are in both the frame 1 and frame 3 of the preceding triple that is greater than a threshold (see Lyons col. 15, ln. 5-30), thus further suggesting that the frame 1 of a new triple and frame 1 of the preceding triple would have a number of tracked features less than a threshold.
Thus, one of ordinary skill in the art, in view of the suggested disclosed features of the embodiments of Li, that would have found it obvious that image frames are associated with a camera having respective poses, and that the at least one keyframe from the second plurality of image frames satisfying a second selection criteria of a threshold number of trifocal features matches with the first frame. Thus, one of ordinary skill in the art, in view of the suggested disclosed features of the embodiments of Li, would have found it obvious and led to combine the disclosed features and arrive at the claimed invention. This modification is rationalized as some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. See also MPEP 2144.01.
Regarding claim 133, Lyons discloses a computer-implemented method for generating a data set for computer vision operations, the method comprising:
detecting features in an initial image frame associated with a camera having a first pose (see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations; see Lyons col. 8, ln. 55-60, where the image data of a first image frame in the sequence is processed to identify features in the image for tracking; suggesting that the corresponding image frames are associated with the corresponding camera poses at the different locations);
evaluating features of an additional image frame having a respective additional pose (see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations; see Lyons col. 8, ln. 63-67 where the tracking is performed of the identified feature points for the next frame in the input sequence);
selecting at least one associate frame a first plurality of image frames based on the evaluation of the additional frame according to a first selection criteria (see Lyons col. 15, ln. 5-30, where a current frame is selected as a key frame based on whether the number of tracked features which are in both the current frame and the first keyframe is greater than a threshold);
evaluating a second plurality of image frames, at least one image frame of the second plurality of image frames having a new respective pose (see Lyons col. 16, ln. 50 – col. 17, ln. 5, where having set up a triple of keyframes, a next keyframe in the sequence after frame 3 of the preceding triple is considered; see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations);
selecting at least one candidate frame from the second plurality of image frames (see Lyons col. 16, ln. 50 – col. 17, ln. 5, where the new key frame is determine whether the number of tracked features which appear in both the current keyframe and frame 3 of the preceding triple is less than a threshold, and when it is determined that the number of tracked features in the current keyframe and frame 3 of the preceding triple is less than the threshold, then the current keyframe is set as frame 1 of a new triple; suggesting that the frame 1 of a new triple and frame 1 of the preceding triple would have a number of tracked features less than a threshold); and
compiling a keyframe set comprising the at least one candidate frame (see Lyons col. 17, ln. 40-50, where a respective "set" of keyframes is created from each triple, a set being a group of keyframes in which the camera projection for each frame in the set is defined relative to another frame in the set).
Please see the above rejection for claim 132, as the rationale to combine the obviousness teachings of Lyons are similar, mutatis mutandis.
Regarding claim 134, please see the above rejection of claim 133. Lyons discloses the method of claim 133, wherein the first selection criteria for evaluating features of the additional image frame comprises identifying feature matches between the initial image frame and the additional frame (see Lyons col. 8, ln. 63-67 where the tracking is performed of the identified feature points for the next frame in the input sequence; see Lyons col. 15, ln. 5-30, where a current frame is selected as a key frame based on whether the number of tracked features which are in both the current frame and the first keyframe is greater than a threshold; see Lyons col. 16, ln. 50 – col. 17, ln. 5, where the new key frame is determine whether the number of tracked features which appear in both the current keyframe and frame 3 of the preceding triple is less than a threshold).
Regarding claim 135, please see the above rejection of claim 134. Lyons discloses the method of claim 134, wherein the number of feature matches is above a first threshold (see Lyons col. 15, ln. 5-30, where a current frame is selected as a key frame based on whether the number of tracked features which are in both the current frame and the first keyframe is greater than a threshold).
Regarding claim 137, please see the above rejection of claim 134. Lyons discloses the method of claim 134, wherein the number of feature matches is below a second threshold (see Lyons col. 16, ln. 50 – col. 17, ln. 5, where the new key frame is determine whether the number of tracked features which appear in both the current keyframe and frame 3 of the preceding triple is less than a threshold).
Regarding claim 139, please see the above rejection of claim 133. Lyons discloses the method of claim 133, wherein the first selection criteria for evaluating features in the additional image frame further comprises exceeding a prescribed camera distance between the initial image frame and the additional frame (see Lyons col. 8, ln. 35-50, where a threshold value is set on the minimum number of frames which must be present between the keyframes selected; see Lyons col. 19, ln. 15-30, where the relative transformations for each keyframe lying between the key frames of the triple, defining the movement of the camera between the key frame are calculated; suggesting that a minimum of camera distance based on the minimum number of frames that must be present between the selected keyframes).
Regarding claim 140, please see the above rejection of claim 139. Lyons discloses the method of claim 139, wherein the prescribed camera distance is a translation distance (see Lyons col. 20 ln. 60-67, where camera transformations includes translation).
Regarding claim 141, please see the above rejection of claim 140. Lyons discloses the method of claim 140, wherein the translation distance is based on an imager-to-object distance (see Lyons col. 18, ln. 5-ln. 40, where the matching feature points in the triple and the calculated positions and orientations of theses frames are used to estimate the position in three-dimensions of the points on the object which the feature points represent).
Regarding claim 142, please see the above rejection of claim 133. Lyons discloses the method of claim 133, wherein selecting the least one candidate frame further comprises satisfying a matching criteria (see Lyons col. 15, ln. 5-30, where a current frame is selected as a key frame based on whether the number of tracked features which are in both the current frame and the first keyframe is greater than a threshold).
Regarding claim 143, please see the above rejection of claim 142. Lyons discloses the method of claim 142, wherein satisfying a matching criteria comprises identifying trifocal features with the initial image frame, associate frame and one other received image frame of the second plurality of image frames (see Lyons col. 15, ln. 15 – col. 16,ln. 55, where two keyframes with a number of tracked common features with the first keyframe that are above a threshold are selected as keyframes of the triple).
Regarding claim 144, please see the above rejection of claim 143. Lyons discloses the method of claim 143, wherein at least three trifocal features are identified (see Lyons col. 15, ln. 15 – col. 16,ln. 55, where two keyframes with a number of tracked common features with the first keyframe that are above a threshold are selected as keyframes of the triple).
Regarding claim 145, please see the above rejection of claim 133. Lyons discloses the method of claim 133, further comprising generating a multi-dimensional model of a subject within the keyframe set (see Lyons col. 18, ln. 5-ln. 40, where the matching feature points in the triple and the calculated positions and orientations of theses frames are used to estimate the position in three-dimensions of the points on the object which the feature points represent).
Regarding claim 146, Lyons discloses system comprising: one or more processors (see Lyons col. 6, ln. 60 – col. 7, ln. 30, where a processing apparatus, containing one or more processors, memory, graphics cards, etc., is programmed according to stored programming instructions to perform the disclosed teachings) configured to:
detect features in a first image frame associated with a camera having a first pose (see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations; see Lyons col. 8, ln. 55-60, where the image data of a first image frame in the sequence is processed to identify features in the image for tracking);
evaluate features of an additional image frame of a first plurality of image frames that excludes the first image frame, each additional image frame associated with a camera having a respective additional pose (see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations; see Lyons col. 8, ln. 63-67 where the tracking is performed of the identified feature points for the next frame in the input sequence);
select at least one second frame from the first plurality of image frames based on the evaluated features of the additional frame satisfying a first selection criteria of a threshold number of feature matches with the first image frame (see Lyons col. 15, ln. 5-30, where a current frame is selected as a key frame based on whether the number of tracked features which are in both the current frame and the first keyframe is greater than a threshold);
evaluate features of an additional image frame of a second plurality of image frames excluding the first image frame and first plurality of image frames, the at least one additional image frame of the second plurality of image frames having a new respective pose (see Lyons col. 16, ln. 50 – col. 17, ln. 5, where having set up a triple of keyframes, a next keyframe in the sequence after frame 3 of the preceding triple is considered; see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations);
select at least one keyframe from the second plurality of image frames based on the at least one keyframe satisfying a second selection criteria of a threshold number of trifocal features matches with the first frame and the selected at least one second frame (see Lyons col. 16, ln. 50 – col. 17, ln. 5, where the new key frame is determine whether the number of tracked features which appear in both the current keyframe and frame 3 of the preceding triple is less than a threshold, and when it is determined that the number of tracked features in the current keyframe and frame 3 of the preceding triple is less than the threshold, then the current keyframe is set as frame 1 of a new triple); and
compile a keyframe set comprising the first image frame, the at least one second image frame, and the at least one keyframe (see Lyons col. 17, ln. 40-50, where a respective "set" of keyframes is created from each triple, a set being a group of keyframes in which the camera projection for each frame in the set is defined relative to another frame in the set).
Please see the above rejection for claim 132, as the rationale to combine the obviousness teachings of Lyons are similar, mutatis mutandis.
Regarding claim 147, Lyons discloses a computer-implemented method for generating a data set for computer vision operations, the method comprising:
receiving a first plurality of reference image frames having respective camera poses (see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations; see Lyons col. 8, ln. 55-60, where the image data of a first image frame in the sequence is processed to identify features in the image for tracking);
evaluating a second plurality of image frames, wherein at least one image frame of the second plurality of image frames is unique relative to the reference image frames (see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations; see Lyons col. 8, ln. 63-67 where the tracking is performed of the identified feature points for the next frame in the input sequence; see Lyons col. 16, ln. 50 – col. 17, ln. 5, where having set up a triple of keyframes, a next keyframe in the sequence after frame 3 of the preceding triple is considered; see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations);
selecting at least one candidate frame from the second plurality of image frames based on feature matching with at least two image frames from the first plurality of reference frames (see Lyons col. 15, ln. 5-30, where a current frame is selected as a key frame based on whether the number of tracked features which are in both the current frame and the first keyframe is greater than a threshold; see Lyons col. 16, ln. 50 – col. 17, ln. 5, where the new key frame is determine whether the number of tracked features which appear in both the current keyframe and frame 3 of the preceding triple is less than a threshold, and when it is determined that the number of tracked features in the current keyframe and frame 3 of the preceding triple is less than the threshold, then the current keyframe is set as frame 1 of a new triple); and
compiling a keyframe set comprising the at least one candidate frame (see Lyons col. 17, ln. 40-50, where a respective "set" of keyframes is created from each triple, a set being a group of keyframes in which the camera projection for each frame in the set is defined relative to another frame in the set).
Please see the above rejection for claim 132, as the rationale to combine the obviousness teachings of Lyons are similar, mutatis mutandis.
Regarding claim 148, Lyons discloses a computer-implemented method for generating a frame reel of related input images, the method comprising:
receiving an initial image frame at a first camera position (see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations; see Lyons col. 8, ln. 55-60, where the image data of a first image frame in the sequence is processed to identify features in the image for tracking);
evaluating at least one additional image frame related to the initial image frame (see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations; see Lyons col. 8, ln. 63-67 where the tracking is performed of the identified feature points for the next frame in the input sequence);
selecting the at least one additional image frame based on a first selection criteria (see Lyons col. 15, ln. 5-30, where a current frame is selected as a key frame based on whether the number of tracked features which are in both the current frame and the first keyframe is greater than a threshold));
evaluating at least one candidate frame related to the selected additional image frame (see Lyons col. 16, ln. 50 – col. 17, ln. 5, where having set up a triple of keyframes, a next keyframe in the sequence after frame 3 of the preceding triple is considered; see Lyons col. 8, ln. 10-20, where the image frames are recorded by a moving camera of a scene at different locations);
selecting the at least one candidate frame based on a second selection criteria (see Lyons col. 16, ln. 50 – col. 17, ln. 5, where the new key frame is determine whether the number of tracked features which appear in both the current keyframe and frame 3 of the preceding triple is less than a threshold, and when it is determined that the number of tracked features in the current keyframe and frame 3 of the preceding triple is less than the threshold, then the current keyframe is set as frame 1 of a new triple);
generating a cumulative frame reel comprising at least the initial image frame, selected additional frame, and selected candidate frame (see Lyons col. 17, ln. 40-50, where a respective "set" of keyframes is created from each triple, a set being a group of keyframes in which the camera projection for each frame in the set is defined relative to another frame in the set).
Please see the above rejection for claim 132, as the rationale to combine the obviousness teachings of Lyons are similar, mutatis mutandis.
Claim 136 is rejected under 35 U.S.C. 103 as being unpatentable over Lyons as applied to claim 135 above, and further in view of Wu et al. (US 2017/0256065), herein Wu.
Regarding claim 136, please see the above rejection of claim 135. Lyons does not explicitly disclose the method of claim 135, wherein the first threshold is 100.
Wu teaches in a related and pertinent method for tracking regions of interest across
video frames (see Wu Abstract), where motion tracking of feature points between frames is performed
(see Wu Fig. 4 [0068]-[0078]), where a determination may be made as to whether the number of feature points exceeds a threshold to continue tracking the features, where the threshold may be any suitable value and is set to 100 (see Wu [0041] and [0071]).
At the time of filing, one of ordinary skill in the art would have found it obvious from the teachings Lyons and Wu, that the threshold value for selecting a current frame as a key frame with a first keyframe can be set to 100.
This modification is rationalized as some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention.
In this instance, Lyons teaches that a current frame is selected as a key frame based on whether the number of tracked features which are in both the current frame and the first keyframe is greater than a threshold.
Wu teaches that a determination may be made as to whether the number of feature points exceeds a threshold to continue tracking the features, where the threshold may be any suitable value and is set to 100.
One of ordinary skill in the art would have found it obvious to set the threshold value to 100 for selecting a current frame as a key frame with a first keyframe, and that would reasonably expect that keyframes with 100 tracked features with a first keyframe would be selected as a key frame of the triple.
Claim 138 is rejected under 35 U.S.C. 103 as being unpatentable over Lyons as applied to claim 137 above, and further in view of Cao et al. (“Robust bundle adjustment for large-scale structure from motion”), herein Cao
Regarding claim 138, please see the above rejection of claim 137. Lyons does not explicitly disclose the method of claim 137, wherein the second threshold is 10,000.
Cao teaches in a related and pertinent robust Bundle Adjustment (RBA) algorithm to optimize the initial 3D point-clouds and camera parameters (see Cao Abstract), where out of memory problem makes the optimization of 3D model challenging in limited memory and high computational cost when solving the norm equation of camera parameters, where, for example, a typical scenario includes more than 10,000 3D scene points, which yields a complex non-linear formula and then should calculate a lot of parameters (see Cao sect. 1 Introduction).
At the time of filing, one of ordinary skill in the art would have found it obvious from the teachings Lyons and Cao, that the threshold value of the number of tracked features for selecting a current frame as a key frame for a new triple be set to 10,000 and limit the out of memory and computational costs problems.
This modification is rationalized as some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention.
In this instance, Lyons teaches that for selecting a current frame as a key frame for a new triple, the number of tracked features is less than a threshold value.
Cao teaches that out of memory problem makes the optimization of 3D model challenging in limited memory and high computational cost when solving the norm equation of camera parameters, where, a typical scenario may include more than 10,000 3D scene points, which yields a complex non-linear formula and then should calculate a lot of parameters.
One of ordinary skill in the art would have found it obvious that by setting the threshold value of the number of tracked features for selecting a current frame as a key frame for a new triple to 10,000, that would reasonably expect that tracked features for selecting keyframes in new triples would be limited to less than 10,000 tracked features, and limit the out of memory and computational costs problems.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIMOTHY WING HO CHOI whose telephone number is (571)270-3814. The examiner can normally be reached 9:00 AM to 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VINCENT RUDOLPH can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIMOTHY CHOI/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights