Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/10/2025 has been entered.
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged. Applicant has not complied with one or more conditions for receiving the benefit of an earlier filing date under 35 U.S.C. §120 as follows:
Provisional application 62/258,316 having a priority date of 20 November 2015 does not have support for the limitations of Claims 18 and 20, categorizing image frame pairs as “uncategorized.” The application is examined with a priority date of 18 November 2016.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5, 7, 8, 10, 13-15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The phrase “global geometry” has no definition in the specification.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 9, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Klusza (US PG Publication 2014/0104387) in view of Rudow (US PG Publication 2014/0253375 A1).
Regarding Claim 1, Klusza (US PG Publication 2014/0104387) discloses a method for refining poses (pose-graph optimization [0053]), the method comprising:
capturing a plurality of image frames using a depth camera (a new RGB-D frame is available for processing, Step 401, Fig. 4A, [0052]-[0055]);
computing a relative pose set (N pose updates, Step 410, Fig. 4A) by:
determining a first set of relative poses (pose updates, Step 409, Fig. 4A; relative pos is computed, step 431, Fig. 4B) between image frame pairs (pair of current frame and one of the keyframes, Step 409, Fig. 4A) for a first subset of the image frame pairs (some of the N keyframes, Step 409, Fig. 4A) …;
and determining a second set of relative poses (pose updates, Step 409, Fig. 4A) between image frame pairs (pair of current frame and one of the keyframes, Step 409, Fig. 4A) for a second subset of the image frame pairs (the others of the N keyframes, Step 409, Fig. 4A) …, wherein the first subset of image frame pairs and the second subset of image frame pairs together comprise every image frame of the plurality of image frames (the current frame and every keyframe is all of the frames—note, this is performed on every incoming frame, Step 409, Fig. 4A);
Klusza does not disclose but Rudow (US PG Publication 2014/0253375 A1) teaches image frame pairs having a temporal separation between image frames of the image frame pairs less than a threshold (if the frame rate is high relative to the camera dynamics [0456]);
image frame pairs having a temporal separation between image frames of the image frame pairs greater than the threshold (i.e., not if the frame rate is high relative to the camera dynamics [0456]);
performing a first relative pose optimization process for the first set of relative poses (the estimate can simply be set to the same values as the previous frame [0456]; note, the “identity matrix” is multiplying the transformation parameters by “1,” which results in the original transformation parameters);
and performing a second relative pose optimization process (initial estimate of the camera position and orientation at the new frame may be computed from the estimated key-points, [0456]) different from the first relative pose optimization process for the second set of relative poses (using the previous transformation parameters is different than estimating based on key-points).
One of ordinary skill in the art before the application was filed would have been motivated to use the initialization of Rudow for temporally close frames in the pose determination of Klusza because assuming no transformation for temporally close frames reduces execution time and avoids costly computations and results in an accurate estimate of the final pose, improving efficiency.
Regarding Claim 2, Klusza (US PG Publication 2014/0104387) discloses the method of claim 1.
Klusza does not disclose but Rudow (US PG Publication 2014/0253375 A1) teaches wherein determining the second set (i.e., if the frame rate is not high [0456]) of relative poses (refining a set of initial camera and structure parameter estimates [0398]) comprises detecting and matching features between the image frame pairs in the second subset (key-points are the features that are common in multiple frames [0353]; finding matching features [0615]; initial estimate of the camera position and orientation at the new frame may be computed from the estimated key-points [0456]).
One of ordinary skill in the art before the application was filed would have been motivated to use the initialization of Rudow for temporally close frames in the pose determination of Klusza because assuming no transformation for temporally close frames reduces execution time and avoids costly computations and results in an accurate estimate of the final pose, improving efficiency.
Regarding Claim 3, Klusza (US PG Publication 2014/0104387) discloses the method of claim 1.
Klusza does not disclose but Rudow (US PG Publication 2014/0253375 A1) teaches further comprising performing feature pair filtering (From the full set of matches, subsets of key-points that agree on the object and its location, scale, and orientation in the new image are identified to filter out good matches [0380]).
One of ordinary skill in the art before the application was filed would have been motivated to use the initialization of Rudow for temporally close frames in the pose determination of Klusza because assuming no transformation for temporally close frames reduces execution time and avoids costly computations and results in an accurate estimate of the final pose, improving efficiency.
Regarding Claim 4, Klusza (US PG Publication 2014/0104387) discloses the method of claim 1, and depth matching (depth data is reprojected 471 into teach of the N selected keyframes, using the current estimated pose [0054]; visual predictor [0052]).
Klusza does not disclose but Rudow (US PG Publication 2014/0253375 A1) teaches wherein determining the second set of relative poses comprises performing [] matching (key-points are the features that are common in multiple frames [0353]; finding matching features [0615]) between the image frame pairs in the second subset (i.e., if the frame rate is not high [0456]).
One of ordinary skill in the art before the application was filed would have been motivated to use the initialization of Rudow for temporally close frames in the pose determination of Klusza because assuming no transformation for temporally close frames reduces execution time and avoids costly computations and results in an accurate estimate of the final pose, improving efficiency.
Regarding Claim 9, Klusza (US PG Publication 2014/0104387) discloses the method of claim 1, further comprising determining a plurality of global poses (pose graph, step 439, Fig. 4B) for the plurality of image frames (stream of RGB-D video frames [0030]) using a portion of the relative pose set (N selected key frames, Step 407, Fig. 4A).
Regarding Claim 17, Klusza (US PG Publication 2014/0104387) discloses the method of claim 1.
Klusza does not disclose but Rudow (US PG Publication 2014/0253375 A1) teaches wherein performing the first relative pose optimization process comprises initialization using an identity matrix (if the frame rate is high relative to the camera dynamics the estimate can simply be set to the same values as the previous frame [0456]).
One of ordinary skill in the art before the application was filed would have been motivated to use the initialization of Rudow for temporally close frames in the pose determination of Klusza because assuming no transformation for temporally close frames reduces execution time and avoids costly computations and results in an accurate estimate of the final pose, improving efficiency.
Claim(s) 5-8, 10-16, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Klusza (US PG Publication 2014/0104387) in view of Rudow (US PG Publication 2014/0253375 A1) and Nakaguro (Volumetric 3D Reconstruction and Parametric Shape Modeling from RGB-D Sequences, Springer, Jan. 1, 2015).
Regarding Claim 5, Klusza (US PG Publication 2014/0104387) discloses the method of claim 1, further comprising:
performing [] reconstruction (create a 3D model of the recorded scene… reconstruct the scene [0036]) for a portion of the relative pose set (a new keyframe is included in the model [0054], Fig. 4C) to produce a global geometry (geo-referenced model of the current scene [0038]);
and refining each relative pose (Alignment is checked 463; If there is a failure of alignment, the keyframes should be relocalized 465 and the procedure re-started [0054], Fig. 4C) of the portion of the relative pose set (a new keyframe is included in the model [0054], Fig. 4C) with respect to the global geometry (the model [0054]) to produce a plurality of refined poses (alignment is correct and all is well [0054]).
Klusza does not disclose but Nakaguro (Volumetric 3D Reconstruction and Parametric Shape Modeling from RGB-D Sequences, Springer, Jan. 1, 2015) teaches volumetric reconstruction (acquire a volumetric model, Section 2, p. 502, Section 2.2, p. 504).
One of ordinary skill in the art before the application was filed would have been motivated to build the 3D model of Klusza using volumetric methods because Nakaguro teaches that it has a clear advantage in shape modeling and enables better estimates of the size, shape, and volume of objects being modeled (Abstract), improving the reconstruction.
Regarding Claim 6, Klusza (US PG Publication 2014/0104387) discloses the method of claim 5.
Klusza does not disclose but Nakaguro (Volumetric 3D Reconstruction and Parametric Shape Modeling from RGB-D Sequences, Springer, Jan. 1, 2015) teaches wherein performing volumetric reconstruction comprises:
performing volumetric fusion (the FastFusion algorithm [17] for volumetric model reconstruction, p. 502);
and applying marching cubes to extract a polygonal mesh (Mesh Generation with Marching Cubes, p. 506).
One of ordinary skill in the art before the application was filed would have been motivated to build the 3D model of Klusza using volumetric methods because Nakaguro teaches that it has a clear advantage in shape modeling and enables better estimates of the size, shape, and volume of objects being modeled (Abstract), improving the reconstruction.
Regarding Claim 7, Klusza (US PG Publication 2014/0104387) discloses the method of claim 5, wherein refining each relative pose (keyframes should be relocalized 465 [0054], Fig. 4C) of the portion of the relative pose set (keyframes [0054], Fig. 4C) with respect to the global geometry (included in the model. Alignment is checked 463 [0054], Fig. 4C) comprises refining each relative pose for a predetermined number of iterations (the procedure re-started [0054], Fig. 4C).
Regarding Claim 8, Klusza (US PG Publication 2014/0104387) discloses the method of claim 5, wherein refining each relative pose (keyframes should be relocalized 465 [0054], Fig. 4C) of the portion of the relative pose set (keyframes [0054], Fig. 4C) with respect to the global geometry (keyframe included in the model. Alignment is checked 463 [0054], Fig. 4C) comprises refining each relative pose for a number of iterations (the procedure re-started [0054], Fig. 4C) … between each relative pose and a corresponding refined pose of the plurality of refined poses (recomputing the relative pose based on the previously updated poses, step 437, Fig. 4B).
Klusza does not disclose but Nakaguro (Volumetric 3D Reconstruction and Parametric Shape Modeling from RGB-D Sequences, Springer, Jan. 1, 2015) teaches iterations (Nonlinear least squares estimation is iterative) based on a threshold for a difference (selects the motion vector that minimizes the combined photometric and geometric residue, p. 504).
One of ordinary skill in the art before the application was filed would have been motivated to build the 3D model of Klusza using volumetric methods because Nakaguro teaches that it has a clear advantage in shape modeling and enables better estimates of the size, shape, and volume of objects being modeled (Abstract), improving the reconstruction.
Regarding Claim 10, Klusza (US PG Publication 2014/0104387) discloses a method for refining poses (pose-graph optimization [0053]), the method comprising:
capturing a plurality of depth image frames using a depth camera (a new RGB-D frame is available for processing, Step 401, Fig. 4A, [0052]-[0055]);
computing a plurality of poses (the current camera pose is predicted, step 406, Fig. 4), each of the plurality of poses being associated with one of the plurality of depth image frames (for each new RGB-D frame, Step 401, Fig. 4);
computing a first portion of a relative pose set (pose updates, Step 409, Fig. 4A; relative pose is computed, step 431, Fig. 4B on a pair of current frame and some of the keyframes, Step 409, Fig. 4A) by performing a first pose optimization process (aligning the keyframes in the selected set, step 409, Fig. 4) for a first set of relative poses (pair of current frame and some of the keyframes, Step 409, Fig. 4A) between depth image frame pairs (pair of current frame and some of the keyframes, Step 409, Fig. 4A) having a temporal separation (current frame and a subset of the keyframes inherently have a temporal separation) …;
and computing a second portion of the relative pose set (pose updates, Step 409, Fig. 4A; relative pose is computed, step 431, Fig. 4B on a pair of current frame and some other keyframes, Step 409, Fig. 4A) by performing a second pose optimization process (aligning the keyframes in the selected set, step 409, Fig. 4) for a first set of relative poses (pair of current frame and some of the keyframes, Step 409, Fig. 4A) for a second set of relative poses (pair of current frame and some other keyframes, Step 409, Fig. 4A) between the depth image frame pairs (RGB-D frames, step 401, Fig. 4) having a temporal separation (current frame and a subset of the keyframes inherently have a temporal separation) …,
and wherein the depth image frame pairs having temporal separation less than the threshold and the depth image frame pairs having temporal separation greater than the threshold together comprise every depth image frame of the plurality of depth image frames (the current frame is compared to each keyframe in the set, step 409, Fig. 4A);
performing volumetric reconstruction (create a 3D model of the recorded scene… reconstruct the scene [0036]) for a portion of the relative pose set (a new keyframe is included in the model [0054], Fig. 4C) to produce a global geometry (geo-referenced model of the current scene [0038]); and
refining each pose of the plurality of poses (Alignment is checked 463; If there is a failure of alignment, the keyframes should be relocalized 465 and the procedure re-started [0054], Fig. 4C) with respect to the global geometry (the model [0054]) to produce a plurality of refined poses (alignment is correct and all is well [0054]).
Klusza does not disclose but Rudow (US PG Publication 2014/0253375 A1) teaches image frame pairs having a temporal separation between depth image frames of the depth image frame pairs less than a threshold;
image frame pairs having a temporal separation between the depth image frames of the depth image frame pairs greater than the threshold,
wherein the second pose optimization process comprises a different initialization than the first pose optimization process.
Klusza does not disclose but Nakaguro (Volumetric 3D Reconstruction and Parametric Shape Modeling from RGB-D Sequences, Springer, Jan. 1, 2015) teaches volumetric reconstruction (acquire a volumetric model, Section 2, p. 502, Section 2.2, p. 504).
One of ordinary skill in the art before the application was filed would have been motivated to use the initialization of Rudow for temporally close frames in the pose determination of Klusza because assuming no transformation for temporally close frames reduces execution time and avoids costly computations and results in an accurate estimate of the final pose, improving efficiency.
One of ordinary skill in the art before the application was filed would have been motivated to build the 3D model of Klusza using volumetric methods because Nakaguro teaches that it has a clear advantage in shape modeling and enables better estimates of the size, shape, and volume of objects being modeled (Abstract), improving the reconstruction.
Regarding Claim 11, the claim is rejected on the grounds provided in Claim 6.
Regarding Claim 12, the claim is rejected on the grounds provided in Claim 6.
Regarding Claim 13, the claim is rejected on the grounds provided in Claim 7.
Regarding Claim 14, the claim is rejected on the grounds provided in Claim 8.
Regarding Claim 15, the claim is rejected on the grounds provided in Claim 5.
Regarding Claim 16, Klusza (US PG Publication 2014/0104387) discloses the method of claim 10.
Klusza does not disclose but Rudow (US PG Publication 2014/0253375 A1) teaches further comprising adjusting each refined pose of the plurality of refined poses (the minimization of the re projection error [0349] from point correspondences between two images [0393]) toward a convergence for each refined pose (converge quickly [0394]).
Regarding Claim 19, the claim is rejected on the grounds provided in Claim 17.
Claim(s) 18 is rejected under 35 U.S.C. 103 as being unpatentable over Klusza (US PG Publication 2014/0104387) in view of Rudow (US PG Publication 2014/0253375 A1) and Narikawa (US 2017/0176997 A1).
Regarding Claim 18, Klusza (US PG Publication 2014/0104387) discloses the method of claim 1.
Klusza does not disclose but Narikawa (US 20170176997 A1) suggests wherein performing the second relative pose optimization process comprises defining one or more of the second subset of the image frame pairs as uncategorized image frame pairs (When the number of corresponding feature points is less than five, the posture estimation between the two images is not executable, and the estimation determiner 14 determines (step S210) whether or not the number of corresponding feature points is less than five [0068]).
One of ordinary skill in the art before the application was filed would have been motivated to supplement the pose tracking of Klusza by leaving out some image frame pairs because Narikawa teaches that when image pairs have insufficient point correspondence, the resulting calculated pose is indefinite, resulting in inaccurate pose estimation; excluding such pairs from pose tracking results in more accurate pose tracking.
Claim(s) 20 is rejected under 35 U.S.C. 103 as being unpatentable over Klusza (US PG Publication 2014/0104387) in view of Rudow (US PG Publication 2014/0253375 A1), Nakaguro (Volumetric 3D Reconstruction and Parametric Shape Modeling from RGB-D Sequences, Springer, Jan. 1, 2015), and Narikawa (US 2017/0176997 A1).
Regarding Claim 20, the claim is rejected on the grounds provided in Claim 18.
Response to Arguments
Applicant’s remarks filed 12/10/2025 have been considered but are moot because they do not apply to the combination of references relied upon in this office action. There is, however, a distinction between Applicant’s interpretation of the claim language and Applicant’s claim. Applicant presents that the claim language “the first subset of image frame pairs and the second subset of image frame pairs together comprise every image frame of the plurality of image frames” should be interpreted as containing all possible image frame pairs. See Remarks at 10. But that isn’t what the claim says. The claim says that the set encompasses all of the image frames.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20120194516 A1 – iterative closest point and volumetric reconstruction
US 20120007943 A1 – 3D modeling via stereo-depth images
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHADAN E HAGHANI whose telephone number is (571)270-5631. The examiner can normally be reached M-F 9AM - 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jay Patel can be reached at 571-272-2988. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHADAN E HAGHANI/Examiner, Art Unit 2485