DETAILED ACTION
Response to Arguments
The Applicant's arguments, in pages 1-3 filed 10/03/2025, with respect to the 35 U.S.C. 102/103 rejection(s) of claim(s) 1-25, have been fully considered but they are not persuasive.
The Applicant argues, from “The Examiner rejects…” to “Accordingly, Applicant respectfully requests…” in pages 1-2 filed 10/03/2025, that “Endres fails to discloses ‘wherein the one or more matching keypoints are associated with one or more depth models of the environment,’ as recited by Claim 1” because allegedly “The depth images in Endres are not depth models of an environment, as they do not model or estimate the depth of the environment—rather, they contain actual depth measurements measured by a depth sensor in an RBG-D camera.” The Examiner respectfully disagrees. The Applicant is reminded that the claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art (see MPEP 2111.01). Under Broadest Reasonable Interpretation and in accordance with the originally filed Specification (dated 03/31/2023) describing depth models at a non-limiting, high level (e.g. paragraphs [0034-0039]), the depth information of Endres is indeed a depth model, because, by virtue of, as the Applicant admits, “contain[ing] actual depth measurements measured by a depth sensor,” they indeed estimate, describe, and thereby model the depth of the environment. Depth information, including depth measurements, depth maps, and depth images, act as depth model(s) of an environment in that they represent the environment and “estimate the depth of the environment.” As such, the depth information of Endres indeed corresponds to the depth model(s) of the claims(s). Accordingly, Endres indeed discloses the limitation “wherein the one or more matching keypoints are associated with one or more depth models of the environment,” as in Endres the keypoints/features ae detected, matched across frames, and are associated with the depth information/model of the environment at the same location, with the depth information indeed corresponding to depth model(s) (see Endres, Fig. 2, pgs. 1-2, 4). Therefore, the rejection(s) is/are maintained.
As such, this action is made FINAL.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 9-10, 18-21 is/are rejected under 35 U.S.C. 102(a)(1) and 102(a)(2) as being anticipated by F. Endres, J. Hess, N. Engelhard, J. Sturm, D. Cremers and W. Burgard, "An evaluation of the RGB-D SLAM system," 2012 IEEE International Conference on Robotics and Automation, Saint Paul, MN, USA, 2012, pp. 1691-1696, doi: 10.1109/ICRA.2012.6225199., hereinafter referred to as Endres.
Regarding claim 1, Endres teaches at least one non-transitory machineEndres, storage embodied in computer and GPU, pgs. 6-7): receive, via interface circuitry, a plurality of frames of image data (Endres, Figs. 1, 2, receive plurality of frames of image data, pgs. 1-2), wherein the frames are captured by one or more sensors from a plurality of poses within an environment, and wherein the frames include a current frame and one or more preceding frames (Endres, Figs. 1, 2, frames are captured by a camera from a plurality of poses within an environment (see Fig. 1(a)), where the frames include a current frame and one or more preceding frames (“previous images”), pgs. 1-2); detect one or more keypoints in the current frame (Endres, Fig. 2, detect features/keypoints of the current frame, pgs. 1-2); find one or more matching keypoints in the one or more preceding frames, wherein the one or more matching keypoints match the one or more keypoints in the current frame (Endres, Fig. 2, “match these [current] features against features from previous images,” pgs. 1-2), and wherein the one or more matching keypoints are associated with one or more depth models of the environment (Endres, Fig. 2, features/keypoints are associated with depth information [the depth information corresponding to a depth model, because depth information indeed describes, and in that models, the environment] of the environment at the same location, pgs. 1-2, 4); and determine, based at least in part on the one or more depth models, a pose of the current frame within the environment (Endres, Figs. 1, 2, determine, based in part on the depth models/information, a pose of the current frame within the environment, pgs. 1-3).
Regarding claim 9, Endres teaches the storage medium of Claim 1, wherein the image data comprises color data and depth data (Endres, Fig. 2, image data comprises color data (RGB) and depth data, pgs. 1-2).
Regarding claims 10, 18, 21, the rationale provided in the rejection of claims 1, 9 is incorporated herein. In addition, the device of claims 10, 18 (Endres, interface circuity and processing circuitry embodied in computer and GPU, pgs. 6-7) and the method of claim 21 corresponds to the storage medium of claims 1, 9, and performs the steps disclosed herein.
Regarding claim 19, Endres teaches the device of Claim 18, wherein the one or more sensors comprise: a camera to capture the color data; and a depth sensor to capture the depth data (Endres, Microsoft Kinect camera is RBG-D, which includes a camera to capture the color data and a depth sensor to capture the depth data, pgs. 1-2).
Regarding claim 20, Endres teaches the device of Claim 10, wherein the device is implemented in a robot, a drone, or a vehicle (Endres, device is implemented in a robot for robot localization, pgs. 1-3, 8).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2, 4, 11, 13, 22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Endres as applied to claims 1, 10, 21 above, in view of Kajal Sharma. 2018. Improved visual SLAM: a novel approach to mapping and localization using visual landmarks in consecutive frames. Multimedia Tools Appl. 77, 7 (April 2018), 7955–7976. https://doi.org/10.1007/s11042-017-4694-x., hereinafter referred to as Sharma.
Regarding claim 2, Endres teaches the storage medium of Claim 1.
However, Endres fails to teach where Sharma teaches wherein: the one or more matching keypoints correspond to one or more landmarks in the environment (Sharma, Fig. 3, keypoints, matched between frames, correspond to “landmarks,” also matched, in the environment, with the landmarks being “obtained using feature [keypoint] matching,” pgs. 3, 7); the one or more landmarks are in one or more regions of the environment (Sharma, Figs. 3, 5, the landmarks are in regions of the environment, pg. 11); and the one or more regions are modeled by the one or more depth models (Sharma, Figs. 3, 5, the regions are modelled to be a 3D map using depth information/map/model, pgs. 9-11).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Endres using the teachings of Sharma to include Sharma’s landmarks, in regions of the environment and modelled by depth models/information, corresponding to matched keypoints, to Endres’ matched keypoints in regions of the environment modelled by depth models/information. Doing so would improve localization across frames by providing landmarks in consecutive frames, which would be used to efficiently determine the camera’s pose.
Regarding claim 4, the combination of Endres and Sharma teaches the storage medium of Claim 2, wherein the instructions that cause the processing circuitry to find the one or more matching keypoints in the one or more preceding frames further cause the processing circuitry to: identify the one or more landmarks corresponding to the one or more matching keypoints (Sharma, Figs. 3, 5, identify landmarks that correspond to the matching keypoints/features, pgs. 3, 7, 11); and associate the one or more keypoints in the current frame with the one or more landmarks (Sharma, Figs. 3, 5, keypoints/features that correspond to the landmark are associated with the landmark and its location for each frame, including the current frame, pgs. 7-11).
Regarding claims 11, 13, 22, the rationale provided in the rejection of claims 2, 4 is incorporated herein. In addition, the device of claims 11, 13 (Endres, interface circuity and processing circuitry embodied in computer and GPU, pgs. 6-7) and the method of claim 22 corresponds to the storage medium of claims 2, 4, and performs the steps disclosed herein.
Claim(s) 3, 12, 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Endres in view of Sharma as applied to claims 2, 11, 22 above, further in view of Jan Wietrzykowski, Piotr Skrzypczyński, PlaneLoc: Probabilistic global localization in 3-D using local planar features, Robotics and Autonomous Systems, Volume 113, 2019, Pages 160-173, ISSN 0921-8890, https://doi.org/10.1016/j.robot.2019.01.008., hereinafter referred to as Wietrzykowski.
Regarding claim 3, the combination of Endres and Sharma teaches the storage medium of Claim 2.
However, the combination of Endres and Sharma fails to teach where Wietrzykowski teaches wherein the instructions further cause the processing circuitry to: detect one or more surfaces in the one or more preceding frames (Wietrzykowski, Fig. 2, detect one or more planar segments [planar segments are a type of surface] in a preceding frame (the preceding, different viewpoint), pg. 9); identify the one or more regions of the environment corresponding to the one or more surfaces (Wietrzykowski, Fig. 2, identify that the surface is in a particular region of the environment (e.g. in the proper regions in Fig. 2(b) top and middle examples), pg. 9); and generate the one or more depth models of the one or more regions (Wietrzykowski, Figs. 5, 9, Eqn. 1, the regions containing the surfaces/planar segments are modelled using depth information (Eqn. 1) and depth images (Fig. 5) to produce a 3D global map, pgs. 11-13, 28).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Endres, as modified by Sharma, using the teachings of Wietrzykowski to include Wietrzykowski’s detection of planar surfaces/segments across frames in a region of the environment for depth modelling to Endres’, as modified by Sharma, detection of keypoints across frames in a region of the environment for depth modelling. Doing so would improve localization across frames by providing surfaces/segments in consecutive frames, which would be used to efficiently determine the camera’s pose.
Regarding claims 12, 23, the rationale provided in the rejection of claim 3 is incorporated herein. In addition, the device of claim 12 (Endres, interface circuity and processing circuitry embodied in computer and GPU, pgs. 6-7) and the method of claim 23 corresponds to the storage medium of claim 3 and performs the steps disclosed herein.
Claim(s) 5, 14, 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Endres in view of Sharma as applied to claims 2, 11, 22 above, further in view of D. Li et al., "A SLAM System Based on RGBD Image and Point-Line Feature," in IEEE Access, vol. 9, pp. 9012-9025, 2021, doi: 10.1109/ACCESS.2021.3049467., hereinafter referred to as Li.
Regarding claim 5, the combination of Endres and Sharma teaches the storage medium of Claim 2.
However, the combination of Endres and Sharma fails to teach where Li teaches wherein the instructions that cause the processing circuitry to determine, based at least in part on the one or more depth models, the pose of the current frame within the environment further cause the processing circuitry to: compute the pose of the current frame based at least in part on: a pose of a keyframe, wherein the keyframe is one of the preceding frames (Li, pose of current frame is based on pose of preceding keyframe; “use the map points and lines of the adjacent reference keyframe to track the current frame pose to check if there are enough correspondences to support that pose is correct,” the adjacent frame being a “previous frame,” pg. 7); the one or more landmarks (Li, Fig. 1, “landmarks (points, lines)” (pg. 10) are used to determine pose of current frame, pgs. 4-7, 10-11); and the one or more depth models (Li, depth information/modelling is used to determine pose of current frame, pgs. 4-5).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Endres, as modified by Sharma, using the teachings of Li to include Li’s computing of the pose of the current frame using keyframe, landmark, and depth information to Endres’, as modified by Sharma, computing of the pose of the current frame using landmark and depth information. Doing so would improve pose computation by providing keyframe information, which would be used to efficiently determine the camera’s pose.
Regarding claims 14, 24, the rationale provided in the rejection of claim 5 is incorporated herein. In addition, the device of claim 14 (Endres, interface circuity and processing circuitry embodied in computer and GPU, pgs. 6-7) and the method of claim 24 corresponds to the storage medium of claim 5 and performs the steps disclosed herein.
Claim(s) 6, 15, 25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Endres in view of Sharma as applied to claims 2, 11, 22 above, further in view of Rafael Munoz-Salinas, Manuel J. Marín-Jimenez, R. Medina-Carnicer, SPM-SLAM: Simultaneous localization and mapping with squared planar markers, Pattern Recognition, Volume 86, 2019, Pages 156-171, ISSN 0031-3203, https://doi.org/10.1016/j.patcog.2018.09.003., hereinafter referred to as Munoz-Salinas, and further in view of Z. Yu and H. Min, "Visual SLAM Algorithm Based on ORB Features and Line Features," 2019 Chinese Automation Congress (CAC), Hangzhou, China, 2019, pp. 3003-3008, doi: 10.1109/CAC48633.2019.8996373., hereinafter referred to as Yu.
Regarding claim 6, the combination of Endres and Sharma teaches the storage medium of Claim 2.
However, the combination of Endres and Sharma fails to teach where Munoz-Salinas teaches wherein the instructions that cause the processing circuitry to determine, based at least in part on the one or more depth models, the pose of the current frame within the environment further cause the processing circuitry to: adjust at least some of the following values to minimize a projection error: poses of keyframes, wherein the keyframes include the current frame and at least one of the preceding frames (Munoz-Salinas, “adjust both the keyframe and marker poses by minimizing the reprojection error…only those keyframes of the map that observe the markers in the current frame will be affected by this process,” meaning the preceding and current keyframe poses are adjusted to minimize re/projection error, pg. 4); parameters of the one or more depth models; and coordinates of the one or more landmarks.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Endres, as modified by Sharma, using the teachings of Munoz-Salinas to include Munoz-Salinas’ minimization of projection error by adjusting keyframe poses for localization to Endres, as modified by Sharma’s keyframe poses for localization. Doing so would improve localization by providing a way to minimize projection error, which would be used to efficiently and accurately determine the camera’s pose with minimum error.
However, the combination of Endres, Sharma, and Munoz-Salinas fails to teach where Yu teaches to adjust at least some of the following values to minimize a projection error: coordinates of the one or more landmarks (Yu, “the points and lines [corresponding to landmarks] are transformed using coordinates to minimize the projection error,” pg. 3).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Endres, as modified by Sharma and Munoz-Salinas, using the teachings of Yu to include Yu’s minimization of projection error by adjusting landmark coordinates to Endres, as modified by Sharma and Munoz-Salinas, minimization of projection error and landmarks. Doing so would improve localization by providing another way to minimize projection error, which would be used to more efficiently and accurately determine the camera’s pose with minimum error.
Regarding claims 15, 25, the rationale provided in the rejection of claim 6 is incorporated herein. In addition, the device of claim 15 (Endres, interface circuity and processing circuitry embodied in computer and GPU, pgs. 6-7) and the method of claim 25 corresponds to the storage medium of claim 6 and performs the steps disclosed herein.
Claim(s) 7-8, 16-17, is/are rejected under 35 U.S.C. 103 as being unpatentable over Endres as applied to claims 1, 10 above, in view of Aswin (US 20190122378 A1).
Regarding claim 7, Endres teaches the storage medium of Claim 1.
However, Endres fails to teach where Aswin teaches wherein the one or more depth models comprise one or more polynomials, wherein the one or more polynomials model a depth of one or more regions of the environment (Aswin, depth models for matched points for, among other purposes, determining pose of camera [0019], comprises a system of polynomials, [0062-0065; 0047, 0056]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Endres using the teachings of Aswin to include Aswin’s polynomial system for depth modelling of matched points to Endres’ depth modelling/information of matched points. Doing so would improve pose computation by providing polynomial computation, which would be used to efficiently determine the camera’s pose.
Regarding claim 8, the combination of Endres and Aswin teaches the storage medium of Claim 7, wherein the one or more polynomials comprise at least one of a zero-order polynomial, a first-order polynomial, or a second-order polynomial (Aswin, system of polynomials comprises second-order polynomials, [0045-0047, 0056, 0065]).
Regarding claims 16-17, the rationale provided in the rejection of claims 7-8 is incorporated herein. In addition, the device of claims 16-17 (Endres, interface circuity and processing circuitry embodied in computer and GPU, pgs. 6-7) corresponds to the storage medium of claims 7-8 and performs the steps disclosed herein.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEELY G YEARGIN whose telephone number is (571)272-5126. The examiner can normally be reached M-Th 8am-6pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEELY GWYNNE YEARGIN/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights