DETAILED ACTION
Contents
Notice of Pre-AIA or AIA Status 2
Response to Amendment 2
Response to Arguments 2
Claim Rejections - 35 USC § 102 3
Claim Rejections - 35 USC § 103 5
Allowable Subject Matter 12
Conclusion 12
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This action is responsive to applicant’s amendment and remarks received on 1/28/26. Claims 1-21 are currently pending.
Response to Arguments
Applicant's arguments filed 1/28/26 have been fully considered but they are not persuasive. Applicant asserts that Shan does not teach “determining keyframe residue comprising at least depth residue based on the processed measurements” because Shan has depth measurements rather than residual error quantities (see pg. 8-9). This argument is not persuasive since Shan incorporates depth readings into the VIO phrase and the VIO attaches depot the feature observations, verifies depth and calculates the camera visual reprojection error using the verified depth features. Furthermore, the term depth residue does not recite the interpretation as expressed by the applicant wherein the residual error quantities are calculated. Rather, Shan’s camera visual residual, reprojection error calculation can be interpreted to meet the claimed limitation. Thus, the reprojection residual can be reasonably meet the claimed residue based on processed depth measurements.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless - (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2-3, 7 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Shan et al (Sensor: “RGBD-Inertial Trajectory Estimation and Mapping for Ground Robots”). Regarding claim 1, Xu discloses a method for performing visual inertial odometry (VIO) at a user equipment (UE) (see abstract, section 1), the method comprising: processing measurements from an inertial measurement unit (IMU), a camera, and a depth sensor of the UE (see abstract, section 1, section 3.1); determining keyframe residue comprising at least depth residue based on the processed measurements (see section 3.1, 4.3); generating and optimizing a sliding window graph based on factors derived from the keyframe residue (see section 3.3, 3.3.2); and estimating an object pose of the UE based on the optimized sliding window graph (see abstract, section 3.3.2, 4, experimental results).
Regarding claims 2, 3, Shan teaches receiving a data stream from the IMU and performing pre-integration for the data stream to generate an IMU pre-integrate term (see 3.2); capturing a frame through the camera, and performing feature detection and tracking for the captured frame to generate a two-dimensional (2D) feature track (see 3.1); and generating a depth measurement for a detected feature of the captured frame through the depth sensor (see 3.3);
determining an IMU residue using the IMU pre-integrate term; determining a 2D feature residue from the 2D feature track; and determining the depth residue from the depth measurement (see section 3.2-3.3, 3.1, 4.1, 4.3).
Regarding claim 7, Shan discloses performing keyframe initialization using the IMU pre-integrate term and the 2D feature track to generate an initial keyframe, wherein the sliding window graph is generated based on the initial keyframe (see section 3.1-3.3).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimedinvention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 11, 12, 13, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Shan et al (Sensor: “RGBD-Inertial Trajectory Estimation and Mapping for Ground Robots”) in view of Liu et al (US 10,390,003 B1).
Regarding claim 11, Shan teaches a method comprising: process measurements from the IMU, the camera, and the depth sensor (see abstract, section 1, section 3.1); determine keyframe residue comprising at least depth residue based on the processed measurements (see section 3.1, 4.3); generate and optimize a sliding window graph based on factors derived from the keyframe residue (see section 3.3, 3.3.2); and estimate an object pose of the UE based on the optimized sliding window graph (see abstract, section 3.3.2, 4, experimental results). Shan does not teach expressly embedding user equipment (UE); an inertial measurement unit (IMU); a camera; a depth sensor; a processor; and a non-transitory computer readable storage medium storing instructions that, when executed, cause the processor to.
Liu, in the same field of endeavor, teaches user equipment (UE) (see col. 5, lines 30-40; apparatus); an inertial measurement unit (IMU) (see col. 6, lines 38-60; IMU); a camera (see col. 4, lines 8-25; camera); a depth sensor (see col. 67, lines 15-25; depth sensor); a processor(see col. 5, lines 55-67; processor); and a non-transitory computer readable storage medium storing instructions that, when executed, cause the processor to (see col. 28, lines 55-67; non-transitory computer readable storage medium).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Shan to utilize the cited limitations as suggested by Liu. The suggestion/motivation for doing so would have been to enhance the speed and accuracy of the odometry system (see col. 3, lines 55-67). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Shan, while the teaching of Liu continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 12, Shan teaches receive a data stream from the IMU and performing pre-integration for the data stream to generate an IMU pre-integrate term (see 3.2); capture a frame through the camera, and performing feature detection and tracking for the captured frame to generate a two-dimensional (2D) feature track (see 3.1); and generate a depth measurement for a detected feature of the captured frame through the depth sensor (see 3.3).
Regarding claim 13, Xu teaches determine an IMU residue using the IMU pre-integrate term; determine a 2D feature residue from the 2D feature track; and determine the depth residue from the depth measurement (see section 3.1-3.3.2, 4.1, 4.3).
Regarding claim 17, Shan discloses perform keyframe initialization using the IMU pre-integrate term and the 2D feature track to generate an initial keyframe, wherein the sliding window graph is generated based on the initial keyframe (see section 3.1-3.3).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Shan et al (Sensors: “RGBD-Inertial Trajectory Estimation and Mapping for Ground Robots”) and in view of Proenca et al (Robotics and Autonomous Systems: “Probabilistic RGB-D odometry based on points, lines and planes under depth uncertainty”).
Regarding claim 8, Shan teaches all elements as mentioned above in claim 2. Shan does not teach expressly synchronizing feature detection and tracking for the captured frame with depth measurement of the detected feature.
Proenca, in the same field of endeavor, teaches synchronizing feature detection and tracking for the captured frame with depth measurement of the detected feature (see section 2-3.2).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Shan to utilize the cited limitations as suggested by Proenca. The suggestion/motivation for doing so would have been to enhance the visual odometry performance (see section 6). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Shan, while the teaching of Proenca continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Shan et al (Sensors: “RGBD-Inertial Trajectory Estimation and Mapping for Ground Robots”) and further in view of Kim et al (Robot Intelligence Technology and Applications: “Visual Odometry Algorithm Using an RGB-D Sensor and IMU in a Highly Dynamic Environment”)
Regarding claim 9, Shan teaches all elements as mentioned above in claim 1. Shan does not teach expressly computing feature velocity in image and depth domains based on the feature detection and tracking and the depth measurement.
Kim, in the same field of endeavor, teaches computing feature velocity in image and depth domains based on the feature detection and tracking and the depth measurement (see section 2-3).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Shan to utilize the cited limitations as suggested by Kim. The suggestion/motivation for doing so would have been to enhance the reliability for computing visual odometry in a highly dynamic environment (see conclusion). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Shan, while the teaching of Kim continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Shan et al (Sensors: “RGBD-Inertial Trajectory Estimation and Mapping for Ground Robots”), Liu et al (US 10,390,003 B1) and further in view of Proenca et al (Robotics and Autonomous Systems: “Probabilistic RGB-D odometry based on points, lines and planes under depth uncertainty”).
Regarding claim 18, Shan with Liu teaches all elements as mentioned above in claim 1. Shan with Liu does not teach expressly synchronize feature detection and tracking for the captured frame with depth measurement of the detected feature.
Proenca, in the same field of endeavor, teaches synchronize feature detection and tracking for the captured frame with depth measurement of the detected feature (see section 2-3.2).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Shan with Liu to utilize the cited limitations as suggested by Proenca. The suggestion/motivation for doing so would have been to enhance the visual odometry performance (see section 6). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Shan with Liu, while the teaching of Proenca continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Shan et al (Sensors: “RGBD-Inertial Trajectory Estimation and Mapping for Ground Robots”) with Liu et al (US 10,390,003 B1) and further in view of Kim et al (Robot Intelligence Technology and Applications: “Visual Odometry Algorithm Using an RGB-D Sensor and IMU in a Highly Dynamic Environment”).
Regarding claim 19, Shan with Liu teaches all elements as mentioned above in claim 1. Shan with Liu does not teach expressly compute feature velocity in image and depth domains based on the feature detection and tracking and the depth measurement.
Kim, in the same field of endeavor, teaches compute feature velocity in image and depth domains based on the feature detection and tracking and the depth measurement (see section 2-3).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Shan with Liu to utilize the cited limitations as suggested by Kim. The suggestion/motivation for doing so would have been to enhance the reliability for computing visual odometry in a highly dynamic environment (see conclusion). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Shan with Liu, while the teaching of Kim continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Shan et al (Sensor: “RGBD-Inertial Trajectory Estimation and Mapping for Ground Robots”) in view of Kerl et al (CVG: “Dense visual SLAM for RGB-D Cameras”).
Regarding claim 21, Shan teaches all elements as mentioned above in claim 1. Shan does not teach expressly depth residue corresponds to a difference between an actual measurement obtained from the depth sensor and a predicted measurement.
Kerl, in the same field of endeavor, teaches depth residue corresponds to a difference between an actual measurement obtained from the depth sensor and a predicted measurement (see section III, III.D.).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Shan to utilize the cited limitations as suggested by Kerl. The suggestion/motivation for doing so would have been to enhance pose accuracy and reduce errors (see abstract). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Shan, while the teaching of Kerl continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Allowable Subject Matter
Claims 4-6, 10, 14-16, 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claims 4-6, 10, 14-16, 20, none of the references of record alone or in combination suggest or fairly teach wherein generating and optimizing the sliding window graph comprises: determining an IMU factor based on the IMU residue; determining an anchor frame vision factor based on at least the depth residue, in case that the captured frame is an anchor frame; determining a non-anchor frame vision factor based on the 2D feature residue and the depth residue, in case that the captured frame is a non-anchor frame; and optimizing the sliding window graph based on the IMU factor and one of the anchor frame vision factor and the non-anchor frame vision factor.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EDWARD PARK. The examiner’s contact information is as follows:
Telephone: (571)270-1576 | Fax: 571.270.2576 | Edward.Park@uspto.gov
For email communications, please notate MPEP 502.03, which outlines procedures pertaining to communications via the internet and authorization. A sample authorization form is cited within MPEP 502.03, section II.
The examiner can normally be reached on M-F 9-6 CST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Moyer, can be reached on (571) 272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EDWARD PARK/
Primary Examiner, Art Unit 2666
Prosecution Insights