DETAILED ACTION
Status
This Office Action is responsive to claims filed on 10/31/2024. Please note Claims 1-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3 and 12-20 are rejected under 35 U.S.C. 103 as being unpatentable over HE (CN 114485637 A, cited on IDS 3/11/2025), in view of BERKEBILE (US 20230016490 A1, cited on IDS 3/11/2025).
Regarding Claim 1, HE discloses a method for detecting and correcting drift in camera poses, the method comprising: obtaining a plurality of images and a plurality of captured camera poses associated with the plurality of images from an augmented reality (AR) tracking system ([0006] “The measurement preprocessing process includes processing the frame images collected by the vision sensor, and simultaneously pre-integrating the inertial measurement results corresponding to two consecutive frame images;”);
detecting inconsistencies associated with the plurality of captured camera poses ([0337] “When loop closures are detected, feature consistency is checked and connections between local sliding windows and loop closure candidates are established by matching the Brief feature descriptors.”) to identify locally rigid captured camera pose groups ([0134] “at least 6 groups of 3D-2D corresponding points are required. In order to estimate the pose of the camera more accurately, the present invention randomly selects 18 points from the currently measured feature points, and when there are less than 18 points, all points are selected, it cannot be calculated when it is less than 6 points, so it is discarded.”);
detecting features in the plurality of images ([0087] “Visual front-end processing, tracking the feature points of each frame of image through the LK sparse optical flow algorithm, and detecting the corner feature points of each frame of image”);
matching features between the plurality of images, wherein matching features comprises: for captured camera poses in a locally rigid captured camera pose group, generating pairs of captured camera poses in the locally rigid captured camera pose group and matching features between the pairs of captured camera poses ([0100] “In the relocation process, the bag-of-words model is used to realize loop closure detection. First, a bag-of-words model is constructed based on the feature point data actually collected in the space operation environment. When the position of the visual sensor changes, a new feature point set is calculated in real time. , after the feature point set is formed into a new description vector, it is matched with the key frame feature description vector. If the matching is successful, the loop closure detection is realized.”); and
for captured camera poses across locally rigid captured camera pose groups, generating pairs of captured camera poses and matching features between the pairs of captured camera poses ([0100] “The bag-of-words model is used to identify the visited locations, and then a feature-level connection between the candidate closed loop and the current frame image is established. By integrating the feature pairs into the visual-inertial odometry VIO based on nonlinear optimization, a drift-free target state estimation.”);
within each locally rigid captured camera pose group, triangulating three-dimensional (3D) landmarks ([0119] “Triangulate all the features in these two frames. If they do not exist, save the latest frame image in the sliding window and wait for a new frame image; for camera pose estimation, the triangulated features are used, and the PnP method is used.”), wherein each landmark comprises a 3D point and a plurality of 2D points of images that correspond to the 3D point ([0119] “For feature point tracking, first check the latest frame image of the image collected by the head-mounted augmented reality system and all frames in the sliding window. The feature correspondence between images, if there are more than a certain number of tracking features and rotation compensation pixels in the latest frame image and any frame image in the sliding window, use the tracking features and rotation compensation pixels for the latest frame image to perform relative rotation and scale translation”);
for a pair of locally rigid captured camera pose groups, wherein the pair includes a first group of locally rigid captured camera poses and a second group of locally rigid camera poses: determining correspondences between 3D landmarks of the first group and two-dimensional (2D) observations of same features in the second group ([0134] “When there are more than 6 points, the present invention adopts SVD (Singular Value Decomposition) to obtain the least squares solution. Construct a system of linear equations using 18 3D-2D corresponding point groups”); and
registering the second group to the first group based on perspective-n-point ([0125] “According to the feature point matching information and feature position estimation, the SFM algorithm is used to obtain the three-dimensional space position of the current feature point and its projected position on the camera, and then the PnP algorithm is directly used to estimate the image pose.”);
performing bundle adjustment of captured camera poses within and across registered groups of locally rigid captured camera pose groups ([0317] “The present invention adopts a visual inertia-oriented cluster adjustment method to perform fusion optimization.”).
HE does not expressly disclose generating a 3D model based on the adjusted camera poses.
However, in the same field of endeavor, BERKEBILE discloses generating a 3D model based on the adjusted camera poses ([0273] “In some embodiments, the 3D model is sharable among multiple users of spatial computing systems so that different users may construct their respective 3D models for their respective physical environments, and these respective 3D models may be integrated together to form a larger 3D model representing, for example, the union of the respective environments and sharable among a plurality of users.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the method of HE with the feature of generating a 3D model based on the adjusted camera poses. Doing so could enhance user experience as taught in BERKEBILE [0273].
Regarding Claim 2, HE-BERKEBILE discloses the method of claim 1, wherein detecting inconsistencies comprises detecting a change in tracking session identification associated with the plurality of captured camera poses (BERKEBILE [0268] “In these one or more embodiments, collaboration among multiple users of spatial computing systems (e.g., an MR system, a laptop computer, a mobile device, etc. having specifically installed software programs) facilitated by spatial alignment relative to a persistent coordinate frame (“PCF”), which may include a coordinate frame that is persistent relative to the physical world around a user and may be generally configured not to drift or move across multiple user sessions.”).
Regarding Claim 3, HE-BERKEBILE discloses the method of claim 1, wherein detecting inconsistencies comprises detecting a change in tracking session state associated with the plurality of captured camera poses (BERKEBILE [0268] “In these one or more embodiments, collaboration among multiple users of spatial computing systems (e.g., an MR system, a laptop computer, a mobile device, etc. having specifically installed software programs) facilitated by spatial alignment relative to a persistent coordinate frame (“PCF”), which may include a coordinate frame that is persistent relative to the physical world around a user and may be generally configured not to drift or move across multiple user sessions.”).
Regarding Claim 12, HE-BERKEBILE discloses the method of claim 1, wherein performing bundle adjustment within registered groups of locally rigid captured camera pose groups comprises using relative pose priors within the same group (HE [0108] “If the newly marginalized keyframe has a loop connected to the existing keyframe, the loopback edge in the pose graph is connected to the loopback frame, and the loopback edge only contains 4 degrees of freedom relative pose transformation, and the result of relocation is used to obtain The value of the loopback edge.”).
Regarding Claim 13, HE-BERKEBILE discloses the method of claim 1, wherein performing bundle adjustment comprises using captured camera poses of the locally rigid captured camera pose groups as enhanced priors in the bundle adjustment process (HE [0338] “For tightly coupled relocalization, the relocalization process efficiently correlates the current sliding window and historical pose graph maintained by the monocular VIO. During relocalization, the poses of all loopback frames are treated as constants and measured using the IMU , local vision measurements, and detection of feature consistency in loop closures to jointly optimize sliding windows.”).
Regarding Claim 14, HE-BERKEBILE discloses the method of claim 1, further comprising: providing an interface for manual restoration of the 3D model, including reconstructing and loading multiple point clouds separately (BERKEBILE [0106] “In some embodiments, each application 140 making use of the universe browser engine's service to render 3D content (e.g. composited 3D content) into the universe browser engine process may be required to first register a listener with the universe browser engine. This listener may be used to inform the application 140 of creation and destruction of rendering Prisms, based upon user movement and user interaction with those Prisms. A listener is an interface object that receives messages from an inter-process communication system.”).
Regarding Claim 15, HE-BERKEBILE discloses the method of claim 14, wherein the interface for manual restoration comprises tools for adjusting positions of separate point clouds corresponding to different pose groups (BERKEBILE [0210] “In these one or more embodiments, a virtual pointer may be utilized to move, size, and otherwise manipulate virtual objects, and simulated physics may be added to enhance operation of the pointer and also manipulation of objects.”).
Regarding Claim 16, HE-BERKEBILE discloses the method of claim 1, further comprising: meshing the triangulated points to create a 3D surface model (BERKEBILE [0177] “In some embodiments, the input mesh may have a high resolution, which may be indicated by the number of triangles. The input mesh may be generated by a reconstruction system (e.g., a volumetric 3D reconstruction system) and the input mesh may include 3D reconstruction data.”).
Regarding Claim 17, it recites similar limitations of claim 1. The rationale of claim 1 rejection is applied to reject claim 17.
Regarding Claim 18, HE-BERKEBILE discloses the system of claim 17, wherein detecting inconsistencies comprises detecting at least one of: a change in tracking session identification associated with the plurality of captured camera poses, a change in tracking session state associated with the plurality of captured camera poses, a translational acceleration exceeding a translational acceleration threshold, a rotational acceleration exceeding a rotational acceleration threshold, and a translational velocity exceeding a translational velocity threshold (see rejections to claims 2 and 3 above).
Regarding Claim 19, HE-BERKEBILE discloses the system of claim 17, wherein performing bundle adjustment within registered groups of locally rigid captured camera pose groups comprises using relative pose priors within the same group, and using captured camera poses of the locally rigid captured camera pose groups as enhanced priors in the bundle adjustment process (see rejections to claims 12 and 3 above).
Regarding Claim 20, it recites similar limitations of claim 1. The rationale of claim 1 rejection is applied to reject claim 20.
Allowable Subject Matter
Claims 4-11 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
ZELEK (US 20240104771 A1) – this reference discloses a system and method for visual simultaneous localization and mapping. The method including: extracting a blend of landmarks; associating descriptors and patches of pixels with the extracted landmarks; using the descriptors and patches of pixels, estimating a camera pose by performing feature matching and relative pose estimation; performing joint multi-objective pose optimization over photometric residuals and geometric residuals; updating a local map by performing Bundle Adjustment on the estimated pose; marginalizing extracted landmarks from the local map that are older than a predetermined number of keyframes and adding the descriptors associated with the marginalized landmarks to a global map; where there are loop closure candidates, performing point matching between a keyframe associated with the loop closure candidate and a keyframe most recently added to the global map; and rejecting the keyframe associated with the loop closure candidate if the number of matches is below a predetermined threshold.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHONG WU whose telephone number is (571)270-5207. The examiner can normally be reached MON-FRI: 9AM-5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHONG WU/Primary Examiner, Art Unit 2613
Prosecution Insights