Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicants
This communication is in response to the amendment filed on 03/03/2026.
Claim 1-11, 18-19 and 22-28 are pending. Claim 12-17 and 20-21 have been cancelled.
Claims 1-5, 7, 8, 18, 19, 22-25, 27 and 28 remain rejected.
Response to Arguments
Applicant's amendments and arguments filed on 03/03/2026 have been fully considered but they are not persuasive. The newly added limitations, “the operation of removing the influence of rotation comprising determining pixel distance differences of the frame images based on information about rotation from an inertia measuring unit” and the amended limitations, “with a having an adjustable size … based on the pixel distance differences” have changed the scope of claims.
Applicant’s arguments and amendments for the double patenting and 112b rejections have been fully considered and they are persuasive and sufficient. The double patenting and 112b rejections have been withdrawn.
Applicant’s arguments (cited on page 13-14 of the remarks filed on 03/03/2026) with respect to the limitations, “removing influence of rotation, the operation of removing the influence of rotation comprising determining pixel distance differences of the frame images based on information about rotation from an inertia measuring unit … based on the pixel distance differences” have been considered but are moot because the new ground of rejection does not rely on reference(s) applied in the prior rejection of record for teaching or matter specifically challenged in the argument.
The applicant states on page 14-15 of the remarks regarding the substance of the examiner’s rejection, ALCANTARILLA does not teach “screening, with a sliding window having an adjustable size.” However, ALCANTARILLA teaches “screening, with a sliding window having an adjustable size” ([0156] If the number of extracted features is sufficient, then the image frame is not rejected and the processor proceeds to step 570. In this step, the processor evaluates whether there are enough keyframes for the sliding window estimator 650. The sliding window is defined in part by having a set number of keyframes (namely, 3 keyframes, in the examples above). However, immediately after initialisation of the system, there will be no keyframes. In step 570, the current image frame is selected 595 for initialisation as a keyframe as long as there are fewer than the required number of keyframes in the sliding window. (This required number will be referred to as the “first threshold”.); [0152] FIG. 10 is a flowchart illustrating a method of selecting frames to be keyframes ... image frames are first selected 550 to be keyframes; [0040] The selecting may include selecting an image frame to be a keyframe if the number of keyframes in a sliding temporal window before the image frame is below a first threshold).
ALCANTARILLA’s sliding window that is in the process of selecting or rejecting to a certain threshold reads on a system operating with an adjustable window size. Because ALCANTARILLA teaches starting with 0 keyframes and continuously adding them until the threshold is met, this inherent fluctuation and growth covers what the claimed specification defines as adjusting the size of the window. (Paragraph [0061] states that “the size of the sliding window can be adjusted” and that “the specific size can be determined based on the actual situation, such as the size of 5 image frames to the size of 10 image frames.”) Furthermore, nothing in the claim language precludes “The sliding window is defined in part by having a set number of keyframes (namely, 3 keyframes, in the examples above)” from reading on the claimed invention. Here, 3 keyframes is just nominal number as an example, not a fixed number. The set number of keyframes (the required number of keyframes or first threshold) could be 5 or 10.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 7, 18, 19 and 27 are rejected under 35 U.S.C. 103 as being unpatentable over ALCANTARILLA et al. (U.S. Publication No. 2023/0117498) (hereafter, "ALCANTARILLA") in view of Nerurkar et al. (U.S. Publication No. 2018/0276863) (hereafter, "Nerurkar") and further in view of Xu et al. (U.S. Publication No. 2019/0206116) (hereafter, "Xu").
Regarding claim 1, ALCANTARILLA teaches A method for simultaneous localization and mapping initialization, comprising ([0037] The method may further comprise initialising new keyframes in a local map, in the current session, from selected image frames of the series of image frames; [0004] a single SLAM session - initialising the system at an unknown location in an unknown environment, mapping (and localising) as the device moves around the environment): obtaining a predetermined number of continuous frame images and ([0010] obtaining the visual data, comprising a series of image frames captured by a camera of the mobile device; [0109] M frames (in the present example, we use M = 5 and N = 3)) preprocessing the predetermined number of continuous frame images ([0037] The method may further comprise initialising new keyframes in a local map, in the current session, from selected image frames of the series of image frames; [0038] Image frames may be selected as keyframes based one or more of a number of criteria. Selecting one or more images to be keyframes may be based at least in part on the inertial data; [0152] image frames are first selected 550 to be keyframes, and then 600 initialised as keyframes), the preprocessing comprising an operation of removing influence of rotation ([0154] In step 555 … if there has been an IMU saturation event (accelerometer or gyroscope), the processor waits at least 15 frames before initialising any new keyframes into the system. In this way, the system avoids inserting keyframes very close to IMU saturation events where the tracking may be unstable due to motion blur ... for image frames captured during a first time interval after a saturation event, the image frames are rejected 590 by the keyframe selection process; [0039] the selecting includes: detecting a saturation event in the inertial data; and in response, rejecting image frames captured during a first time interval after the saturation event. A saturation event occurs when one or more inertial sensors reports an inertial measurement equal to a maximum measurable value. Rejecting image frames means preventing the relevant frames from being selected as keyframes; [0161] & [0162]), the operation of removing the influence of rotation comprising ([0154] the system avoids inserting keyframes very close to IMU saturation events; [0039] Rejecting image frames means preventing the relevant frames from being selected as keyframes; [0162] In step 560, the processor compares the rotational uncertainty measure against a rotational uncertainty threshold. If the measure is above the threshold, the processor selects 595 the current image frame as a keyframe. Otherwise, ... it can reject 590 the current image frame as a keyframe) … screening, with a sliding window having an adjustable size ([0156] the processor evaluates whether there are enough keyframes for the sliding window estimator 650. The sliding window is defined in part by having a set number of keyframes (namely, 3 keyframes, in the examples above). However, immediately after initialisation of the system, there will be no keyframes. In step 570, the current image frame is selected 595 for initialisation as a keyframe as long as there are fewer than the required number of keyframes in the sliding window. (This required number will be referred to as the “first threshold”.); [0040] The selecting may include selecting an image frame to be a keyframe if the number of keyframes in a sliding temporal window before the image frame is below a first threshold), initial key frames from the predetermined number of continuous frame images ([0155] If the number of visual features extracted is below a threshold number, the processor rejects 590 the image frame; [0156] In this step, the processor evaluates whether there are enough keyframes for the sliding window estimator 650. The sliding window is defined in part by having a set number of keyframes (namely, 3 keyframes, in the examples above). However, immediately after initialisation of the system, there will be no keyframes. In step 570, the current image frame is selected 595 for initialisation as a keyframe as long as there are fewer than the required number of keyframes in the sliding window; [0157] If there is already a sufficient number of keyframes in the sliding window, then the current frame is not (yet) selected as a keyframe; [0152] FIG. 10 is a flowchart illustrating a method of selecting frames to be keyframes ... image frames are first selected 550 to be keyframes; [0149] the use of the sliding window avoids the need to optimise over the full set of keyframes and frames. Likewise, the use of the predicted pose from the IMU helps to better target the computational effort in the matching step 840, by means of the visibility prediction in step 830) ... the initial key frames comprising a plurality of key frames ([0037] The method may further comprise initialising new keyframes in a local map, in the current session, from selected image frames of the series of image frames); and performing, based on the plurality of key frames, simultaneous localization and mapping initialization ([0150] In both the first mode (SLAM mode) and the second mode (localisation mode) … the system selected new keyframes for initialisation 600 as a pose was calculated for each image frame).
ALCANTARILLA does not expressly teach determining pixel distance differences of the frame images based on information about rotation from an inertia measuring unit … based on the pixel distance differences.
However, Nerurkar teaches determining pixel distance differences of the frame images based on information about rotation ([0024] the electronic device 100 can identify spatial features present in one set of captured images of the image data 136, determine the initial distances to these spatial features, and then track the changes in position and distances of these spatial features in subsequent captured imagery to determine the change in pose and/or scene of the electronic device 100 in a free frame of reference. In this approach, certain non-image sensor data, such as gyroscopic data or accelerometer data, can be used to correlate spatial features observed in one image with spatial features observed in a subsequent image) from an inertia measuring unit ([0011] the electronic device includes one or more imaging sensors (e.g., imaging cameras) and includes one or more non-image sensors, such as an inertial measurement unit (IMU), that can provide information indicative of the pose and scene (e.g., local environment) of the electronic device).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of ALCANTARILLA to incorporate the step/system of using gyroscopes (IMU data) to track changes in distances/positions of frame images taught by Nerurkar. Even if Nerurka does not use the exact words "pixel distance differences," determining pixel distance differences between frames using IMU data for rotation or the combination of image features and gyroscopic data to track device pose are well-established in the art.
The suggestion/motivation for doing so would have been to improve the efficiency and accuracy of merging maps for simultaneous localization and mapping (SLAM) ([0013] to achieve a more efficient and accurate merger of ADFs, the map merger represents relationships among ADFs in an undirected graph, with vertices representing maps and edges representing transformations between maps. In this way, the map merger can use the undirected graph to more accurately represent the relations between any two maps, allowing more efficient merger of new maps to a previously stored collection of maps, and allowing for the development of more flexible and efficient algorithms for manipulating the merged maps). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results.
The combination of ALCANTARILLA and Nerurkar does not expressly teach “based on the pixel distance differences”.
However, Xu teaches screening , with a sliding window … the key frames … from … frame images based on pixel distance differences ([0060] After obtaining the frame, in step 303, a processor of the device processes the frame to obtain feature points, and computes feature descriptor for each feature points ... the feature point may include more than one pixel. In certain embodiments, the feature descriptors are ORB feature descriptors; [0061] In step 305, the frame, the feature points, and the associated feature descriptors are stored in a buffer. If the buffer has more images than a pre-defined threshold, the oldest image is discarded. This buffer is also referred to as sliding window; [0063] When there is no initial 3D map available, at step 308, the device determines whether there are a pair of good images (frames) from the sliding window. The quality of the pair of images are evaluated by certain predetermined criteria. For example, a pair of images that having more feature points are favorable. In certain embodiments, a sequence of images in the sliding window are scored using visual parallax feature, planar feature, etc., and a pair of images having a high score, i.e., having images with large visual parallax and without pure planar structure, are selected to construct the initial 3D map).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of ALCANTARILLA and Nerurkar to incorporate the step/system of selecting good images in the sliding window by using visual parallax feature derived from and evaluated by measuring pixel distance differences between frames as the camera moves taught by Xu.
The suggestion/motivation for doing so would have been to improve the accuracy of the SLAM solution by selecting key frames from frames captured by the camera having high quality for estimating the pose of the camera ([0050] disadvantageously affects accuracy of the SLAM solution; [0052] the present invention provides an improved SLAM solution to overcome the above discussed disadvantages … the solution optimizes pose of the camera by bundle adjustment (BA) using each captured frame instead of only using key frames, so as to improve the accuracy of pose estimation of the camera. Here key frames are frames selected from a sequence of frames captured by the camera that have more feature points and have high quality for estimating the pose of the camera). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine ALCANTARILLA and Nerurkar with Xu to obtain the invention as specified in claim 1.
Regarding claim 7, the combination of ALCANTARILLA and Nerurkar with Xu teaches all the limitations of claim 1 above. ALCANTARILLA teaches wherein the screening, with a sliding window having an adjustable size, initial key frames from the predetermined number of continuous frame images ([0156] If the number of extracted features is sufficient, then the image frame is not rejected and the processor proceeds to step 570. In this step, the processor evaluates whether there are enough keyframes for the sliding window estimator 650. The sliding window is defined in part by having a set number of keyframes (namely, 3 keyframes, in the examples above). However, immediately after initialisation of the system, there will be no keyframes. In step 570, the current image frame is selected 595 for initialisation as a keyframe as long as there are fewer than the required number of keyframes in the sliding window. (This required number will be referred to as the “first threshold”.); [0152] FIG. 10 is a flowchart illustrating a method of selecting frames to be keyframes ... image frames are first selected 550 to be keyframes; [0040] The selecting may include selecting an image frame to be a keyframe if the number of keyframes in a sliding temporal window before the image frame is below a first threshold; [0149] the use of the sliding window avoids the need to optimise over the full set of keyframes and frames. Likewise, the use of the predicted pose from the IMU helps to better target the computational effort in the matching step 840, by means of the visibility prediction in step 830).
ALCANTARILLA does not expressly teach “based on the pixel distance differences comprises: screening the initial key frames in the sliding window based on the pixel distance differences”.
However, Xu teaches “based on the pixel distance differences comprises: screening the initial key frames in the sliding window based on the pixel distance differences” ([0060] After obtaining the frame, in step 303, a processor of the device processes the frame to obtain feature points, and computes feature descriptor for each feature points ... the feature point may include more than one pixel. In certain embodiments, the feature descriptors are ORB feature descriptors; [0061] In step 305, the frame, the feature points, and the associated feature descriptors are stored in a buffer. If the buffer has more images than a pre-defined threshold, the oldest image is discarded. This buffer is also referred to as sliding window; [0063] When there is no initial 3D map available, at step 308, the device determines whether there are a pair of good images (frames) from the sliding window. The quality of the pair of images are evaluated by certain predetermined criteria. For example, a pair of images that having more feature points are favorable. In certain embodiments, a sequence of images in the sliding window are scored using visual parallax feature, planar feature, etc., and a pair of images having a high score, i.e., having images with large visual parallax and without pure planar structure, are selected to construct the initial 3D map).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of ALCANTARILLA to incorporate the step/system of selecting good images in the sliding window by using visual parallax feature derived from and evaluated by measuring pixel distance differences between frames as the camera moves taught by Xu.
Motivation for this combination has been stated in claim 1.
With respect to claim 18, arguments analogous to those presented for claim 1, are applicable.
With respect to claim 19, arguments analogous to those presented for claim 1, are applicable.
With respect to claim 27, arguments analogous to those presented for claim 7, are applicable.
Claim(s) 2, 5, 22 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over ALCANTARILLA et al. (U.S. Publication No. 2023/0117498) (hereafter, "ALCANTARILLA") in view of Nerurkar et al. (U.S. Publication No. 2018/0276863) (hereafter, "Nerurkar") and further in view of Xu et al. (U.S. Publication No. 2019/0206116) (hereafter, "Xu") and DAS et al. (U.S. Publication No. 2021/0042996) (hereafter, "DAS").
Regarding claim 2, the combination of ALCANTARILLA and Nerurkar with Xu teaches all the limitations of claim 1 above. ALCANTARILLA teaches wherein the performing, based on the plurality of key frames, simultaneous localization and mapping initialization comprises ([0150] In both the first mode (SLAM mode) and the second mode (localisation mode) … the system selected new keyframes for initialisation 600 as a pose was calculated for each image frame).
ALCANTARILLA does not expressly teach determining relative poses of a first key frame and a last key frame in the plurality of key frames; obtaining, based on the relative poses of the first key frame and the last key frame, three-dimensional space points of respective key frames in the plurality of key frames; determining relative poses of respective key frames in the plurality of key frames based on the relative poses of the first key frame and the last key frame and the three-dimensional space points of respective key frames in the plurality of key frames; and establishing an initial map based on the three-dimensional space points of respective key frames in the plurality of key frames and the relative poses of respective key frames in the plurality of key frames.
However, DAS teaches determining relative poses of a first key frame and a last key frame in the plurality of key frames ([0053] a) Identifying a plurality of key frames in the input image sequence based on edge correspondences of 2D points between successive images frames in the input image sequence. The key frame detection comprises detecting edges in a first frame and a second frame in the input image sequences; [0054] b) Determining initial pose of the image sensing device by obtaining rotation and translation of a second key frame with respect to a first key frame from the plurality of key frames); obtaining, based on the relative poses of the first key frame and the last key frame, three-dimensional space points of respective key frames in the plurality of key frames ([0055] c) Determining an initial 3D map of the area of interest using the initial pose, wherein the initial 3D map provides a plurality of initial 3D points); determining relative poses of respective key frames in the plurality of key frames based on the relative poses of the first key frame and the last key frame and the three-dimensional space points of respective key frames in the plurality of key frames; and ([0056] d) Obtaining a plurality of successive initial poses of the image sensing device based on a resection technique that utilizes the initial 3D map and edge correspondences of the 2D Points in each of successive key frames among the plurality of key frames) establishing an initial map based on the three-dimensional space points of respective key frames in the plurality of key frames and the relative poses of respective key frames in the plurality of key frames ([0057] e) Determining initializations of a plurality of successive 3D points for each of the successive keyframes using a triangulation technique, wherein the triangulation technique determines associated each of the edge correspondences between the 2D points of each of the successive key-frames; [0058] f) Performing a bundle adjustment for the SLAM based on the initializations of the plurality of successive 3D points and the plurality of successive initial poses).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of ALCANTARILLA to incorporate the step/system of determining relative poses of a first key frame and a second key frame, obtaining 3d points of frames based on the relative poses, determining initial poses based on the 3d points and the relative poses such as edge correspondences and building a 3D map based on the 3D points and the initial poses taught by DAS.
The suggestion/motivation for doing so would have been to improve the accuracy of SLAM systems ([0005] Object SLAM is a relatively new paradigm wherein SLAM information is augmented with objects in the form of its poses to achieve more semantically meaningful maps with the eventful objective of improving the accuracy of SLAM systems). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine ALCANTARILLA with DAS to obtain the invention as specified in claim 2.
Regarding claim 5, the combination of ALCANTARILLA, Nerurkar and Xu with DAS teaches all the limitations of claim 2 above. ALCANTARILLA teaches determining a second reprojection error based on the three-dimensional space points of respective key frames in the plurality of key frames ([0047] The measurements may be used to calculate a re-projection error, associated with the matched visual features); performing a global optimization based on the second reprojection error to obtain the three-dimensional space points of respective key frames in the plurality of key frames and the relative poses of respective key frames in the plurality of key frames after the optimization ([0047] Refining the second pose may comprise minimising the sum of these errors; [0050] The refining may comprise bundle adjustment, in which the 3-D positions of the landmarks from the existing map are held constant. Bundle adjustment involves simultaneously refining the 3-D positions of the landmarks, as well as the poses of the frames and keyframes in the sliding window) … establishing the initial map based on the three-dimensional space points of respective key frames in the plurality of key frames and the relative poses of respective key frames in the plurality of key frames after the optimization ([0117] Once the visual-inertial SLAM system finishes processing a sequence (that is, finishes a mapping session), we use the set of optimised keyframe poses and 3D landmarks Θ = {X*, Y*} for learning how to predict the visibility of known 3D points with respect to a query camera pose in the large-scale environment).
ALCANTARILLA does not expressly teach before the establishing an initial map based on the three-dimensional space points of respective key frames in the plurality of key frames and the relative poses of respective key frames in the plurality of key frames, further comprising … and wherein the establishing an initial map based on the three-dimensional space points of respective key frames in the plurality of key frames and the relative poses of respective key frames in the plurality of key frames comprises.
However, DAS teaches before the establishing an initial map based on the three-dimensional space points of respective key frames in the plurality of key frames and the relative poses of respective key frames in the plurality of key frames, further comprising ([0057] e) Determining initializations of a plurality of successive 3D points for each of the successive keyframes using a triangulation technique, wherein the triangulation technique determines associated each of the edge correspondences between the 2D points of each of the successive key-frames; [0058] f) Performing a bundle adjustment for the SLAM based on the initializations of the plurality of successive 3D points and the plurality of successive initial poses) … and wherein the establishing an initial map based on the three-dimensional space points of respective key frames in the plurality of key frames and the relative poses of respective key frames in the plurality of key frames comprises ([0057] e) Determining initializations of a plurality of successive 3D points for each of the successive keyframes using a triangulation technique, wherein the triangulation technique determines associated each of the edge correspondences between the 2D points of each of the successive key-frames; [0058] f) Performing a bundle adjustment for the SLAM based on the initializations of the plurality of successive 3D points and the plurality of successive initial poses).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the optimization (by using reprojection error) method of ALCANTARILLA to incorporate the step/system of building a 3D map based on the 3D points and the initial poses taught by DAS.
Motivation for this combination has been stated in claim 2.
With respect to claim 22, arguments analogous to those presented for claim 2, are applicable.
With respect to claim 25, arguments analogous to those presented for claim 5, are applicable.
Claim(s) 3, 4, 8, 23, 24 and 28 are rejected under 35 U.S.C. 103 as being unpatentable over ALCANTARILLA et al. (U.S. Publication No. 2023/0117498) (hereafter, "ALCANTARILLA") in view of Nerurkar et al. (U.S. Publication No. 2018/0276863) (hereafter, "Nerurkar") and further in view of Xu et al. (U.S. Publication No. 2019/0206116) (hereafter, "Xu"), DAS et al. (U.S. Publication No. 2021/0042996) (hereafter, "DAS") and LIU et al. (U.S. Publication No. 2022/0292715) (hereafter, "LIU").
Regarding claim 3, the combination of ALCANTARILLA, Nerurkar and Xu with DAS teaches all the limitations of claim 2 above. DAS teaches wherein the determining relative poses of respective key frames in the plurality of key frames based on the relative poses of the first key frame and the last key frame and the three-dimensional space points of respective key frames in the plurality of key frames comprises ([0056] d) Obtaining a plurality of successive initial poses of the image sensing device based on a resection technique that utilizes the initial 3D map and edge correspondences of the 2D Points in each of successive key frames among the plurality of key frames).
DAS does not expressly teach determining positions of the three-dimensional space points of respective key frames in the plurality of key frames projected into the first key frame and the last key frame; determining a first reprojection error based on the positions obtained by the projection; and determining the relative poses of respective key frames in the plurality of key frames based on the first reprojection error and the three-dimensional space points of respective key frames in the plurality of key frames.
However, LIU teaches determining positions of the three-dimensional space points of respective key frames in the plurality of key frames projected into the first key frame and the last key frame ([0122] feature points and descriptors on a similar key frame obtained by looking up a current frame may be extracted, and a three-dimensional (3D) point cloud corresponding to the current frame and the similar key frame may be obtained through a depth value corresponding to each key frame).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of DAS to incorporate the step/system of obtaining 3D point cloud corresponding to the current frame and the similar key frame taught by LIU.
The suggestion/motivation for doing so would have been to improve the accuracy and calculation speed of pose estimation ([0057] the method of estimating a pose of a device, according to embodiments of the present disclosure, improves the robustness of image frame tracking, and helps to improve the accuracy and calculation speed of pose estimation). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results.
The combination of DAS and LIU does not expressly teaches determining a first reprojection error based on the positions obtained by the projection; and determining the relative poses of respective key frames in the plurality of key frames based on the first reprojection error and the three-dimensional space points of respective key frames in the plurality of key frames.
However, Xu teaches determining a first reprojection error based on the positions obtained by the projection; and ([0050] a SLAM solution, in which an error in the correspondences between the 3D map points of the 3D map and the feature points from one frame propagates to the next frame) determining the relative poses of respective key frames in the plurality of key frames based on the first reprojection error and the three-dimensional space points of respective key frames in the plurality of key frames ([0055] the camera pose is further optimized using Bundle Adjustment (BA) optimization on the sliding window. BA is a non-linear optimization approach that optimizes the 3D map points' locations and camera's poses all together by minimizing the re-projection errors of the 3D map points on the camera view planes. ORB-SLAM and VINS-Mono perform BA whenever there is a new key frame so as to correct pose tracking error at regular or irregular intervals).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of DAS and LIU to incorporate the step/system of determining projection error and refining the pose relative to the 3D map by using the projection error and 3D map points taught by Xu.
The suggestion/motivation for doing so would have been to improve the accuracy of the SLAM solution by selecting key frames from frames captured by the camera having high quality for estimating the pose of the camera ([0050] disadvantageously affects accuracy of the SLAM solution; [0052] the present invention provides an improved SLAM solution to overcome the above discussed disadvantages … the solution optimizes pose of the camera by bundle adjustment (BA) using each captured frame instead of only using key frames, so as to improve the accuracy of pose estimation of the camera. Here key frames are frames selected from a sequence of frames captured by the camera that have more feature points and have high quality for estimating the pose of the camera). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine DAS with LIU and Xu to obtain the invention as specified in claim 3.
Regarding claim 4, the combination of ALCANTARILLA, Nerurkar, DAS, LIU and Xu teaches all the limitations of claim 3 above. ALCANTARILLA teaches … after removing the influence of rotation ([0154] In step 555 … if there has been an IMU saturation event (accelerometer or gyroscope), the processor waits at least 15 frames before initialising any new keyframes into the system. In this way, the system avoids inserting keyframes very close to IMU saturation events where the tracking may be unstable due to motion blur ... for image frames captured during a first time interval after a saturation event, the image frames are rejected 590 by the keyframe selection process).
ALCANTARILLA does not expressly teach wherein the first reprojection error is a reprojection error.
However, Xu teaches wherein the first reprojection error is a reprojection error ([0050] a SLAM solution, in which an error in the correspondences between the 3D map points of the 3D map and the feature points from one frame propagates to the next frame).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of ALCANTARILLA to incorporate the step/system of determining projection error taught by Xu.
Motivation for this combination has been stated in claim 1.
Regarding claim 8, the combination of ALCANTARILLA, Nerurkar and Xu with DAS teaches all the limitations of claim 2 above. DAS teaches wherein the determining relative poses of a first key frame and a last key frame in the plurality of key frames comprises ([0053] a) Identifying a plurality of key frames in the input image sequence based on edge correspondences of 2D points between successive images frames in the input image sequence. The key frame detection comprises detecting edges in a first frame and a second frame in the input image sequences; [0054] b) Determining initial pose of the image sensing device by obtaining rotation and translation of a second key frame with respect to a first key frame from the plurality of key frames).
DAS does not expressly teach extracting two-dimensional key points of the first key frame and the last key frame to obtain a two-dimensional key point of the first key frame and a two-dimensional key point of the last key frame and determining the relative poses of the first key frame and the last key frame with the two- dimensional key point of the first key frame and the two-dimensional key point of the last key frame.
However, LIU teaches extracting two-dimensional key points of the first key frame and the last key frame to obtain a two-dimensional key point of the first key frame and a two-dimensional key point of the last key frame ([0088] In operation S102, ... obtaining data-related information between image frames based on a feature matching relationship between the current frame and the similar key frame; [0090] The extracted features may include features such as a point, a line, and a plane; [0091] the features may include a key point extracted when extracting a feature point and a descriptor); and determining the relative poses of the first key frame and the last key frame with the two- dimensional key point of the first key frame and the two-dimensional key point of the last key frame ([0093] In operation S103, the method of estimating a pose of a device may include obtaining the pose of the device based on the data-related information; [0094] the method of estimating a pose of a device may include obtaining a more accurate pose result by obtaining the data-related information and then calculating the pose of the device based on the data-related information according to the bundle adjustment or the like).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of DAS to incorporate the step/system of extracting key points of the current frame and the similar key frame and estimating poses based on the key points taught by LIU.
The suggestion/motivation for doing so would have been to improve the accuracy and calculation speed of pose estimation ([0057] the method of estimating a pose of a device, according to embodiments of the present disclosure, improves the robustness of image frame tracking, and helps to improve the accuracy and calculation speed of pose estimation). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine DAS and LIU to obtain the invention as specified in claim 8.
With respect to claim 23, arguments analogous to those presented for claim 3, are applicable.
With respect to claim 24, arguments analogous to those presented for claim 4, are applicable.
With respect to claim 28, arguments analogous to those presented for claim 8, are applicable.
Allowable Subject Matter
Claim 6, 9, 10,11 and 26 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL C. CHANG whose telephone number is (571)270-1277. The examiner can normally be reached Monday-Thursday and Alternate Fridays 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan S. Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL C CHANG/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights