DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 12-14, 16 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over D1 (U.S. PG-PUB NO. 2016/0209217) in view of D2 (U.S. PATENT NO. 7860301).
-Regarding claim 1, D1 discloses a data positioning apparatus (see abstract), comprising: a stereo camera which acquires visual data corresponding to 2D (stereo camera, [0031]); an inertial measurement unit which acquires inertia data corresponding to acceleration and angular velocity data (inertial measurement unit (IMU), [0035]); and a computing module which receives data acquired from the stereo camera and the inertial measurement unit and performs computation of a visual SLAM engine (controller 116, [0035]).
D1 is silent to teaching that 3D data. However, the claimed limitation is well known in the art as evidenced by D2.
In the same field of endeavor, D2 teaches 3D data (M2 is the dense stereo routine which computes dense 3D data from pairs of stereo images (the pair being made up of one image each from the monocular cameras 14 and 16) by matching image intensity or color distributions between the stereo images, col. 8 lines 58-68).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of D1 with the teaching of D2 in order to derive dense 3D/depth data from the same right/left pair on which D1 already performs a disparity calculation.
-Regarding claim 12, the combination further discloses the graph optimizing unit updates a pose estimated for the current frame to the pose trajectory estimated for a previous frame, examines outlier data, and optimizes the trajectory (D1, In the inner window 412, the optimization function tries to minimize the feature re-projection error combined with the orientation error between the gyroscope and the visual system, [0054]).
-Regarding claim 13, the combination further discloses the graph optimizing unit optimizes data for the estimated pose in real time on the basis of bundle adjustment (BA) and a loop closure algorithm when a loop is generated in the camera pose trajectory due to accumulation of the estimated pose (D1, The first summation minimizes the re-projection error, the second summation minimizes the relative pose between the two frames, [0056]; the error between the quaternion rotation given by the IMU and the observed quaternion rotation between two keyframes. In the back end, loop closure using place recognition is used. If loop closure occurs 419, an additional constraint is added to the SLAM graph. Again optimization 420 is carried out to improve the localization accuracy, [0057]).
-Regarding claim 14, the combination further discloses a visualization module which visualizes data acquired from the stereo camera, the inertial measurement unit, or the computing module and outputs the data to a display (D1, result is rendered, based on the mapped location, a position in the indoor environment, such as on a paper or onscreen display of the interior environment showing the path and/or location traversed, [0040]).
-Regarding claim 16, D1 discloses a data positioning method which is performed in a computing module of a data positioning apparatus (see abstract), comprising: a step of receiving visual data corresponding to 2D acquired from a stereo camera (stereo camera, [0031]) and acceleration and angular velocity data acquired from an inertial measurement unit (inertial measurement unit (IMU), [0035]); and a step of performing a computation of a visual SLAM engine on the basis of the received data (controller 116, [0035]).
D1 is silent to teaching that 3D data. However, the claimed limitation is well known in the art as evidenced by D2.
In the same field of endeavor, D2 teaches 3D data (M2 is the dense stereo routine which computes dense 3D data from pairs of stereo images (the pair being made up of one image each from the monocular cameras 14 and 16) by matching image intensity or color distributions between the stereo images, col. 8 lines 58-68).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of D1 with the teaching of D2 in order to derive dense 3D/depth data from the same right/left pair on which D1 already performs a disparity calculation.
-Regarding claim 19, the combination further discloses a computer program stored in a computer readable medium in which when an instruction of the computer program is executed, the method according to claim 16 is performed (D1, [0065]).
Claim(s) 2-4 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over D1 (U.S. PG-PUB NO. 2016/0209217) in view of D2 (U.S. PATENT NO. 7860301) and further in view of D3 (WO 2022/262878).
-Regarding claim 2, the combination is silent to teaching that the stereo camera acquires an RGB image corresponding to 2D data and depth data corresponding to 3D data. However, the claimed limitation is well known in the art as evidenced by D3.
In the same field of endeavor, D3 teaches the stereo camera acquires an RGB image corresponding to 2D data and depth data corresponding to 3D data (the input of the visual feature extraction module is two adjacent frames of RGB pictures superimposed along the channel, and the output is 1024-dimensional visual features, [0039]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of D3 in order to provide high relative displacement and relative pose estimation accuracy, and good robustness to data damage.
-Regarding claim 3, the combination further discloses the computing module includes: a data input unit which receives an RGB image and depth data from the stereo camera and inertia data from the inertial measurement unit (D1, controller 116 receives gyroscopic input 310 and image input 311 from the gyroscope 114 and cameras 112-1, 112-2, respectively, [0048]); a data computing unit which normalizes the RGB image and the inertia data (D1, Feature matching in the front end is performed using a zero mean sum of squared difference technique, [0052]); a pose estimating unit which estimates a pose with six degrees of freedom for a current frame using the normalized data (D1, pose estimation 312, [0049]); and a graph optimizing unit which updates a camera pose trajectory on the basis of the estimated pose (D1, back end 400 uses a graph 411 that maintains the constraints and relationships between the keyframes 410 sent by the front end 300, [0053]).
-Regarding claim 4, the combination further discloses the data computing unit performs a 2D convolution computation on the RGB data and extracts image feature information corresponding to a texture, an edge, or a color pattern by the 2D convolution computation (D3, visual feature extraction module is formed by sequentially stacking 10 layers of convolutional neural networks, and the convolution kernel sizes of the first three layers of convolutional neural networks in the 10-layer convolutional neural network are 7×7, 5×5, 5×5 in sequence, [0010]).
-Regarding claim 17, the combination further discloses the step of performing a computation of a visual SLAM engine includes: a step of normalizing an RGB image corresponding to the 2D data and inertia data corresponding to the acceleration and the angular velocity data (D1, Feature matching in the front end is performed using a zero mean sum of squared difference technique, [0052]; D3, RGB, [0039]); a step of estimating a pose with six degrees of freedom for a current frame using the normalized data (D3, the dimension of the second-layer fully-connected network is 128, and the dimension of the third-layer fully-connected network is 64. The dimension of the fourth-layer fully-connected network is 6, [0045]); and a step of updating a camera pose trajectory on the basis of the estimated pose (D3, pseudo labels, real labels, and mixed labels include relative displacements and relative rotations on the x, y, and z axes, [0053]).
Claim(s) 9, 10 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over D1 (U.S. PG-PUB NO. 2016/0209217) in view of D2 (U.S. PATENT NO. 7860301), D3 (WO 2022/262878) and further in view of D4 (WO 2019/180414).
-Regarding claim 9, the combination is silent to teaching that a depth estimating unit which is an auto-encoder-based model to estimate a depth value from the RGB image and generate a depth map. However, the claimed limitation is well known in the art as evidenced by D4.
In the same field of endeavor, D4 teaches a depth estimating unit which is an auto-encoder-based model to estimate a depth value from the RGB image and generate a depth map (mapping-net 300 may be an encoder-decoder (or autoencoder) type architecture, page 17).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of D4 in order to recover dense per-pixel depth from a single frame.
-Regarding claim 10, the combination further discloses an image warping unit which generates a warped image using the depth map received from the depth estimating unit and updates a pose on the basis of the warped image and a pose estimated by the pose estimating unit (D4, Based on this equation, Ik is synthesized by warping image 4 from image Ik+1 through a spatial transformer, page 13).
-Regarding claim 18, the combination further discloses the step of performing a computation of a visual SLAM engine includes: a step of estimating a depth value from the RGB image and generating a depth map (D4, (D4, Based on this equation, Ik is synthesized by warping image 4 from image Ik+1 through a spatial transformer, page 13), page 19); a step of generating a warped image using the depth map; and a step of updating a pose on the basis of the warped image and the estimated pose (D4, Based on this equation, Ik is synthesized by warping image 4 from image Ik+1 through a spatial transformer, page 13).
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over D1 (U.S. PG-PUB NO. 2016/0209217) in view of D2 (U.S. PATENT NO. 7860301), D4 (WO 2019/180414) and further in view of D5 (U.S. PG-PUB NO. 2020/0211206).
-Regarding claim 11, the combination is silent to teaching that the depth estimating unit calculates a 3D point using a depth value for each pixel of the RGB image and the image warping unit transforms the 3D points into a target camera coordinate system by applying a predicted camera orientation transformation, acquires a pixel coordinate by projecting the transformed 3D points onto a target image plane, and generates a warped image by sampling a pixel value corresponding to the pixel coordinate from the target image. However, the claimed limitation is well known in the art as evidenced by D5.
In the same field of endeavor, D5 teaches the depth estimating unit calculates a 3D point using a depth value for each pixel of the RGB image and the image warping unit transforms the 3D points into a target camera coordinate system by applying a predicted camera orientation transformation, acquires a pixel coordinate by projecting the transformed 3D points onto a target image plane, and generates a warped image by sampling a pixel value corresponding to the pixel coordinate from the target image ((pt|Dt)=Dt(pt)K−1h(pt) is a back-projection function from 2D to 3D space, [0055]; A product is obtained (410) between the back-projected pixel in 3D space ϕ(pt|Dt) and the relative camera pose Tt->s, [0056]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of D5 in order to provide better performance and more efficient convergence.
Allowable Subject Matter
Claims 5-8 and 15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PING Y HSIEH whose telephone number is (571)270-3011. The examiner can normally be reached Monday-Friday, 9am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PING Y HSIEH/ Primary Examiner, Art Unit 2664
Prosecution Insights