DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because it contains an embedded hyperlink and/or other form of browser-executable code. Applicant is required to delete the embedded hyperlink and/or other form of browser-executable code; references to websites should be limited to the top-level domain name without any prefix such as http:// or other browser-executable code. See MPEP § 608.01.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-9 and 11 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the term “a level” in lines 8 and 12 when referring to the “image processing neural network” and the “point cloud processing neural network”. A “level” is not understood as a standard, well-known component of a neural network, thus rendering the claim indefinite due to the undefined term.
The term “close” in claim 1 line 17 is a relative term which renders the claim indefinite. The term “close” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. It is not clear as to how “close” points mut be to meet this limitation.
Claim 1 recites the limitation "projection of this point of the point cloud" in line 18. There is insufficient antecedent basis for this limitation in the claim, as “this point” is unclear in which point of the introduced “every point of the point cloud” is being referred to.
Claim 1 recites the phrase “which increases otherwise” in line 19. It is unclear from the claim language what the term “otherwise” refers to and thus the rendering the claim indefinite as the boundaries of the claimed subject matter cannot be clearly determined.
Claim 1 recites “determining an error which decrease” in line 16 and “minimizing the determined error” in line 20. It is unclear if there is a difference between decreasing the error and minimizing the error. Furthermore, when the error gets minimized, does the error further get “decreased/minimized’ after the determination? Clarification/explanation is respectfully requested.
Claim 2 recites “the obtaining step” in lines 4 and 8. It is unclear which obtaining step is being referenced in each use of the term rendering the claim indefinite.
The term “close” in claim 2 line 12 is a relative term which renders the claim indefinite. The term “close” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. It is not clear as to how “close” points mut be to meet this limitation.
Claim 2 recites the limitation "projection of this point of the point cloud" in line 13. There is insufficient antecedent basis for this limitation in the claim, as “this point” is unclear in which point of the introduced “every point of the point cloud” is being referred to.
The term “if they are remote from one another” in claim 2 line 14 is a relative term which renders the claim indefinite. The term “if they are remote from one another” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Claim 2, line 18 recites “each distance”. It is unclear what each distance is referring to.
Claim 5 recites the limitation "the colour information of the image" in line 4. There is insufficient antecedent basis for this limitation in the claim.
Claim 6 recites “a point clouds” in line 3. It is unclear whether “a point cloud” is the same as previously recited “point cloud” in claim 1 rendering the claim indefinite. Additionally, the phrase recites an article mismatch between the singular article “a” and the plurality of the term “clouds” making the phrase unclear.
Claims 3-4, 7-9, and 11 are rejected due to their dependency on the rejected base claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 6, 8-9 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al “Learning 2D-3D Correspondences To Solve The Blind Perspective-n-Point Problem” (Included in IDS) (hereinafter “Liu”) in view of Bork US 20230277249 (hereinafter “Bork”).
Regarding Claim 1, Liu teaches a method for determining a 6D pose of a camera used to acquire an image of a scene using a point cloud of the scene (see abstract, model that determines the 6DoF camera pose based on a 2D image and 3d scene points), the method comprising a plurality of consecutive iterations each associated with a level of a plurality of consecutive levels and comprising (see Figure 2, section F implementation details, F.2 and section 3.2, the model contains a plurality of replicate modules and the algorithms include a plurality of iterations):
obtaining an initial 6D pose see section 4.2, the state-of the art Simultaneous Pose and Correspondence Determination (SoftPOSIT) model requires an initial estimation of rotation and translation [pose], the initial pose can be obtained by adding perturbation to the ground pose or averaging the rotation and using the mean of the translation),
obtaining, from a processing of the image using an image processing neural network configured to receive the image as input, and configured to be able to output, for the level of the iteration, which is a level in the image processing neural network (see section 3.2 and Figures 1 and 2, the two stream network contains a first branch with a feature extractor that takes the 2D points in the image coordinate system and determines features for each of the 2D points. The feature extractor contains multiple MLP [neural networks] and replicated modules [layers]):
one feature map (see Figure 2, the feature extractor outputs feature vectors),
obtaining, from a processing of the point cloud using a point cloud processing neural network configured to receive the point cloud as input, and configured to be able to output, for the level of the iteration, which is a level in the point cloud processing neural network (see section 3.2 and Figures 1 and 2, the two stream network contains a second branch with a feature extractor that takes the 3D points in the image coordinate system and determines features for each of the 3D points. The feature extractor contains multiple MLP [neural networks] replicated modules [layers]):
one plurality of features, wherein each feature of the plurality of features is associated with a respective point of the point cloud (see section 3.2 and Figures 1 and 2, the two-stream network contains a second branch with a feature extractor that takes the 3D points in the image coordinate system and determines features for each of the 3D points),
determining an error which decreases when every feature of every point of the point cloud is close to every feature in the feature map at a pixel location corresponding to a projection of this point of the point cloud on the image when the camera is at the initial 6D pose, and which increases otherwise (see section 3.3-3.4, refining the correspondences is performed through the use of linear equation Ap=0 where the correspondence supplies matrix A and p is the solution vector as seen in equation 8 to determine Ap and the associated error. The correspondences (x.sub.i, y.sub.j) represent the association between the 3D points and its 2D projection. The correspondence may be retrieved for refining through i*=argmin.sub.i ||f.sub.x.sub.i-f.sub.y.sub.j||2 [where f.sub.x.sub.i is a feature of the features determine for each 3D point and f.sub.y.sub.j is a feature of the feature vector for each 2D point] to minimize the distance between features of the feature vectors of the 2D and 3D points returning the nearest 3D neighbor for each 2D point. Additionally, a list of prioritized matches can be used to sort the increasing distances of the correspondences and the solution associated with the smallest eigenvalue is chosen which is interpreted as a smaller error corresponding to a smaller distance as the aim of section 3.3 is minimize the distance between features and prioritize the associated correspondence to obtain the solution vector),
minimizing the determined error for the 6D pose to obtain an intermediary 6D pose (see section 3.4, the use of RANSAC with a P3P solver for estimating a camera pose [R and T] of the camera using the eigenvector with the smallest eigenvalue [minimization of the eigenvalue, thus minimizing the error associated with AP. The use of RANSAC as an iterative algorithm [section F implementation details] is interpreted as producing an intermediary pose in all iteration except the first and last),
determining the 6D pose to be used in a subsequent iteration as the initial 6D pose or to be used as the 6D pose of the camera using a damping factor (see section 3.4 and 4.2, estimating the camera pose by taking the SVD of A transposed diag(w)A, the state-of the art SoftPOSIT is initiated using a prior pose).
Liu does not teach obtain one feature map having the same resolution as the image.
Bork teaches obtain one feature map having the same resolution as the image (see paragraph 0047, the use of a full resolution original image with a full resolution feature map generator).
Liu and Bork are analogous art because they are from the same field of endeavor of a method for determining 6D pose of a camera using point clouds and feature point with a matching process
Before the effective filling date of the invention, it would have been obvious to one of ordinary skill in the art to modify Liu to include obtaining a feature map of the same resolution as taught by Bork. The motivation for doing so would have been to avoid loss of spatial resolution. (Bork, paragraphs 0047).
Regarding claim 4, Liu and Bork teach the method according to claim 1.
Liu teaches minimizing the determined error comprises using the Levenberg-Marquardt algorithm (see section 3.4 RANSAC and Nonlinear Optimization, the refinement of pose can include the optimization of error using Levenberg-Marquardt).
Regarding claim 6, Liu and Bork teach the method according to claim 1.
Liu teaches determining the 6D pose of a camera used to acquire each image of a plurality of (See Abstract and section 4 Megadepth [20], the use of the model to estimate the 6D pose of the camera through the use of 3D scene points while using the photo dataset of multiple landmark scenes. See Algorithm 1 and Section 4.2, the final pose is obtained by RANSAC [which is terminated at 30s] using the determined Weight matrix (determined at the end of iterations of Sinkhorn), the termination of the P3P-RANSAC marks output of the final pose. See section 4.2 it is known in the art to initiate the process using a prior pose as seen in state-of-the-art SoftPOSIT).
Liu does not teach the use of consecutive images of a scene and the use of the final pose as the initial pose for N consecutive image.
Bork teaches determining the 6D pose of a camera used to acquire each image of a plurality of consecutive images of scenes (see paragraph 0029, the pose of the camera may be determined based on images by a camera, which may include a video stream [interpreted as plurality of consecutive images of a scene), and
the use of the final pose as the initial pose for N consecutive image (0044, the trained models may be used to determine the initial pose, the initial pose may be used with the images [see figure 2 input of a video stream 202 (containing N consecutive images)] to register the images and later aid in determining camera pose).
Regarding claim 8, Liu and Bork teach the method according to claim 1.
Liu teaches a preliminary joint training phase of the image processing neural network and of the point cloud processing neural network.
Claim 9 is analogous system to the method claim 1, and thus is similarly analyzed and rejected as claim 1.
Regarding claim 11, Liu and Bork teach the method according to claim 1.
Liu teaches a non-transitory storage medium storing instructions that are executable by one or more processors and that cause the one or more processors to perform functions of the method according to claim 1 (Section 4 Implementation details: the model is trained on a NVIDIA Tesla P40 GPU and uses algorithms).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Liu in view of Bork in view of Wang CN111080627 (hereinafter “Wang”).
Regarding claim 5, Liu and Bork teach the method according to claim 1.
Wang teaches after all the iterations have been carried out to obtain a final 6D pose of the camera, assigning a colour to every point of the point cloud projectable on the image when the camera is at the final 6D pose using the colour information of the image (see page 12 S4-S5, after estimating the pose of the camera mapping the 3D point cloud to the image space according to the camera pose and then assigning color information of the 2d characteristics points to the corresponding 3D characteristic points.
PNG
media_image1.png
436
934
media_image1.png
Greyscale
Wang, Liu and Bork are analogous art because they are from the same field of endeavor of a method for determining 6D pose of a camera using point clouds and feature point with a matching process.
Before the effective filling date of the invention, it would have been obvious to one of ordinary skill in the art to modify Liu and Bork to assign color information to the point cloud as taught by Wang. The motivation for doing so would have been to allow for texture information and semantic segmentation. (Wang, page 12).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Liu in view of Bork in view of Yin CN112837383 (hereinafter “Yin”).
Regarding claim 7, Liu and Bork teach the method according to claim 1.
Liu teaches providing (see section 3.4, the use of ground truth rotation and translation [reference pose]) (see section 3.4, the use of ground truth rotation and translation [reference pose] to calculate a pose lose compared to the estimated camera pose [The use of RANSAC is iterative, and the use of loss can be performed after the estimation of the camera pose thus is performed after the final iteration, section 4 Implementation details]. This pose loss can then be minimized [optimize] to adjust R).
Liu nor Bork teach providing a reference image showing a reference scene, a reference point cloud of the reference scene, and a corresponding reference pose of the camera used to acquire the reference image.
Yin teaches providing a reference image showing a reference scene, a reference point cloud of the reference scene, and a corresponding reference pose of the camera used to acquire the reference image (see page 7 step 101, a reference camera image and a reference point cloud are obtained at a reference position [a fixed location or scene, page 8 paragraph 4] when the relative pose [reference pose] is unchanged), wherein the point cloud of the scene overlaps the point cloud of the reference point cloud of the reference scene (see page 9 substep 1032-1033, establishing point pairs between the reference point cloud and the observation image based on point clouds, with the use of homography matrix between the two [the use of the homography matrix is interpreted as the reference and observation point clouds containing overlap).
Yin, Liu and Bork are analogous art because they are from the same field of endeavor of a method for determining a pose of a sensor using point clouds and feature point with a matching process.
Before the effective filling date of the invention, it would have been obvious to one of ordinary skill in the art to modify Liu and Bork to use reference images, point clouds, and scenes as taught by Yin. The motivation for doing so would have been to estimate the motion of the camera and recalibrate the system. (Yin, page 9-10).
Conclusion
No claims are allowed. Claim 2-3 are free of prior art rejection. Final allowability is contingent on overcoming all rejections presented in this office action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Please see the attached 892 notice of reference cited.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EMILY R. HAUK whose telephone number is (571)272-5966. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at 571-272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMILY ROSE HAUK/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights