DETAILED ACTION
Notice of Pre-AIA or AIA Status
Claims 1-30 are pending in this application. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 3-6, 14-18, 27, and 29 are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Sarlin et al. (US PGPub US2021/0150252 A1, hereby referred to as “Sarlin”).
Consider Claims 1 and 27.
Sarlin teaches:
1. An apparatus for matching keypoints between images, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: / 27. An apparatus for matching keypoints between images, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: (Sarlin: abstract, [0004] This document describes certain aspects of what may be termed “the deep middle-end matcher”, a neural network configured to match two sets of local features by jointly finding correspondences and rejecting non-matchable points. Such a neural network configuration may be utilized in association with spatial computing resources such as those illustrated in FIG. 8, including but not limited to the camera and processing resources that comprise such spatial computing systems. Within the deep middle-end matcher type of configuration, assignments may be estimated by solving an optimal transport problem, whose costs are predicted by a graph neural network. We describe a flexible context aggregation mechanism based on attention, which enables the deep middle-end matcher configuration to reason about the underlying 3D scene and feature assignments jointly. Compared to traditional, hand-designed heuristics, our technique learns priors over geometric transformations and regularities of the 3D world through end-to-end training from images to correspondences. The deep middle-end matcher outperforms other learned approaches and sets a new state-of-the-art on the task of pose estimation in challenging real-world indoor and outdoor environments. These methods and configurations match in real-time on a modern graphical processing unit (“GPU”), and can be readily integrated into modern structure-from-motion (“SfM”) or simultaneous localization and mapping (“SLAM”) systems, all of which may be incorporated into systems such as that illustrated in FIG. 8. [0005] The invention provides a computer system including a computer-readable medium, a processor connected to the computer-readable medium and a set of instructions on the computer-readable medium. The set of instructions may include a deep middle-end matcher architecture that may include an attentional graph neural network having a keypoint encoder to map keypoint positions p and their visual descriptors d into a single vector, and alternating self- and cross-attention layers that, based on the vector, repeated L times to create representations f; and an optimal matching layer that creates an M by N score matrix from the representations f and finds an optimal partial assignment based on the M by N score matrix.)
1. receive first descriptors of first keypoints of a first image; / 27. receive transformed first descriptors of first keypoints of a first image, the first image captured from a first viewing angle, the transformed first descriptors related to a second viewing angle; (Sarlin: [0005] The invention provides a computer system including a computer-readable medium, a processor connected to the computer-readable medium and a set of instructions on the computer-readable medium. The set of instructions may include a deep middle-end matcher architecture that may include an attentional graph neural network having a keypoint encoder to map keypoint positions p and their visual descriptors d into a single vector, and alternating self- and cross-attention layers that, based on the vector, repeated L times to create representations f; and an optimal matching layer that creates an M by N score matrix from the representations f and finds an optimal partial assignment based on the M by N score matrix. [0006] The computer system may further include that in the keypoint encoder, an initial representation (0)xi, for each keypoint i combines visual appearance and location, with the respective keypoint position embedded into a high-dimensional vector with a Multilayer Perceptron (MLP) as follows:
PNG
media_image1.png
35
161
media_image1.png
Greyscale
)
1. transform the first descriptors to obtain transformed first descriptors; and / 27. obtain a second image captured from the second viewing angle; (Sarlin: [0033] In this work, learning feature matching is viewed as finding the partial assignment between two sets of local features. We revisit the classical graph-based strategy of matching by solving a linear assignment problem, which, when relaxed to an optimal transport problem, can be solved differentiably [See references 50, 9, 31 below]. The cost function of this optimization is predicted by a Graph Neural Network (GNN). Inspired by the success of the Transformer [see reference 48 below], it uses self- (intra-image) and cross- (inter-image) attention to leverage both spatial relationships of the keypoints and their visual appearance. This formulation enforces the assignment structure of the predictions while enabling the cost to learn complex priors, elegantly handling occlusion and non-repeatable keypoints. Our method is trained end-to-end from images to correspondences—we learn priors for pose estimation from a large annotated dataset, enabling the deep middle-end matcher to reason about the 3D scene and the assignment. Our work can be applied to a variety of multiple-view geometry problems that require high-quality feature correspondences.)
1. determine second keypoints of a second image based on the transformed first descriptors, wherein the second keypoints match the first keypoints. / 27. and determine second keypoints of the second image based on the transformed first descriptors, wherein the second keypoints match the first keypoints. (Sarlin: [0053] Our formulation provides maximum flexibility as the network can learn to focus on a subset of key-points based on specific attributes. In FIG. 4, masks αij are shown as rays. Attentional aggregation builds a dynamic graph between keypoints. Self-attention (top) can attend anywhere in the same image, e.g., distinctive locations, and is thus not restricted to nearby locations. Cross-attention (bottom) attends to locations in the other image, such as potential matches that have a similar local appearance. The deep middle-end matcher can retrieve, or attend based on both appearance and keypoint location as they are encoded in the representation xi. This includes attending to a nearby keypoint and retrieving the relative positions of similar or salient keypoints. This enables representations of the geometric transformation and the assignment. The final matching descriptors are linear projections:
PNG
media_image2.png
29
286
media_image2.png
Greyscale
and similarly for keypoints in B. 3.2. Optimal Matching Layer [0054] The second major block of the deep middle-end matcher (see Section 3b) is the optimal matching layer, which produces a partial assignment matrix. As in the standard graph matching formulation, the assignment P can be obtained by computing a score matrix S∈RM×N for all possible matches and maximizing the total score Σi,j)
Consider Claims 14.
Sarlin teaches:
14. An apparatus for sharing image data for matching of keypoints between images, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: (Sarlin: abstract, [0004] This document describes certain aspects of what may be termed “the deep middle-end matcher”, a neural network configured to match two sets of local features by jointly finding correspondences and rejecting non-matchable points. Such a neural network configuration may be utilized in association with spatial computing resources such as those illustrated in FIG. 8, including but not limited to the camera and processing resources that comprise such spatial computing systems. Within the deep middle-end matcher type of configuration, assignments may be estimated by solving an optimal transport problem, whose costs are predicted by a graph neural network. We describe a flexible context aggregation mechanism based on attention, which enables the deep middle-end matcher configuration to reason about the underlying 3D scene and feature assignments jointly. Compared to traditional, hand-designed heuristics, our technique learns priors over geometric transformations and regularities of the 3D world through end-to-end training from images to correspondences. The deep middle-end matcher outperforms other learned approaches and sets a new state-of-the-art on the task of pose estimation in challenging real-world indoor and outdoor environments. These methods and configurations match in real-time on a modern graphical processing unit (“GPU”), and can be readily integrated into modern structure-from-motion (“SfM”) or simultaneous localization and mapping (“SLAM”) systems, all of which may be incorporated into systems such as that illustrated in FIG. 8. [0005] The invention provides a computer system including a computer-readable medium, a processor connected to the computer-readable medium and a set of instructions on the computer-readable medium. The set of instructions may include a deep middle-end matcher architecture that may include an attentional graph neural network having a keypoint encoder to map keypoint positions p and their visual descriptors d into a single vector, and alternating self- and cross-attention layers that, based on the vector, repeated L times to create representations f; and an optimal matching layer that creates an M by N score matrix from the representations f and finds an optimal partial assignment based on the M by N score matrix.)
14. obtain a first image of a scene captured from a first viewing angle; (Sarlin: [0005] The invention provides a computer system including a computer-readable medium, a processor connected to the computer-readable medium and a set of instructions on the computer-readable medium. The set of instructions may include a deep middle-end matcher architecture that may include an attentional graph neural network having a keypoint encoder to map keypoint positions p and their visual descriptors d into a single vector, and alternating self- and cross-attention layers that, based on the vector, repeated L times to create representations f; and an optimal matching layer that creates an M by N score matrix from the representations f and finds an optimal partial assignment based on the M by N score matrix. [0006] The computer system may further include that in the keypoint encoder, an initial representation (0)xi, for each keypoint i combines visual appearance and location, with the respective keypoint position embedded into a high-dimensional vector with a Multilayer Perceptron (MLP) as follows:
PNG
media_image1.png
35
161
media_image1.png
Greyscale
)
14.generate first descriptors of first keypoints of the first image; (Sarlin: [0033] In this work, learning feature matching is viewed as finding the partial assignment between two sets of local features. We revisit the classical graph-based strategy of matching by solving a linear assignment problem, which, when relaxed to an optimal transport problem, can be solved differentiably [See references 50, 9, 31 below]. The cost function of this optimization is predicted by a Graph Neural Network (GNN). Inspired by the success of the Transformer [see reference 48 below], it uses self- (intra-image) and cross- (inter-image) attention to leverage both spatial relationships of the keypoints and their visual appearance. This formulation enforces the assignment structure of the predictions while enabling the cost to learn complex priors, elegantly handling occlusion and non-repeatable keypoints. Our method is trained end-to-end from images to correspondences—we learn priors for pose estimation from a large annotated dataset, enabling the deep middle-end matcher to reason about the 3D scene and the assignment. Our work can be applied to a variety of multiple-view geometry problems that require high-quality feature correspondences.)
14.transform the first descriptors to obtain transformed first descriptors based on a second viewing angle of the scene; and transmit the transformed first descriptors. (Sarlin: [0053] Our formulation provides maximum flexibility as the network can learn to focus on a subset of key-points based on specific attributes. In FIG. 4, masks αij are shown as rays. Attentional aggregation builds a dynamic graph between keypoints. Self-attention (top) can attend anywhere in the same image, e.g., distinctive locations, and is thus not restricted to nearby locations. Cross-attention (bottom) attends to locations in the other image, such as potential matches that have a similar local appearance. The deep middle-end matcher can retrieve, or attend based on both appearance and keypoint location as they are encoded in the representation xi. This includes attending to a nearby keypoint and retrieving the relative positions of similar or salient keypoints. This enables representations of the geometric transformation and the assignment. The final matching descriptors are linear projections:
PNG
media_image2.png
29
286
media_image2.png
Greyscale
and similarly for keypoints in B. 3.2. Optimal Matching Layer [0054] The second major block of the deep middle-end matcher (see Section 3b) is the optimal matching layer, which produces a partial assignment matrix. As in the standard graph matching formulation, the assignment P can be obtained by computing a score matrix S∈RM×N for all possible matches and maximizing the total score Σi,j)
Consider Claims 2 and 28.
Sarlin teaches:
2. The apparatus of claim 1, wherein the at least one processor is further configured to determine a relative pose between a first viewing angle of the first image and a second viewing angle of the second image based on the first keypoints and the second keypoints. / 28. The apparatus of claim 27, wherein the at least one processor is further configured to determine a relative pose between a first viewing angle of the first image and a second viewing angle of the second image based on the first keypoints and the second keypoints. (Sarlin: [0058] Sinkhorn Algorithm: The solution of the above optimization problem corresponds to the optimal transport [see reference 31 below] between discrete distributions a and b with score S−. It can be approximately solved with the Sinkhorn algorithm [see references 43, 9 below], a differentiable version of the Hungarian algorithm [see reference 28 below], classically used for bipartite matching. It solves a regularized transport problem, naturally resulting in a soft assignment. This normalization amounts to iteratively performing alternating Softmax along rows and columns, and is thus easily parallelized on GPU. After T iterations, we drop the dustbins and recover P=P−1:M,1:N . 3.3. Loss [0059] By design, both the graph neural network and the optimal matching layer are differentiable—this enables backpropagation from matches to visual descriptors. The deep middle-end matcher is trained in a supervised manner from ground truth matches M={(i, j)}⊂A×B. These are estimated from ground truth relative transformations—using poses and depth maps or homographies. This also lets us label some keypoints I⊆A and J⊆B as unmatched if they do not have any reprojection in their vicinity. Given the labels, we minimize the negative log-likelihood of the assignment P−:
PNG
media_image3.png
76
388
media_image3.png
Greyscale
)
Consider Claims 15 and 29.
Sarlin teaches:
15. The apparatus of claim 14, wherein the at least one processor is further configured to: transform the first descriptors to obtain a plurality of transformed first descriptors based on a respective plurality of viewing angles of the scene; and transmit the plurality of transformed first descriptors. / 29. The apparatus of claim 27, wherein the at least one processor is further configured to: receive a plurality of transformed descriptors including the transformed first descriptors; and compare each transformed descriptor of the plurality of transformed descriptors to the second image to determine the transformed first descriptors. (Sarlin: [0033] In this work, learning feature matching is viewed as finding the partial assignment between two sets of local features. We revisit the classical graph-based strategy of matching by solving a linear assignment problem, which, when relaxed to an optimal transport problem, can be solved differentiably [See references 50, 9, 31 below]. The cost function of this optimization is predicted by a Graph Neural Network (GNN). Inspired by the success of the Transformer [see reference 48 below], it uses self- (intra-image) and cross- (inter-image) attention to leverage both spatial relationships of the keypoints and their visual appearance. This formulation enforces the assignment structure of the predictions while enabling the cost to learn complex priors, elegantly handling occlusion and non-repeatable keypoints. Our method is trained end-to-end from images to correspondences—we learn priors for pose estimation from a large annotated dataset, enabling the deep middle-end matcher to reason about the 3D scene and the assignment. Our work can be applied to a variety of multiple-view geometry problems that require high-quality feature correspondences.)
Consider Claims 3 and 16.
Sarlin teaches:
3. The apparatus of claim 1, wherein the first descriptors are transformed using a mapping function. / 16. The apparatus of claim 15, wherein the first descriptors are transformed to obtain the plurality of transformed first descriptors using a plurality of mapping functions. (Sarlin: [0004] The deep middle-end matcher outperforms other learned approaches and sets a new state-of-the-art on the task of pose estimation in challenging real-world indoor and outdoor environments. These methods and configurations match in real-time on a modern graphical processing unit (“GPU”), and can be readily integrated into modern structure-from-motion (“SfM”) or simultaneous localization and mapping (“SLAM”) systems, all of which may be incorporated into systems such as that illustrated in FIG. 8. [0020]-[0021] The invention also provides a computer-implemented method system that may include mapping, with a keypoint encoder of an attentional graph neural network of a deep middle-end matcher architecture, keypoint positions p and their visual descriptors d into a single vector; and executing, with alternating self- and cross-attention layers of an attentional graph neural network of the deep middle-end matcher architecture, based on the vector, for L repeated times, to create representations f, and executing an optimal matching layer, of the attentional graph neural network of the deep middle-end matcher architecture, to create an M by N score matrix from the representations f and finding an optimal partial assignment based on the M by N score matrix.)
Consider Claims 4 and 17.
Sarlin teaches:
4. The apparatus of claim 3, wherein the mapping function comprises a neural network trained to transform first-viewing-angle descriptors into second-viewing-angle descriptors, and wherein the transformed first descriptors are viewing-angle descriptors. / 17. The apparatus of claim 16, wherein each mapping function of the plurality of mapping functions comprises a neural network trained to transform first-viewing-angle descriptors into second-viewing-angle descriptors, and wherein the transformed first descriptors are viewing-angle descriptors. (Sarlin: [0004] The deep middle-end matcher outperforms other learned approaches and sets a new state-of-the-art on the task of pose estimation in challenging real-world indoor and outdoor environments. These methods and configurations match in real-time on a modern graphical processing unit (“GPU”), and can be readily integrated into modern structure-from-motion (“SfM”) or simultaneous localization and mapping (“SLAM”) systems, all of which may be incorporated into systems such as that illustrated in FIG. 8. [0020]-[0021] The invention also provides a computer-implemented method system that may include mapping, with a keypoint encoder of an attentional graph neural network of a deep middle-end matcher architecture, keypoint positions p and their visual descriptors d into a single vector; and executing, with alternating self- and cross-attention layers of an attentional graph neural network of the deep middle-end matcher architecture, based on the vector, for L repeated times, to create representations f, and executing an optimal matching layer, of the attentional graph neural network of the deep middle-end matcher architecture, to create an M by N score matrix from the representations f and finding an optimal partial assignment based on the M by N score matrix.)
Consider Claims 5 and 18.
Sarlin teaches:
5. The apparatus of claim 3, wherein the at least one processor is further configured to select the mapping function from among a plurality of mapping functions. / 18. The apparatus of claim 14, wherein the first descriptors are transformed using a mapping function. (Sarlin: [0004] The deep middle-end matcher outperforms other learned approaches and sets a new state-of-the-art on the task of pose estimation in challenging real-world indoor and outdoor environments. These methods and configurations match in real-time on a modern graphical processing unit (“GPU”), and can be readily integrated into modern structure-from-motion (“SfM”) or simultaneous localization and mapping (“SLAM”) systems, all of which may be incorporated into systems such as that illustrated in FIG. 8. [0020]-[0021] The invention also provides a computer-implemented method system that may include mapping, with a keypoint encoder of an attentional graph neural network of a deep middle-end matcher architecture, keypoint positions p and their visual descriptors d into a single vector; and executing, with alternating self- and cross-attention layers of an attentional graph neural network of the deep middle-end matcher architecture, based on the vector, for L repeated times, to create representations f, and executing an optimal matching layer, of the attentional graph neural network of the deep middle-end matcher architecture, to create an M by N score matrix from the representations f and finding an optimal partial assignment based on the M by N score matrix.)
Consider Claims 6.
Sarlin teaches:
6. The apparatus of claim 5, wherein the plurality of mapping functions are stored locally at the apparatus. (Sarlin: [0004] The deep middle-end matcher outperforms other learned approaches and sets a new state-of-the-art on the task of pose estimation in challenging real-world indoor and outdoor environments. These methods and configurations match in real-time on a modern graphical processing unit (“GPU”), and can be readily integrated into modern structure-from-motion (“SfM”) or simultaneous localization and mapping (“SLAM”) systems, all of which may be incorporated into systems such as that illustrated in FIG. 8. [0020]-[0021] The invention also provides a computer-implemented method system that may include mapping, with a keypoint encoder of an attentional graph neural network of a deep middle-end matcher architecture, keypoint positions p and their visual descriptors d into a single vector; and executing, with alternating self- and cross-attention layers of an attentional graph neural network of the deep middle-end matcher architecture, based on the vector, for L repeated times, to create representations f, and executing an optimal matching layer, of the attentional graph neural network of the deep middle-end matcher architecture, to create an M by N score matrix from the representations f and finding an optimal partial assignment based on the M by N score matrix.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1, 7-12, 14, 19-21, 23, 25, 27 and 30 are rejected under 35 U.S.C. 103 as being unpatentable over Sarlin et al. (US PGPub US2021/0150252 A1, hereby referred to as “Sarlin”), in view of Trenholm et al. (US PGPub 20190138786), hereby referred to as “Trenholm”.
Consider Claims 1, 14 and 27.
Sarlin teaches:
1. An apparatus for matching keypoints between images, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: / 27. An apparatus for matching keypoints between images, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: / 14. An apparatus for sharing image data for matching of keypoints between images, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: (Sarlin: abstract, [0004] This document describes certain aspects of what may be termed “the deep middle-end matcher”, a neural network configured to match two sets of local features by jointly finding correspondences and rejecting non-matchable points. Such a neural network configuration may be utilized in association with spatial computing resources such as those illustrated in FIG. 8, including but not limited to the camera and processing resources that comprise such spatial computing systems. Within the deep middle-end matcher type of configuration, assignments may be estimated by solving an optimal transport problem, whose costs are predicted by a graph neural network. We describe a flexible context aggregation mechanism based on attention, which enables the deep middle-end matcher configuration to reason about the underlying 3D scene and feature assignments jointly. Compared to traditional, hand-designed heuristics, our technique learns priors over geometric transformations and regularities of the 3D world through end-to-end training from images to correspondences. The deep middle-end matcher outperforms other learned approaches and sets a new state-of-the-art on the task of pose estimation in challenging real-world indoor and outdoor environments. These methods and configurations match in real-time on a modern graphical processing unit (“GPU”), and can be readily integrated into modern structure-from-motion (“SfM”) or simultaneous localization and mapping (“SLAM”) systems, all of which may be incorporated into systems such as that illustrated in FIG. 8. [0005] The invention provides a computer system including a computer-readable medium, a processor connected to the computer-readable medium and a set of instructions on the computer-readable medium. The set of instructions may include a deep middle-end matcher architecture that may include an attentional graph neural network having a keypoint encoder to map keypoint positions p and their visual descriptors d into a single vector, and alternating self- and cross-attention layers that, based on the vector, repeated L times to create representations f; and an optimal matching layer that creates an M by N score matrix from the representations f and finds an optimal partial assignment based on the M by N score matrix.)
1. receive first descriptors of first keypoints of a first image; / 27. receive transformed first descriptors of first keypoints of a first image, the first image captured from a first viewing angle, the transformed first descriptors related to a second viewing angle; / 14. obtain a first image of a scene captured from a first viewing angle; (Sarlin: [0005] The invention provides a computer system including a computer-readable medium, a processor connected to the computer-readable medium and a set of instructions on the computer-readable medium. The set of instructions may include a deep middle-end matcher architecture that may include an attentional graph neural network having a keypoint encoder to map keypoint positions p and their visual descriptors d into a single vector, and alternating self- and cross-attention layers that, based on the vector, repeated L times to create representations f; and an optimal matching layer that creates an M by N score matrix from the representations f and finds an optimal partial assignment based on the M by N score matrix. [0006] The computer system may further include that in the keypoint encoder, an initial representation (0)xi, for each keypoint i combines visual appearance and location, with the respective keypoint position embedded into a high-dimensional vector with a Multilayer Perceptron (MLP) as follows:
PNG
media_image1.png
35
161
media_image1.png
Greyscale
)
1. transform the first descriptors to obtain transformed first descriptors; and / 27. obtain a second image captured from the second viewing angle; / 14.generate first descriptors of first keypoints of the first image; (Sarlin: [0033] In this work, learning feature matching is viewed as finding the partial assignment between two sets of local features. We revisit the classical graph-based strategy of matching by solving a linear assignment problem, which, when relaxed to an optimal transport problem, can be solved differentiably [See references 50, 9, 31 below]. The cost function of this optimization is predicted by a Graph Neural Network (GNN). Inspired by the success of the Transformer [see reference 48 below], it uses self- (intra-image) and cross- (inter-image) attention to leverage both spatial relationships of the keypoints and their visual appearance. This formulation enforces the assignment structure of the predictions while enabling the cost to learn complex priors, elegantly handling occlusion and non-repeatable keypoints. Our method is trained end-to-end from images to correspondences—we learn priors for pose estimation from a large annotated dataset, enabling the deep middle-end matcher to reason about the 3D scene and the assignment. Our work can be applied to a variety of multiple-view geometry problems that require high-quality feature correspondences.)
1. determine second keypoints of a second image based on the transformed first descriptors, wherein the second keypoints match the first keypoints. / 27. and determine second keypoints of the second image based on the transformed first descriptors, wherein the second keypoints match the first keypoints. / 14.transform the first descriptors to obtain transformed first descriptors based on a second viewing angle of the scene; and transmit the transformed first descriptors. (Sarlin: [0053] Our formulation provides maximum flexibility as the network can learn to focus on a subset of key-points based on specific attributes. In FIG. 4, masks αij are shown as rays. Attentional aggregation builds a dynamic graph between keypoints. Self-attention (top) can attend anywhere in the same image, e.g., distinctive locations, and is thus not restricted to nearby locations. Cross-attention (bottom) attends to locations in the other image, such as potential matches that have a similar local appearance. The deep middle-end matcher can retrieve, or attend based on both appearance and keypoint location as they are encoded in the representation xi. This includes attending to a nearby keypoint and retrieving the relative positions of similar or salient keypoints. This enables representations of the geometric transformation and the assignment. The final matching descriptors are linear projections:
PNG
media_image2.png
29
286
media_image2.png
Greyscale
and similarly for keypoints in B. 3.2. Optimal Matching Layer [0054] The second major block of the deep middle-end matcher (see Section 3b) is the optimal matching layer, which produces a partial assignment matrix. As in the standard graph matching formulation, the assignment P can be obtained by computing a score matrix S∈RM×N for all possible matches and maximizing the total score Σi,j)
5. The apparatus of claim 3, wherein the at least one processor is further configured to select the mapping function from among a plurality of mapping functions. / 18. The apparatus of claim 14, wherein the first descriptors are transformed using a mapping function. (Sarlin: [0004] The deep middle-end matcher outperforms other learned approaches and sets a new state-of-the-art on the task of pose estimation in challenging real-world indoor and outdoor environments. These methods and configurations match in real-time on a modern graphical processing unit (“GPU”), and can be readily integrated into modern structure-from-motion (“SfM”) or simultaneous localization and mapping (“SLAM”) systems, all of which may be incorporated into systems such as that illustrated in FIG. 8. [0020]-[0021] The invention also provides a computer-implemented method system that may include mapping, with a keypoint encoder of an attentional graph neural network of a deep middle-end matcher architecture, keypoint positions p and their visual descriptors d into a single vector; and executing, with alternating self- and cross-attention layers of an attentional graph neural network of the deep middle-end matcher architecture, based on the vector, for L repeated times, to create representations f, and executing an optimal matching layer, of the attentional graph neural network of the deep middle-end matcher architecture, to create an M by N score matrix from the representations f and finding an optimal partial assignment based on the M by N score matrix.)
Sarlin does not teach limitations from dependent claims 7 and 19: request the mapping function from the server or wherein the mapping function is selected based on a coarse relative pose between a first viewing angle of the first image and a second viewing angle of the second image.
Trenholm teaches:
1. An apparatus for matching keypoints between images, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: / 27. An apparatus for matching keypoints between images, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: / 14. An apparatus for sharing image data for matching of keypoints between images, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: (Trenholm: abstract, A method and system for analysis of an object of interest in a scene using 3D reconstruction. The method includes: receiving image data comprising a plurality of images captured of the scene, the image data comprising multiple perspectives of the scene; generating at least one reconstructed image by determining three-dimensional structures of the object from the imaging data using a reconstruction technique, the three-dimensional structures comprising depth information of the object; identifying the object from each of the reconstructed images, using a trained machine learning model, by segmenting the object in the reconstructed image, segmentation comprises isolating patterns in the reconstructed image that are classifiable as the object, the machine learning model trained using previous reconstructed multiple perspective images with identified objects; and outputting the analysis of the reconstructed images.)
1. receive first descriptors of first keypoints of a first image; / 27. receive transformed first descriptors of first keypoints of a first image, the first image captured from a first viewing angle, the transformed first descriptors related to a second viewing angle; / 14. obtain a first image of a scene captured from a first viewing angle; (Trenholm: [0047] Image device 102 may be configured to acquire multiple images of a scene (to provide depth information), the scene containing an object of interest that a user wants to identify and classify. In some embodiments, the user may want to detect and potentially categorize a defect in an object. The multiple images may be acquired from multiple angles or perspectives in the hope of avoiding issues relating to occlusion, which may hinder the effectiveness of an object identification and classification process. In some embodiments, multiple image devices 102 are used to acquire multiple images from multiple perspectives; for example, image device 102-1 acquires an image from a first perspective, image device 102-2 acquires an image from a second perspective, and image device 102-3 acquires an image from a third perspective. The image device 102 can be configured to acquire two-dimensional (2D) or three-dimensional (3D) images of the scene. Image device 102 may acquire images continuously, such as video, by using a video camera or a camera in burst mode. While multiple images need not be taken in immediate succession, it may be desired so as to limit any significant degradation or alteration to the form of the object of interest (e.g. a rapidly melting ice sculpture). Severe degradation to the object of interest or alteration to its form before multiple images are acquired by image device 102 may prevent effective feature mapping between images acquired from adjacent viewpoints. This may produce imperfect 3D reconstruction of the object, which can further result in imperfect segmentation, identification and classification of the object of interest. In variations, image device 102 may be a video camera, single lens reflex camera, point and shoot camera, embedded camera (such as in a mobile device such as a cell phone or tablet computer), or other suitable device for acquiring 2D or 3D image data (e.g. lidar), or some combination thereof.)
1. transform the first descriptors to obtain transformed first descriptors; and / 27. obtain a second image captured from the second viewing angle; / 14.generate first descriptors of first keypoints of the first image; (Trenholm: [0053] AI module 118 may comprise a neural network trained on reconstructed multi-orientation 3D representations or multi-view 2D projections. The trained neural network can be used to identify and classify objects in their (i) reconstructed multi-orientation 3D representations, or (ii) multi-view 2D projections, or (iii) static 2D images. In some embodiments, the static 2D images comprise the input 2D images used to reconstruct the 3D image using SFM or other technique. [0054] The AI module 118 can use machine learning (ML) to transform raw data from a reconstructed 3D image of the scene into a descriptor. The descriptor may be information associated with a particular type of object (e.g. object identification/classification) or particular defect in the object. The descriptor can then be used by the AI module 118 to determine a classifier for the object or defect. As an example, the AI module 118 can do this detection and classification with auto-encoders as part of a deep belief network. In this sense, ML can be used as part of a feature descriptor extraction process, otherwise called “feature learning”. In some cases, the AI module can perform the machine learning remotely over a network to a system located elsewhere.)
1. determine second keypoints of a second image based on the transformed first descriptors, wherein the second keypoints match the first keypoints. / 27. and determine second keypoints of the second image based on the transformed first descriptors, wherein the second keypoints match the first keypoints. / 14.transform the first descriptors to obtain transformed first descriptors based on a second viewing angle of the scene; and transmit the transformed first descriptors. (Trenholm: [0061] Feature detection may utilize binary descriptors; for example, binary robust independent elementary features (BRIEF), binary robust invariant scalable keypoints (BRISK), oriented fast and rotated BRIEF (ORB), Accelerated KAZE (AKAZE), fast retina keypoint (FREAK), or other techniques. At block 404, the features can be mapped across the image set in order to reconstruct the 3D image. One or more feature descriptor techniques may be used to determine which key points in various images in the image set are 2D representations of the same 3D point; for example, gradient location and orientation histogram (GLOH), speeded-up robust features (SURF), scale-invariant feature transform (SIFT), histogram of oriented gradients (HOG), or the like. This can be done by computing a feature vector or feature descriptor with local characteristics to describe a local patch. Feature descriptors are matched between different images in the image set by associating key points from one image to another in the set. At block 405, once all the feature descriptors are matched, a global map of feature visibility among views can be created. [0062] Referring now to FIG. 5, shown therein are the steps of optimization 303 and increasing point cloud density 304 of method 300 in greater detail, in accordance with an embodiment. At block 501, a non-linear optimization step called “bundle adjustment” is performed on the image set to jointly refine relative poses of the image devices 102 and the 3D position of points. This step provides information regarding (i) the location of the image devices 102 and their orientation in a local reference frame, which may be determined with respect to a reference image device, and (ii) where a given image device 102 is with respect to the 3D object—i.e. what was the location and orientation of that particular imaging device to create that particular 2D image on which the features were detected (at step 302))
7. The apparatus of claim 5, wherein the mapping function is selected based on a coarse relative pose between a first viewing angle of the first image and a second viewing angle of the second image. / 19. The apparatus of claim 18, wherein the mapping function comprises a neural network trained to transform first-viewing-angle descriptors into second-viewing-angle descriptors, and wherein the transformed first descriptors are viewing-angle descriptors. (Trenholm: [0051] The system 100 may implement one or more deep learning neural network techniques, such as a convolutional neural network (“CNN”). The system 100 may be used to segment, identify, and classify an object in a 3D image. Segmenting, identifying, and classifying a 3D image may avoid issues associated with applying the same techniques to 2D images, such as occlusion or nearby clutter. For example, in a 2D image, occlusion can occur where the object of interest is blocked or overlapped by another object when viewed from a certain angle; self-occlusion can occur where the object of interest blocks relevant features of itself that are crucial for identification when viewed from a certain angle; and poor lighting such as glare and backlight can result in overexposure or underexposure respectively in certain scenes can lead to missing details thus making the image unusable. System 100 can overcome such disadvantages by adding depth information to a scene through the 3D reconstruction process and generating a 3D image of the object of interest. To do this, image device 102 can be configured to acquire multiple 2D images of a scene from different vantage points in order to acquire necessary depth information to reduce occlusion and cluttering effects.)
It would have been obvious before the effective filing date of the claimed invention was made to one of ordinary skill in the art to modify Sarlin’s method and system for feature matching with the image analysis of Trenholm as they are directed towards the same field of image processing and analysis. The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify Sarlin in order to improve the overall machine learning algorithm to include additional parameters for image analysis for 3D object and reconstruction as disclosed by Trenholm. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Sarlin, while the teaching of Trenholm continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of incorporation of feature parameters for image analysis of 3D object reconstruction. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claims 7 and 19.
Claims 7 and 19 are rejected for the same reasons as above.
The combination of Sarlin and Trenholm teaches:
7. The apparatus of claim 5, wherein the mapping function is selected based on a coarse relative pose between a first viewing angle of the first image and a second viewing angle of the second image. / 19. The apparatus of claim 18, wherein the mapping function comprises a neural network trained to transform first-viewing-angle descriptors into second-viewing-angle descriptors, and wherein the transformed first descriptors are viewing-angle descriptors. (Trenholm: [0051] The system 100 may implement one or more deep learning neural network techniques, such as a convolutional neural network (“CNN”). The system 100 may be used to segment, identify, and classify an object in a 3D image. Segmenting, identifying, and classifying a 3D image may avoid issues associated with applying the same techniques to 2D images, such as occlusion or nearby clutter. For example, in a 2D image, occlusion can occur where the object of interest is blocked or overlapped by another object when viewed from a certain angle; self-occlusion can occur where the object of interest blocks relevant features of itself that are crucial for identification when viewed from a certain angle; and poor lighting such as glare and backlight can result in overexposure or underexposure respectively in certain scenes can lead to missing details thus making the image unusable. System 100 can overcome such disadvantages by adding depth information to a scene through the 3D reconstruction process and generating a 3D image of the object of interest. To do this, image device 102 can be configured to acquire multiple 2D images of a scene from different vantage points in order to acquire necessary depth information to reduce occlusion and cluttering effects.)
Consider Claims 8, 22 and 23.
Claims 8, 22 and 23 are rejected for the same reasons as above.
The combination of Sarlin and Trenholm teaches:
8. The apparatus of claim 3, wherein the mapping function is received from a server. / 22. The apparatus of claim 18, wherein the mapping function is received from a server. / 23. The apparatus of claim 22, wherein the at least one processor is further configured to request the mapping function from the server. (Trenholm: [0054] The AI module 118 can use machine learning (ML) to transform raw data from a reconstructed 3D image of the scene into a descriptor. The descriptor may be information associated with a particular type of object (e.g. object identification/classification) or particular defect in the object. The descriptor can then be used by the AI module 118 to determine a classifier for the object or defect. As an example, the AI module 118 can do this detection and classification with auto-encoders as part of a deep belief network. In this sense, ML can be used as part of a feature descriptor extraction process, otherwise called “feature learning”. In some cases, the AI module can perform the machine learning remotely over a network to a system located elsewhere. [0055] In further embodiments, instead of, or along with, feature learning, “feature engineering” can also be undertaken by the AI module 118 to determine appropriate values for discrimination of distinct classes in the reconstructed 3D image data. The AI module 118 can then use ML to provide a posterior probability distribution for class assignment of the object to the class labels. In some cases, “feature engineering” can include input from a user such as a data scientist or computer vision engineer. [0056] In an embodiment, the AI module 118 can provide class labels of types of objects. For example, in a machine part identification and classification application used as part of a tool inventory verification process, the class labels may comprise tool types. In another embodiment, the AI module can provide class labels of types of defects.)
Consider Claims 9 and 20.
Claims 9 and 20 are rejected for the same reasons as above.
The combination of Sarlin and Trenholm teaches:
9. The apparatus of claim 8, wherein the at least one processor is further configured to request the mapping function from the server. / 20. The apparatus of claim 18, wherein the at least one processor is further configured to select the mapping function from among a plurality of mapping functions. (Trenholm: [0049] In a particular embodiment, image device 102 acquires 2D images, which are combined with local motion signals. 3D structures can then be estimated from the 2D image sequences using a “Structure-from-Motion” (“SFM”) or similar technique. With SFM, the intent is to capture details of a 3D object from as many orientations as possible. For example, image device 102 can acquire images while (i) rotating the object of interest; (ii) moving the object side-to-side; moving the image device 102 side-to-side; moving the image device 102 around the object; or (v) some combination of two or more of (i)-(iv). Reconstruction module 120 is configured to implement an SFM algorithm, which, at a high level, finds correspondence between images by selecting boundaries, corner points, edges and blobs that have gradients in multiple directions and which can be tracked from one image to the next. In some implementations, image device 102 comprises a mobile device, such as a cellular phone, that is equipped with a high resolution digital camera, providing a low-cost mechanism for capturing images for SFM. This can allow for developing mobile device-based client applications that can exploit other features such as an ability to capture and transmit images to cloud servers managing complex enterprise systems. A user can take several images of an object from various angles and send them to a 3D image reconstruction and object identification application (such as by system 100) for processing, thereby performing object classification tasks on the go. Such mobile device cameras may be capable of taking pictures with image metadata in “exchangeable image file format” (EXIF). The EXIF metadata tags can cover a broad spectrum including date and time information, geolocation information, image orientation and rotation, camera settings such as focal length, aperture, shutter speed, metering mode, ISO, etc. The EXIF information can be used for calibrating images acquired by image device 102, which can better facilitate reconstruction of 3D images from the acquired images by the system 100. [0050] In alternative embodiments, reconstruction module 120 can be configured to perform 3D reconstruction according to other techniques. Multiple cameras 102 or a single camera 102 may acquire images from a fixed location or multiple locations while illumination and/or camera characteristics are tweaked. Such techniques may include shape from focus, shape from shading, and stereoscopy.)
Consider Claims 10 and 21.
Claims 10 and 21 are rejected for the same reasons as above.
The combination of Sarlin and Trenholm teaches:
10. The apparatus of claim 9, wherein the mapping function is requested based on a coarse relative pose between a first viewing angle of the first image and a second viewing angle of the second image. / 21. The apparatus of claim 20, wherein the mapping function is selected based on a coarse relative pose between the first viewing angle and the second viewing angle. / 24. The apparatus of claim 23, wherein the mapping function is requested based on a coarse relative pose between a first viewing angle of the first image and the second viewing angle. (Trenholm: [0059] At block 302, feature detection is performed on the image set. In doing so, image processing techniques are applied to an image to detect salient features in the image that have high contrast, such as corners, edges, blobs, and lines. These features have a high likelihood of being easily identifiable in adjacent images for the purpose of feature tracking across multiple images (i.e. such as across an image pair). Feature tracking across multiple images facilitates point correspondence and relative pose estimation, which can be performed incrementally or globally. At block 303, camera poses and 3D points are jointly estimated through an optimization process. The optimization process, known as “bundle adjustment”, seeks to minimize the projection errors of point onto an image given an estimated pose, and by doing so produces a sparse point cloud. At block 304, the sparse point cloud generated from point correspondences is made denser using one or more techniques such as patch matching. At block 305, surface reconstruction is performed on the point cloud (once dense enough). [0060] Referring now to FIGS. 4 to 6, certain steps of method 300 for 3D reconstruction are described in further exemplary detail. The steps of the methods described in FIGS. 4 to 6 may be implemented as part of 3D reconstruction module 120 of system 100. [0061] Turning first to FIG. 4, the steps of pre-processing 301 and feature detection/extraction 302 are shown in greater detail. The steps shown in FIG. 4 correspond to an embodiment wherein image device 102 is a video camera; however, with appropriate modifications image device 102 could be any suitable imaging device. The video camera 102 acquires a video sequence of a scene, the scene containing an object of interest, and the video sequence is provided to system 100. At block 401, the video sequence is decomposed into a plurality of images. At block 402, metadata such as EXIF data/tags are inserted into the plurality of images. Metadata may include resolution, lens information, focal length, sensor width, or the like. Metadata/EXIF data can be used to extrapolate camera calibration in order to perform a 3D spatial alignment procedure, which determines the relative location of the image device 102 in the 3D point cloud being reconstructed and estimates the camera poses (i.e. the position and orientation of the image device 102 when an image was acquired). At block 403, feature detection and description is performed on each image to extract features. Feature detection can be performed by examining key points in an image)
Consider Claims 11 and 25.
Claims 11 and 25 are rejected for the same reasons as above.
The combination of Sarlin and Trenholm teaches:
11. The apparatus of claim 8, wherein the mapping function is selected based on an expected relative pose between a first viewing angle of the first image and a second viewing angle of the second image. / 25. The apparatus of claim 22, wherein the mapping function is selected based on an expected relative pose between a first viewing angle of the first image and the second viewing angle. (Trenholm: [0059] At block 302, feature detection is performed on the image set. In doing so, image processing techniques are applied to an image to detect salient features in the image that have high contrast, such as corners, edges, blobs, and lines. These features have a high likelihood of being easily identifiable in adjacent images for the purpose of feature tracking across multiple images (i.e. such as across an image pair). Feature tracking across multiple images facilitates point correspondence and relative pose estimation, which can be performed incrementally or globally. At block 303, camera poses and 3D points are jointly estimated through an optimization process. The optimization process, known as “bundle adjustment”, seeks to minimize the projection errors of point onto an image given an estimated pose, and by doing so produces a sparse point cloud. At block 304, the sparse point cloud generated from point correspondences is made denser using one or more techniques such as patch matching. At block 305, surface reconstruction is performed on the point cloud (once dense enough). [0060] Referring now to FIGS. 4 to 6, certain steps of method 300 for 3D reconstruction are described in further exemplary detail. The steps of the methods described in FIGS. 4 to 6 may be implemented as part of 3D reconstruction module 120 of system 100. [0061] Turning first to FIG. 4, the steps of pre-processing 301 and feature detection/extraction 302 are shown in greater detail. The steps shown in FIG. 4 correspond to an embodiment wherein image device 102 is a video camera; however, with appropriate modifications image device 102 could be any suitable imaging device. The video camera 102 acquires a video sequence of a scene, the scene containing an object of interest, and the video sequence is provided to system 100. At block 401, the video sequence is decomposed into a plurality of images. At block 402, metadata such as EXIF data/tags are inserted into the plurality of images. Metadata may include resolution, lens information, focal length, sensor width, or the like. Metadata/EXIF data can be used to extrapolate camera calibration in order to perform a 3D spatial alignment procedure, which determines the relative location of the image device 102 in the 3D point cloud being reconstructed and estimates the camera poses (i.e. the position and orientation of the image device 102 when an image was acquired). At block 403, feature detection and description is performed on each image to extract features. Feature detection can be performed by examining key points in an image)
Consider Claims 12 and 30.
Claims 12 and 30 are rejected for the same reasons as above.
The combination of Sarlin and Trenholm teaches:
12. The apparatus of claim 1, wherein the at least one processor is further configured to determine second descriptors of the second keypoints of the second image, wherein the second keypoints are determined based on a comparison between the transformed first descriptors and the second descriptors. / 30. The apparatus of claim 27, wherein the at least one processor is further configured to determine second descriptors of the second keypoints of the second image; wherein the second keypoints are determined based on a comparison between the transformed first descriptors and the second descriptors. (Trenholm: [0059] At block 302, feature detection is performed on the image set. In doing so, image processing techniques are applied to an image to detect salient features in the image that have high contrast, such as corners, edges, blobs, and lines. These features have a high likelihood of being easily identifiable in adjacent images for the purpose of feature tracking across multiple images (i.e. such as across an image pair). Feature tracking across multiple images facilitates point correspondence and relative pose estimation, which can be performed incrementally or globally. At block 303, camera poses and 3D points are jointly estimated through an optimization process. The optimization process, known as “bundle adjustment”, seeks to minimize the projection errors of point onto an image given an estimated pose, and by doing so produces a sparse point cloud. At block 304, the sparse point cloud generated from point correspondences is made denser using one or more techniques such as patch matching. At block 305, surface reconstruction is performed on the point cloud (once dense enough). [0060] Referring now to FIGS. 4 to 6, certain steps of method 300 for 3D reconstruction are described in further exemplary detail. The steps of the methods described in FIGS. 4 to 6 may be implemented as part of 3D reconstruction module 120 of system 100. [0061] Turning first to FIG. 4, the steps of pre-processing 301 and feature detection/extraction 302 are shown in greater detail. The steps shown in FIG. 4 correspond to an embodiment wherein image device 102 is a video camera; however, with appropriate modifications image device 102 could be any suitable imaging device. The video camera 102 acquires a video sequence of a scene, the scene containing an object of interest, and the video sequence is provided to system 100. At block 401, the video sequence is decomposed into a plurality of images. At block 402, metadata such as EXIF data/tags are inserted into the plurality of images. Metadata may include resolution, lens information, focal length, sensor width, or the like. Metadata/EXIF data can be used to extrapolate camera calibration in order to perform a 3D spatial alignment procedure, which determines the relative location of the image device 102 in the 3D point cloud being reconstructed and estimates the camera poses (i.e. the position and orientation of the image device 102 when an image was acquired). At block 403, feature detection and description is performed on each image to extract features. Feature detection can be performed by examining key points in an image)
Claims 13 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Sarlin et al. (US PGPub US2021/0150252 A1, hereby referred to as “Sarlin”), in view of Trenholm et al. (US PGPub 20190138786), hereby referred to as “Trenholm”, further in view of Tang et al. (US PGPub 2021/0319236, hereby referred to as “Tang”).
Consider Claims 13 and 16.
The combination of Sarlin and Trenholm teaches:
The apparatus of Claim 1 and The apparatus of Claim 14, as presented above.
The combination of Sarlin and Trenholm does not teach: elements of claim 13 and 26 for a parat of a vehicle or a roadside camera
Tang teaches:
13. The apparatus of claim 1, wherein the apparatus is part of a vehicle. / 26. The apparatus of claim 14, wherein the apparatus is associated with a vehicle or a roadside camera. (Tang: [0023] FIG. 1 illustrates an example of an ego vehicle 100 (e.g., ego agent) in an environment 150 according to aspects of the present disclosure. As shown in FIG. 1, the ego vehicle 100 is traveling on a road 110. A first vehicle 104 (e.g., other agent) may be ahead of the ego vehicle 100, and a second vehicle 116 may be adjacent to the ego vehicle 100. In this example, the ego vehicle 100 may include a 2D camera 108, such as a 2D RGB camera, and a second sensor 106. The second sensor 106 may be another RGB camera or another type of sensor, such as RADAR and/or ultrasound. Additionally, or alternatively, the ego vehicle 100 may include one or more additional sensors. For example, the additional sensors may be side facing and/or rear facing sensors. [0024] In one configuration, the 2D camera 108 captures a 2D image that includes objects in the 2D camera's 108 field of view 114. The second sensor 106 may generate one or more output streams. The 2D image captured by the 2D camera includes a 2D image of the first vehicle 104, as the first vehicle 104 is in the 2D camera's 108 field of view 114.)
It would have been obvious before the effective filing date of the claimed invention was made to one of ordinary skill in the art to modify the combination of Sarlin and Trenholm for a method and system for feature matching and image analysis and apply it to the field of vehicular imaging as disclosed by Tang as they are directed towards the same field of image processing and analysis. The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the combination of Sarlin and Trenholm in order to improve the overall machine learning algorithm to include additional parameters for image analysis for 3D object and reconstruction and further apply it to the field of vehicular imaging as they also use key-points. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of the combination of Sarlin and Trenholm, while the teaching of Tang continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of incorporation of feature parameters for image analysis of 3D object reconstruction. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Conclusion
The prior art made of record in form PTO-892 and not relied upon is considered pertinent to applicant's disclosure.
PNG
media_image4.png
144
1011
media_image4.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
/Tahmina Ansari/
April 4, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
Prosecution Insights