DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This is in response to Applicant’s Arguments/Remarks filed on February 19th, 2026, which has
been entered and made of record.
Response to Arguments
Claim Rejections - 35 USC § 102/103
Applicant’s arguments regarding the current claim(s) have been fully considered. But, the
arguments/remarks are directed to the claims as amended, and so are believed to be answered
by and therefore moot in view of the new grounds of rejection presented below.
Status of Claims
Claims 1-20 are pending. Claim(s) 1, 14, and 19-20 were amended. No claim(s) were canceled. No new claim(s) were added. Claims 1-20 are considered below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-9, and 11-12 are rejected under 35 U.S.C. 103 as being unpatentable by Shi (US 20240029300 A1) in view of Datta (US 12254548 B1).
Regarding Claim 1, Shi teaches a system comprising:
an image acquisition device mounted to an object, the image acquisition device configured to acquire a query frame of an environment containing the object ([0029]: re-localization of a robot, [0032] At operation 220, the re-localization apparatus may extract image features of a current frame captured by the robot);
a memory device configured to store an image database ([0022]: the apparatus may receive Red Green Blue Depth (RGBD) images from a visual system of the robot and acquire keyframes from the RGBD images. Then the apparatus may perform a feature extraction process for each keyframe to extract image features of the keyframe and then save the image features of the keyframe into a keyframe database of the robot); and
at least one processor configured to execute computer-readable instructions that direct the at least one processor to:
perform a coarse-matching algorithm to identify a set of coarsely matched frames of data stored in the image database that coarsely match the query frame ([0033] At operation 230, the re-localization apparatus may determine one or more rough matching frames from the keyframes based on comparison between the global descriptor of each keyframe and the global descriptor of the current frame);
perform a fine-matching algorithm to identify a candidate image in the set of coarsely matched frames that match the query frame ([0035] At operation 240, the re-localization apparatus may determine a final matching frame from the one or more rough matching frames based on comparison between the local descriptors of each rough matching frame and the local descriptors of the current frame); wherein execution of the fine-matching algorithm causes the at least one processor to:
receive confidence metrics for putative matches between local descriptors for the query frame and local descriptors for each coarsely matched frame in the set of coarsely matched frames;
sum the confidence metrics for each of the coarsely matched frames; and
designate a coarsely matched frame with the highest sum as the candidate image
designate the candidate image as a matching image based on whether the candidate image satisfies a validity check ([0044] At operation 340, the re-localization apparatus may calculate a reprojection error between the keypoint in the current frame and the matching point in the rough matching frame based on the fundamental matrix or the homography matrix, [0047] Then at operation 360, the re-localization apparatus may select a rough matching frame with a largest number of inline points as the final matching frame, [0048]: when the number of inline points in the rough matching frame is greater than the predetermined threshold, the re-localization apparatus may derive a Rotation (R) transform and a Translation (T) transform between the rough matching frame and the current frame based on the fundamental matrix or the homography matrix, and calculate the pose of the current frame , [0049]: the re-localization apparatus may calculate a deviation between a predetermined percentage (e.g. 30%) of poses of the current frame calculated based on left rough matching frames. When the deviation does not exceed a predetermined threshold, the matching results may be considered as credible results. Then the rough matching frame with the largest number of inline points may be selected as the final matching frame); and
perform a pose-solving algorithm based on the acquired query frame, the matching image, and parameters for the image acquisition device to estimate a pose of the object in six degrees of freedom ([0005]: the system may use a perspective-n-point (PNP) algorithm between the keypoints of the current frame and the 3D landmarks to directly calculate a 6-Degree-of-Freedom (DoF) camera pose of the current frame).
Shi does not explicitly teach wherein execution of the fine-matching algorithm causes the at least one processor to:
receive confidence metrics for putative matches between local descriptors for the query frame and local descriptors for each coarsely matched frame in the set of coarsely matched frames;
sum the confidence metrics for each of the coarsely matched frames; and
designate a coarsely matched frame with the highest sum as the candidate image.
Datta teaches wherein execution of the fine-matching algorithm causes the at least one processor to:
receive confidence metrics for putative matches between local descriptors for the query frame and local descriptors for each coarsely matched frame in the set of coarsely matched frames ([0148]: the above processes of image comparing feature points and performing motion estimation across putative matching images may be performed multiple times for a particular query image to compare the query image to multiple potential matches among the stored database images. Dozens of comparisons may be performed before one (or more) satisfactory matches that exceed the relevant thresholds (for both matching feature points and motion estimation) may be found. The thresholds may also include a confidence threshold);
sum the confidence metrics for each of the coarsely matched frames ([0148]: Dozens of comparisons may be performed before one (or more) satisfactory matches that exceed the relevant thresholds (for both matching feature points and motion estimation) may be found. The thresholds may also include a confidence threshold, [0148]: system may continue attempting to match an image until a certain number of potential matches are identified, a certain confidence score is reached (either individually with a single potential match or among multiple matches)); and
designate a coarsely matched frame with the highest sum as the candidate image ([0148]: the system may stop processing additional candidate matches and simply select the high confidence match as the final match).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the teachings of Shi to include the teachings of Datta by including confidence metrics to select the final matching image. Doing so would improve the accuracy of selecting a matching image.
Regarding Claim 2, Shi and Datta teaches the system of claim 1. In addition, Shi teaches wherein the computer-readable instructions that direct the at least one processor to perform the coarse-matching algorithm further direct the at least one processor to:
calculate a query general descriptor for the query frame ([0032] At operation 220, the re-localization apparatus may extract image features of a current frame captured by the robot. The image features of the current frame may include a global descriptor and local descriptors of the current frame);
acquire database general descriptors for a plurality of frames stored in the image database ([0030] At operation 210, for each keyframe in the keyframe database of the robot, the re-localization apparatus may retrieve the image features and the pose of the keyframe. The image features of the keyframe may include a global descriptor and local descriptors of the keyframe);
compare the database general descriptors to the query general descriptor for each of the plurality of frames ([0033] At operation 230, the re-localization apparatus may determine one or more rough matching frames from the keyframes based on comparison between the global descriptor of each keyframe and the global descriptor of the current frame); and
designate a number of frames in the plurality of frames as the set of coarsely matched frames ([0033] At operation 230, the re-localization apparatus may determine one or more rough matching frames. See Fig. 5, element 510, select the first 15 smallest distance frames).
Regarding Claim 3, Shi and Datta teaches the system of claim 2. In addition, Shi teaches wherein the query general descriptor for the query frame is calculated using a machine learning model stored on the memory device ([0032]: the image features of the current frame may include a global descriptor and local descriptors of the current frame. The image features of the current frame may be extracted via the HF-Net model, [0023]: Hierarchical Feature Network (HF-Net) model. The HF-Net model is a deep learning model based on deep learning algorithms to extract image features).
Regarding Claim 5, Shi and Datta teaches the system of claim 2. In addition, Shi teaches wherein the database general descriptors are stored on the memory device after being received from a central repository ([0031]: extract the image features of the keyframes via the HF-Net model; obtain the poses of the keyframes from the SLAM system of the robot; and store the image features and the poses of the keyframes in the keyframe database. Accordingly, when the re-localization of the robot is needed, the re-localization apparatus may retrieve the image features and the poses of the keyframes from the keyframe database, [0030] At operation 210, for each keyframe in the keyframe database of the robot, the re-localization apparatus may retrieve the image features and the pose of the keyframe. The image features of the keyframe may include a global descriptor and local descriptors of the keyframe), wherein the database general descriptors were calculated by a plurality of processors at the central repository ([0069]: computing system 700 that can implement a method for re-localization …may include a processor 702 in communication with a memory 704. The memory 704 can include any device, combination of devices, circuitry, and the like that is capable of storing, accessing, organizing and/or retrieving data. Non-limiting examples include SANs (Storage Area Network), cloud storage networks, [0072] The processor 702 may be a single processor or multiple processors, and the memory 704 may be a single memory or multiple memories).
Regarding Claim 6, Shi and Datta teaches the system of claim 1. In addition, Shi teaches wherein the computer-readable instructions that direct the at least one processor to perform the fine-matching algorithm further direct the at least one processor to:
calculate query local descriptors for the query frame ([0032]: the image features of the current frame may include a global descriptor and local descriptors of the current frame. The image features of the current frame may be extracted via the HF-Net model);
acquire database local descriptors for a plurality of frames stored in the image database ([0030] At operation 210, for each keyframe in the keyframe database of the robot, the re-localization apparatus may retrieve the image features and the pose of the keyframe. The image features of the keyframe may include a global descriptor and local descriptors of the keyframe);
compare the database local descriptors to the query local descriptors for each of the frames in the set of coarsely matched frames ([0035] at operation 240, the re-localization apparatus may determine a final matching frame from the one or more rough matching frames based on comparison between the local descriptors of each rough matching frame and the local descriptors of the current frame); and
identify the candidate image in the set of coarsely matched frames ([0035] at operation 240, the re-localization apparatus may determine a final matching frame from the one or more rough matching frames based on comparison between the local descriptors of each rough matching frame and the local descriptors of the current frame).
Regarding Claim 7, Shi and Datta teaches the system of claim 6. In addition, Shi teaches wherein the query local descriptors for the query frame are calculated using a machine learning model stored on the memory device ([0032]: the image features of the current frame may include a global descriptor and local descriptors of the current frame. The image features of the current frame may be extracted via the HF-Net model).
Regarding Claim 8, Shi and Datta teaches the system of claim 6. In addition, Shi teaches wherein the query local descriptors are calculated using a learning based local descriptor algorithm ([0032]: the image features of the current frame may include a global descriptor and local descriptors of the current frame. The image features of the current frame may be extracted via the HF-Net model, [0023]: Hierarchical Feature Network (HF-Net) model. The HF-Net model is a deep learning model based on deep learning algorithms to extract image features).
Regarding Claim 9, Shi and Datta teaches the system of claim 6. In addition, Shi teaches wherein the database local descriptors are stored on the memory device after being received from a central repository ([0031]: extract the image features of the keyframes via the HF-Net model; obtain the poses of the keyframes from the SLAM system of the robot; and store the image features and the poses of the keyframes in the keyframe database. Accordingly, when the re-localization of the robot is needed, the re-localization apparatus may retrieve the image features and the poses of the keyframes from the keyframe database, [0030] At operation 210, for each keyframe in the keyframe database of the robot, the re-localization apparatus may retrieve the image features and the pose of the keyframe. The image features of the keyframe may include a global descriptor and local descriptors of the keyframe), wherein the database local descriptors were calculated by a plurality of processors at the central repository ([0069]: computing system 700 that can implement a method for re-localization …may include a processor 702 in communication with a memory 704. The memory 704 can include any device, combination of devices, circuitry, and the like that is capable of storing, accessing, organizing and/or retrieving data. Non-limiting examples include SANs (Storage Area Network), cloud storage networks, [0072] The processor 702 may be a single processor or multiple processors, and the memory 704 may be a single memory or multiple memories).
Regarding Claim 11, Shi and Datta teaches the system of claim 1. In addition, Shi teaches wherein the computer-readable instructions that direct the at least one processor to designate the candidate image as a matching image further direct the at least one processor to:
calculate a homography matrix based on correspondences between query local descriptors for the query frame and database local descriptors for the candidate image ([0043]: 330, the re-localization apparatus may calculate a fundamental matrix or a homography matrix between the current frame and the rough matching frame based on the one or more matching pairs including the keypoints in the current frame and the matching points in the rough matching frame, [0023]: a number of local descriptors that describe features of keypoints in the image frame);
map boundaries for the query frame onto the candidate image ([0044]: re-localization apparatus may calculate a reprojection error between the keypoint in the current frame and the matching point in the rough matching frame based on the fundamental matrix or the homography matrix, [0045]: the re-localization apparatus may determine the matching point as an inline point when the reprojection error calculated at operation 340 is smaller than a predetermined threshold.); and
determine that the candidate image is the matching image when at least one of the query local descriptors correspond to the database local descriptors within the mapped boundaries and the mapped boundaries are associated with a valid mapping on the candidate image ([0047]: the re-localization apparatus may select a rough matching frame with a largest number of inline points as the final matching frame).
Regarding Claim 12, Shi and Datta teaches the system of claim 1. In addition, Shi teaches wherein the pose-solving algorithm is a perspective-N- point algorithm ([0005]: the system may use a perspective-n-point (PNP) algorithm between the keypoints of the current frame and the 3D landmarks to directly calculate a 6-Degree-of-Freedom (DoF) camera pose of the current frame).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20240029300 A1) and Datta (US 12254548 B1) in view of Filip (F. Radenović, G. Tolias and O. Chum, "Fine-Tuning CNN Image Retrieval with No Human Annotation," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 7, pp. 1655-1668, 1 July 2019, doi: 10.1109/TPAMI.2018.2846566).
Regarding Claim 4, Shi and Datta teaches the system of claim 3. However, neither Shi nor Data explicitly teach the remaining limitation of Claim 4. -Filip teaches wherein the query general descriptor is calculated using generalized mean pooling ([Section 3.2, paragraph 1]: generalized mean pooling and image descriptor…we exploit the generalized mean [55] and propose to use generalized-mean (GeM) pooling).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the teachings of Shi and Datta to include the teachings of Filip by substituting Shi’s model for extracting and comparing global features for image retrieval by Filip’s model that includes a generalized mean pooing layer. Doing so would improve the accuracy of image retrieval performance (Filip [abstract]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20240029300 A1) and Datta (US 12254548 B1) in view of Sarlin (US 20210150252 A1).
Regarding Claim 10, Shi and Datta teaches the system of claim 6. However, neither Shi nor Datta explicitly teach the remaining limitations of Claim 10. Sarlin teaches wherein the database local descriptors are compared to the query local descriptors using an attentional graphical neural network algorithm ([abstract]: description relates the feature matching. Our approach establishes pointwise correspondences between challenging image pairs. It takes off-the-shelf local features as input and uses an attentional graph neural network to solve an assignment optimization problem, [0004]: neural network configured to match two sets of local features by jointly finding correspondences and rejecting non-matchable points).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present invention to have modified the teachings of Shi and Datta by Sarlin by substituting Shi’s teaching of local feature matching for Sarlin’s teaching of local feature matching through a graph neural network. Doing so would provide the predictable result of a matching operation between two local feature sets of two images.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20240029300 A1) and Datta (US 12254548 B1) in view of Bai (CN 112419374 B).
Regarding Claim 13, Shi and Datta teach the system of claim 1. However, neither Shi nor Datta explicitly teach the remaining limitations of Claim 13.
Bai teaches further comprising one or more additional sensors, wherein the one or more additional sensors provide navigation measurements of heading and altitude ([pg. 5, paragraph 3]: obtaining the flight height of the unmanned aerial vehicle from the height sensor carried by the unmanned aerial vehicle…obtaining the flight direction of the unmanned aerial vehicle from the heading sensor carried by the unmanned aerial vehicle), wherein the at least one processor performs the coarse-matching algorithm for the data in the image database at an orientation associated with the heading and a scale associated with the altitude ([pg. 5, paragraph 3]: performing rotation conversion and scale conversion on the unmanned aerial vehicle shooting image, making it have the same direction and scale with the map image, [abstract]: matching the feature of the unmanned aerial vehicle shooting image with the map image, obtaining the corresponding relation of the key point image coordinate in the two images).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present invention to have modified the teachings of Shi and Datta to include the teachings of Bai. Doing so would improve a matching operation by correcting for misalignment from rotation and height variations between the captured image and database image.
Claim 14-19 is rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20240029300 A1) in view of Holzschneider (US 20210397838 A1).
Regarding Claim 14, Shi teaches a method comprising:
acquiring a query frame from an image sensor mounted to an object ([0029]: re-localization of a robot, [0032] At operation 220, the re-localization apparatus may extract image features of a current frame captured by the robot);
acquiring image data from an image database ([0022]: the apparatus may receive Red Green Blue Depth (RGBD) images from a visual system of the robot and acquire keyframes from the RGBD images. Then the apparatus may perform a feature extraction process for each keyframe to extract image features of the keyframe and then save the image features of the keyframe into a keyframe database of the robot); and
performing a coarse-matching algorithm to identify a set of coarsely matched frames of data stored in the image database that coarsely match the query frame ([0033] At operation 230, the re-localization apparatus may determine one or more rough matching frames from the keyframes based on comparison between the global descriptor of each keyframe and the global descriptor of the current frame);
performing a fine-matching algorithm to identify a candidate image in the set of coarsely matched frames that match the query frame ([0035] At operation 240, the re-localization apparatus may determine a final matching frame from the one or more rough matching frames based on comparison between the local descriptors of each rough matching frame and the local descriptors of the current frame);
designating the candidate image as a matching image based on whether the candidate image satisfies a validity check ([0044] At operation 340, the re-localization apparatus may calculate a reprojection error between the keypoint in the current frame and the matching point in the rough matching frame based on the fundamental matrix or the homography matrix, [0047] Then at operation 360, the re-localization apparatus may select a rough matching frame with a largest number of inline points as the final matching frame, [0048]: when the number of inline points in the rough matching frame is greater than the predetermined threshold, the re-localization apparatus may derive a Rotation (R) transform and a Translation (T) transform between the rough matching frame and the current frame based on the fundamental matrix or the homography matrix, and calculate the pose of the current frame , [0049]: the re-localization apparatus may calculate a deviation between a predetermined percentage (e.g. 30%) of poses of the current frame calculated based on left rough matching frames. When the deviation does not exceed a predetermined threshold, the matching results may be considered as credible results. Then the rough matching frame with the largest number of inline points may be selected as the final matching frame); wherein execution of the validity check comprises:
identifying a homography matrix based on local descriptors for the candidate image and query local descriptors for the query frame ([0043]: 330, the re-localization apparatus may calculate a fundamental matrix or a homography matrix between the current frame and the rough matching frame based on the one or more matching pairs including the keypoints in the current frame and the matching points in the rough matching frame);
mapping a boundary of the query frame to the candidate image based on the homography matrix; and
determining at least one of:
whether the boundary has an irregular shape; and
whether application of the homography matrix to the query local descriptors causes the query local descriptors to be outside of the boundary;
performing a pose-solving algorithm based on the acquired query frame, the matching image, and parameters for the image acquisition device to estimate a pose of the object in six degrees of freedom ([0005]: the system may use a perspective-n-point (PNP) algorithm between the keypoints of the current frame and the 3D landmarks to directly calculate a 6-Degree-of-Freedom (DoF) camera pose of the current frame).
Shi does not explicitly teach wherein execution of the validity check comprises:
mapping a boundary of the query frame to the candidate image based on the homography matrix; and
determining at least one of:
whether the boundary has an irregular shape; and
whether application of the homography matrix to the query local descriptors causes the query local descriptors to be outside of the boundary.
Holzschneider teaches wherein execution of the validity check comprises:
mapping a boundary of the query frame to the candidate image based on the homography matrix ([0074]: In some embodiments, step 446 is executed in a similar manner as described in FIG. 2B, where a homographic transformation is applied to the reference image. See Fig. 2B, the image being mapped element 250 appears to be within the second image 240, thereby the boundaries of the image 250 appear to be mapped/aligned after homographic transform); and
determining at least one of:
whether the boundary has an irregular shape; and
whether application of the homography matrix to the query local descriptors causes the query local descriptors to be outside of the boundary ([0074]: homographic transformation is applied to the reference image, and then a determination is made whether at least part of the query image, including the consensus set of features, fit into the transformed reference image, from which it is determined whether the two images match).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present invention to have modified Shi to include the teachings of Holzschneider by including a validity check prior to determining a final matching frame involving use of the homography matrix to transform an image and determining whether matched features fit into the transformed image. Doing so would improve the accuracy of image matching.
Regarding Claim 15, Shi and Holzschneider teaches the system of claim 14. In addition, Shi teaches wherein performing the coarse-matching algorithm further comprises:
calculating a query general descriptor for the query frame ([0032] At operation 220, the re-localization apparatus may extract image features of a current frame captured by the robot. The image features of the current frame may include a global descriptor and local descriptors of the current frame);
acquiring database general descriptors for a plurality of frames stored in the image database ([0030] At operation 210, for each keyframe in the keyframe database of the robot, the re-localization apparatus may retrieve the image features and the pose of the keyframe. The image features of the keyframe may include a global descriptor and local descriptors of the keyframe);
comparing the database general descriptors to the query general descriptor for each of the plurality of frames ([0033] At operation 230, the re-localization apparatus may determine one or more rough matching frames from the keyframes based on comparison between the global descriptor of each keyframe and the global descriptor of the current frame); and
designating a number of frames in the plurality of frames as the set of coarsely matched frames ([0033] At operation 230, the re-localization apparatus may determine one or more rough matching frames. See Fig. 5, element 510, select the first 15 smallest distance frames).
Regarding Claim 16, Shi and Holzschneider teaches the system of claim 15. In addition, Shi teaches wherein performing the coarse-matching algorithm comprises calculating the query general descriptor for the query frame using a machine learning model ([0032]: the image features of the current frame may include a global descriptor and local descriptors of the current frame. The image features of the current frame may be extracted via the HF-Net model, [0023]: Hierarchical Feature Network (HF-Net) model. The HF-Net model is a deep learning model based on deep learning algorithms to extract image features).
Regarding Claim 17, Shi and Holzschneider teaches the system of claim 14. In addition, Shi teaches wherein performing the fine-matching algorithm further comprises:
calculating query local descriptors for the query frame ([0032]: the image features of the current frame may include a global descriptor and local descriptors of the current frame. The image features of the current frame may be extracted via the HF-Net model);
acquiring database local descriptors for a plurality of frames stored in the image database ([0030] At operation 210, for each keyframe in the keyframe database of the robot, the re-localization apparatus may retrieve the image features and the pose of the keyframe. The image features of the keyframe may include a global descriptor and local descriptors of the keyframe);
comparing the database local descriptors to the query local descriptors for each of the frames in the set of coarsely matched frames ([0035] at operation 240, the re-localization apparatus may determine a final matching frame from the one or more rough matching frames based on comparison between the local descriptors of each rough matching frame and the local descriptors of the current frame); and
identifying the candidate image in the set of coarsely matched frames ([0035] at operation 240, the re-localization apparatus may determine a final matching frame from the one or more rough matching frames based on comparison between the local descriptors of each rough matching frame and the local descriptors of the current frame).
Regarding Claim 18, Shi and Holzschneider teaches the system of claim 17. In addition, Shi teaches wherein the query local descriptors for the query frame are calculated using a machine learning model ([0032]: the image features of the current frame may include a global descriptor and local descriptors of the current frame. The image features of the current frame may be extracted via the HF-Net model).
Regarding Claim 19, Shi and Holzschneider teaches the system of claim 14. In addition, Shi teaches wherein identifying the homography matrix comprises:
calculating a homography matrix based on correspondences between query local descriptors for the query frame and database local descriptors for the candidate image ([0043]: 330, the re-localization apparatus may calculate a fundamental matrix or a homography matrix between the current frame and the rough matching frame based on the one or more matching pairs including the keypoints in the current frame and the matching points in the rough matching frame, [0023]: a number of local descriptors that describe features of keypoints in the image frame);
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20240029300 A1) in view of Datta (US 12254548 B1) and Holzschneider (US 20210397838 A1).
Regarding Claim 20, Shi teaches a system comprising:
a central repository ([0069]: computing system 700 that can implement a method for re-localization …may include a processor 702 in communication with a memory 704. The memory 704 can include any device, combination of devices, circuitry, and the like that is capable of storing, accessing, organizing and/or retrieving data. Non-limiting examples include SANs (Storage Area Network), cloud storage networks) comprising:
a plurality of processors ([0072] processor 702 may be a single processor or multiple processors, and the memory 704 may be a single memory or multiple memories); and
an image database storing a repository of image data acquired from a third party ([0022]: the apparatus may receive Red Green Blue Depth (RGBD) images from a visual system of the robot and acquire keyframes from the RGBD images. Then the apparatus may perform a feature extraction process for each keyframe to extract image features of the keyframe and then save the image features of the keyframe into a keyframe database);
wherein the plurality of processors executes a plurality of machine learning models using a portion of the repository of image data to create a plurality of three-dimensional images, local descriptors, and general descriptors for images in the plurality of three-dimensional images ([0022]: the apparatus may receive Red Green Blue Depth (RGBD) images from a visual system of the robot and acquire keyframes from the RGBD images. Then the apparatus may perform a feature extraction process for each keyframe to extract image features of the keyframe and then save the image features of the keyframe into a keyframe database, [0031]: extract the image features of the keyframes via the HF-Net model); and
a navigation system comprising:
an image sensor mounted to an object, the image sensor configured to acquire a query frame of an environment containing the navigation system ([0029]: re-localization of a robot, [0032] At operation 220, the re-localization apparatus may extract image features of a current frame captured by the robot);
a memory device configured to store the three-dimensional images, the local descriptors, and the general descriptors received from the central repository ([0022]: the apparatus may receive Red Green Blue Depth (RGBD) images from a visual system of the robot and acquire keyframes from the RGBD images, [0031]: apparatus may retrieve the image features and the poses of the keyframes from the keyframe database, [0030] apparatus may retrieve the image features and the pose of the keyframe. The image features of the keyframe may include a global descriptor and local descriptors of the keyframe); and
at least one processor configured to execute computer-readable instructions that direct the at least one processor to ([0072] processor 702 may be a single processor or multiple processors):
perform a coarse-matching algorithm to identify a set of coarsely matched frames of data in the three-dimensional images that coarsely match the query frame ([0033] At operation 230, the re-localization apparatus may determine one or more rough matching frames from the keyframes based on comparison between the global descriptor of each keyframe and the global descriptor of the current frame);
perform a fine-matching algorithm to identify a candidate image in the set of coarsely matched candidates that match the query frame ([0035] At operation 240, the re-localization apparatus may determine a final matching frame from the one or more rough matching frames based on comparison between the local descriptors of each rough matching frame and the local descriptors of the current frame); wherein execution of the fine-matching algorithm causes the at least one processor to:
receive confidence metrics for putative matches between local descriptors for the query frame and local descriptors for each coarsely matched frame in the set of coarsely matched frames;
sum the confidence metrics for each of the coarsely matched frames; and
designate a coarsely matched frame with the highest sum as the candidate image;
designate the candidate image as a matching image based on whether the candidate image satisfies a validity check ([0048]: when the number of inline points in the rough matching frame is greater than the predetermined threshold, the re-localization apparatus may derive a Rotation (R) transform and a Translation (T) transform between the rough matching frame and the current frame based on the fundamental matrix or the homography matrix, and calculate the pose of the current frame , [0049]: the re-localization apparatus may calculate a deviation between a predetermined percentage (e.g. 30%) of poses of the current frame calculated based on left rough matching frames. When the deviation does not exceed a predetermined threshold, the matching results may be considered as credible results. Then the rough matching frame with the largest number of inline points may be selected as the final matching frame), wherein execution of the validity check comprises:
identifying a homography matrix based on local descriptors for the candidate image and query local descriptors for the query frame;
mapping a boundary of the query frame to the candidate image based on the homography matrix; and
determining at least one of:
whether the boundary has an irregular shape; and
whether application of the homography matrix to the query local descriptors causes the query local descriptors to be outside of the boundary;
perform a pose-solving algorithm based on the acquired query frame, the matching image, and parameters for the image sensor to estimate a pose of the object in six degrees of freedom ([0005]: the system may use a perspective-n-point (PNP) algorithm between the keypoints of the current frame and the 3D landmarks to directly calculate a 6-Degree-of-Freedom (DoF) camera pose of the current frame).
Shi does not explicitly teach wherein execution of the fine-matching algorithm causes the at least one processor to:
receive confidence metrics for putative matches between local descriptors for the query frame and local descriptors for each coarsely matched frame in the set of coarsely matched frames;
sum the confidence metrics for each of the coarsely matched frames; and
designate a coarsely matched frame with the highest sum as the candidate image.
Datta teaches wherein execution of the fine-matching algorithm causes the at least one processor to:
receive confidence metrics for putative matches between local descriptors for the query frame and local descriptors for each coarsely matched frame in the set of coarsely matched frames ([0148]: the above processes of image comparing feature points and performing motion estimation across putative matching images may be performed multiple times for a particular query image to compare the query image to multiple potential matches among the stored database images. Dozens of comparisons may be performed before one (or more) satisfactory matches that exceed the relevant thresholds (for both matching feature points and motion estimation) may be found. The thresholds may also include a confidence threshold);
sum the confidence metrics for each of the coarsely matched frames ([0148]: Dozens of comparisons may be performed before one (or more) satisfactory matches that exceed the relevant thresholds (for both matching feature points and motion estimation) may be found. The thresholds may also include a confidence threshold, [0148]: system may continue attempting to match an image until a certain number of potential matches are identified, a certain confidence score is reached (either individually with a single potential match or among multiple matches)); and
designate a coarsely matched frame with the highest sum as the candidate image ([0148]: the system may stop processing additional candidate matches and simply select the high confidence match as the final match).
Further, Shi does not explicitly teach wherein execution of the validity check causes the at least one processor to:
map a boundary of the query frame to the candidate image based on the homography matrix; and
determining at least one of:
whether the boundary has an irregular shape; and
whether application of the homography matrix to the query local descriptors causes the query local descriptors to be outside of the boundary.
Holzschneider teaches wherein execution of the validity check causes the at least one processor to:
map a boundary of the query frame to the candidate image based on the homography matrix ([0074]: In some embodiments, step 446 is executed in a similar manner as described in FIG. 2B, where a homographic transformation is applied to the reference image. See Fig. 2B, the image being mapped element 250 appears to be within the second image 240, thereby the boundaries of the image 250 appear to be mapped/aligned after homographic transform); and
determining at least one of:
whether the boundary has an irregular shape; and
whether application of the homography matrix to the query local descriptors causes the query local descriptors to be outside of the boundary ([0074]: homographic transformation is applied to the reference image, and then a determination is made whether at least part of the query image, including the consensus set of features, fit into the transformed reference image, from which it is determined whether the two images match).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the teachings of Shi to include the teachings of Datta by including confidence metrics to select the final matching image. Doing so would improve the accuracy of selecting a matching image. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present invention to have modified Shi to include the teachings of Holzschneider by including a validity check prior to determining a final matching frame involving use of the homography matrix to transform an image and determining whether matched features fit into the transformed image. Doing so would also improve the accuracy of image matching.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JANICE VAZ whose telephone number is (703)756-4685. The examiner can normally be reached Monday-Friday 9:00-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JANICE E. VAZ/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights