DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This is in response to Applicant’s Arguments/Remarks filed on July 29th, 2026 which has been
entered and made of record.
Response to Arguments
Claim Rejections – 35 USC§ 102/103
Applicant’s arguments regarding the current claim(s) have been fully considered. But, the
arguments/remarks are directed to the claims as amended, and so are believed to be answered
by and therefore moot in view of the new grounds of rejection presented below.
Status of Claims
Claims 1-10 and 12-20 are pending. Claims 1, 14, and 20 were amended. Claim 11 was canceled. No new claim(s) were added. Claims 1-10 and 12-20 are considered below.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 19 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. Claim 14 already recites, “calculating a homography matrix based on correspondences between query local descriptors for the query frame and database local descriptors for the candidate image”. Claim 19 appears to be reciting the same limitation, thereby failing to further limit the independent claim from which it depends. Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-9, 12, and 14-19 are rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20240029300 A1) in view of Holzschneider (US 20210397838 A1) and Fuhao (US 20220345621 A1).
Regarding Claim 1, representative of Claim 14 and 19, Shi teaches a system comprising:
an image acquisition device mounted to an object ([abstract]: method for re-localization of the robot), the image acquisition device configured to acquire a query frame of an environment containing the object ([abstract]: extracting image features of a current frame captured by the robot);
a memory device configured to store an image database ([0022]: apparatus may receive Red Green Blue Depth (RGBD) images from a visual system of the robot and acquire keyframes from the RGBD images. Then the apparatus may perform a feature extraction process for each keyframe to extract image features of the keyframe and then save the image features of the keyframe into a keyframe database); and
at least one processor configured to execute computer-readable instructions that direct the at least one processor to:
perform a coarse-matching algorithm to identify a set of coarsely matched frames of data stored in the image database that coarsely match the query frame ([0033] At operation 230, the re-localization apparatus may determine one or more rough matching frames from the keyframes based on comparison between the global descriptor of each keyframe and the global descriptor of the current frame)
perform a fine-matching algorithm to identify a candidate image in the set of coarsely matched frames that match the query frame ([0058]: At the operation 520, for each rough matching frame, the matching between the local descriptors of the rough matching frame and the local descriptors of the current frame may be performed. For each local descriptor in the current frame, the re-localization apparatus may calculate the Euclidean distance from each local descriptor of the rough matching frame. For example, when the minimum Euclidean distance is less than 0.9 times the second smallest Euclidean distance, the matching may be considered as a successful matching. Examiner interpreting an “identified candidate image” to be a rough matching frame with a successful matching pair of local descriptors);
designate the candidate image as a matching image based on whether the candidate image satisfies a validity check ([0060] apparatus may calculate a reprojection error between the keypoint in the current frame and the matching point in the rough matching frame based on the fundamental matrix or the homography matrix, and may determine the matching point as an inline point when the reprojection error is smaller than a predetermined threshold, [0061] When the number of inline points in the rough matching frame is not greater than a predetermined threshold, the rough matching frame may be discarded… When the number of inline points in the rough matching frame is greater than the predetermined threshold, the re-localization apparatus may…calculate the pose); and
perform a pose-solving algorithm based on the acquired query frame, the matching image, and parameters for the image acquisition device to estimate a pose of the object in six degrees of freedom ([0061]: when the number of inline points in the rough matching frame is not greater than a predetermined threshold, the rough matching frame may be discarded. When the number of inline points in the rough matching frame is greater than the predetermined threshold, the re-localization apparatus may…calculate the pose of the current frame based on the R and T transforms and the pose of the rough matching frame, [0005]: the system may use a perspective-n-point (PNP) algorithm between the keypoints of the current frame and the 3D landmarks to directly calculate a 6-Degree-of-Freedom (DoF) camera pose of the current frame);
wherein the computer-readable instructions that direct the at least one processor to designate the candidate image as a matching image further direct the at least one processor to:
calculate a homography matrix based on correspondences between the query local descriptors for the query frame and the database local descriptors for the candidate image ([0059] The operations 530 and 540 may correspond to the operation 330 in FIG. 3 and may calculate a fundamental matrix and a homography matrix between the current frame and the rough matching frame based on the one or more matching pairs);
Shi does not explicitly teach map boundaries for the query frame onto the candidate image; and
determine that the candidate image is the matching image when at least one of the query local descriptors corresponds to the database local descriptors within the mapped boundaries and the mapped boundaries are associated with a valid mapping on the candidate image.
Holzhneider teaches map boundaries for the ([0074]: where a homographic transformation is applied to the reference image); and
determine that the candidate image is the matching image when at least one of the query local descriptors corresponds to the database local descriptors within the mapped boundaries and the mapped boundaries are associated with a valid mapping on the candidate image ([0074]: where a homographic transformation is applied to the reference image, and then a determination is made whether at least part of the query image, including the consensus set of features, fit into the transformed reference image, from which it is determined whether the two images match).
Although Holzhneider teaches mapping boundaries for a reference frame onto a query frame rather than the claimed mapping of a query frame onto a reference frame/candidate image, it is known in the art that a query frame can be mapped to a reference frame in an image matching operation as taught by Fuhuo ([0080]: Different image matching methods may be applied in different examples, as described herein, such as direct image correlation-based matching and image feature based homography transform estimation. In some examples, translation estimator 364 may estimate two-dimensional translation by comparing (e.g., features of) current images of a current frame to (e.g., features of) reference images of a reference frame, and estimating the two-dimensional translation of the current images relative to the reference images based on the comparison, [0081] use information from …translation estimator 364, to map…a real-world scene of current frame data to the images of a reference frame).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the teachings of Shi to include the teachings of Holzhneider. Shi generally teaches a method for relocalizing a robot involving matching global descriptors of a current frame to reference frames, and then matching local descriptors. Further, Shi teaches a validity check by selecting a final frame based on reprojection error of keypoints associated with matched local descriptors. Shi does not explicitly teach that these keypoints/features also need to be within boundaries of a current frame mapped onto the reference frame. Holzschneider teaches an image matching method involving mapping the boundaries of a reference frame onto a query frame, and considering a match to have occurred based on correspondences/features fitting into the transformed frame/within the mapped bounds. The modification of Shi with Holzschneider would improve the accuracy of selecting a matching frame. Further, it would have been obvious to one of ordinary skill in the art to have modified the Shi and Holzschneider combination to include the teachings of Fuhao by substituting the mapping of a reference frame onto query frame to identify a matching frame as taught by Holzschneider, for Fuhao’s mapping of a query frame onto a reference frame. Doing so would provide the predictable result of a mapping between two frames to determine if the frames match.
Regarding Claim 2, representative of Claim 15, the Shi, Holzschneider, and Fuhao combination (hereinafter referred to as the Shi combination) teaches the system of claim 1. In addition, Shi teaches wherein the computer-readable instructions that direct the at least one processor to perform the coarse-matching algorithm further direct the at least one processor to:
calculate a query general descriptor for the query frame ([0032] At operation 220, the re-localization apparatus may extract image features of a current frame captured by the robot. The image features of the current frame may include a global descriptor and local descriptors of the current frame);
acquire database general descriptors for a plurality of frames stored in the image database ([0030] At operation 210, for each keyframe in the keyframe database of the robot, the re-localization apparatus may retrieve the image features and the pose of the keyframe. The image features of the keyframe may include a global descriptor and local descriptors of the keyframe);
compare the database general descriptors to the query general descriptor for each of the plurality of frames ([0033] At operation 230, the re-localization apparatus may determine one or more rough matching frames from the keyframes based on comparison between the global descriptor of each keyframe and the global descriptor of the current frame); and
designate a number of frames in the plurality of frames as the set of coarsely matched frames ([0033] At operation 230, the re-localization apparatus may determine one or more rough matching frames. See Fig. 5, element 510, select the first 15 smallest distance frames).
Regarding Claim 3, representative of Claim 16, the Shi combination teaches the system of claim 2. In addition, Shi teaches wherein the query general descriptor for the query frame is calculated using a machine learning model stored on the memory device ([0032]: the image features of the current frame may include a global descriptor and local descriptors of the current frame. The image features of the current frame may be extracted via the HF-Net model, [0023]: Hierarchical Feature Network (HF-Net) model. The HF-Net model is a deep learning model based on deep learning algorithms to extract image features).
Regarding Claim 5, the Shi combination teaches the system of claim 2. In addition, Shi teaches wherein the database general descriptors are stored on the memory device after being received from a central repository ([0031]: extract the image features of the keyframes via the HF-Net model; obtain the poses of the keyframes from the SLAM system of the robot; and store the image features and the poses of the keyframes in the keyframe database. Accordingly, when the re-localization of the robot is needed, the re-localization apparatus may retrieve the image features and the poses of the keyframes from the keyframe database, [0030] At operation 210, for each keyframe in the keyframe database of the robot, the re-localization apparatus may retrieve the image features and the pose of the keyframe. The image features of the keyframe may include a global descriptor and local descriptors of the keyframe), wherein the database general descriptors were calculated by a plurality of processors at the central repository ([0069]: computing system 700 that can implement a method for re-localization …may include a processor 702 in communication with a memory 704. The memory 704 can include any device, combination of devices, circuitry, and the like that is capable of storing, accessing, organizing and/or retrieving data. Non-limiting examples include SANs (Storage Area Network), cloud storage networks, [0072] The processor 702 may be a single processor or multiple processors, and the memory 704 may be a single memory or multiple memories).
Regarding Claim 6, representative of Claim 17, the Shi combination teaches the system of claim 1. In addition, Shi teaches wherein the computer-readable instructions that direct the at least one processor to perform the fine-matching algorithm further direct the at least one processor to:
calculate query local descriptors for the query frame ([0032]: the image features of the current frame may include a global descriptor and local descriptors of the current frame. The image features of the current frame may be extracted via the HF-Net model);
acquire database local descriptors for a plurality of frames stored in the image database ([0030] At operation 210, for each keyframe in the keyframe database of the robot, the re-localization apparatus may retrieve the image features and the pose of the keyframe. The image features of the keyframe may include a global descriptor and local descriptors of the keyframe);
compare the database local descriptors to the query local descriptors for each of the frames in the set of coarsely matched frames ([0058]: At the operation 520, for each rough matching frame, the matching between the local descriptors of the rough matching frame and the local descriptors of the current frame may be performed); and
identify the candidate image in the set of coarsely matched frames ([0058]: At the operation 520, for each rough matching frame, the matching between the local descriptors of the rough matching frame and the local descriptors of the current frame may be performed. For each local descriptor in the current frame, the re-localization apparatus may calculate the Euclidean distance from each local descriptor of the rough matching frame. For example, when the minimum Euclidean distance is less than 0.9 times the second smallest Euclidean distance, the matching may be considered as a successful matching. Examiner interpreting an “identified candidate image” to be a rough matching frame with a successful matching pair of local descriptors).
Regarding Claim 7, representative of Claim 18, the Shi combination teaches the system of claim 6. In addition, Shi teaches wherein the query local descriptors for the query frame are calculated using a machine learning model stored on the memory device ([0032]: the image features of the current frame may include a global descriptor and local descriptors of the current frame. The image features of the current frame may be extracted via the HF-Net model).
Regarding Claim 8, the Shi combination teaches the system of claim 6. In addition, Shi teaches wherein the query local descriptors are calculated using a learning based local descriptor algorithm ([0032]: the image features of the current frame may include a global descriptor and local descriptors of the current frame. The image features of the current frame may be extracted via the HF-Net model, [0023]: Hierarchical Feature Network (HF-Net) model. The HF-Net model is a deep learning model based on deep learning algorithms to extract image features).
Regarding Claim 9, the Shi combination teaches the system of claim 6. In addition, Shi teaches wherein the database local descriptors are stored on the memory device after being received from a central repository ([0031]: extract the image features of the keyframes via the HF-Net model; obtain the poses of the keyframes from the SLAM system of the robot; and store the image features and the poses of the keyframes in the keyframe database. Accordingly, when the re-localization of the robot is needed, the re-localization apparatus may retrieve the image features and the poses of the keyframes from the keyframe database, [0030] At operation 210, for each keyframe in the keyframe database of the robot, the re-localization apparatus may retrieve the image features and the pose of the keyframe. The image features of the keyframe may include a global descriptor and local descriptors of the keyframe), wherein the database local descriptors were calculated by a plurality of processors at the central repository ([0069]: computing system 700 that can implement a method for re-localization …may include a processor 702 in communication with a memory 704. The memory 704 can include any device, combination of devices, circuitry, and the like that is capable of storing, accessing, organizing and/or retrieving data. Non-limiting examples include SANs (Storage Area Network), cloud storage networks, [0072] The processor 702 may be a single processor or multiple processors, and the memory 704 may be a single memory or multiple memories).
Regarding Claim 12, the Shi combination teaches the system of claim 1. In addition, Shi teaches wherein the pose-solving algorithm is a perspective-N- point algorithm ([0005]: the system may use a perspective-n-point (PNP) algorithm between the keypoints of the current frame and the 3D landmarks to directly calculate a 6-Degree-of-Freedom (DoF) camera pose of the current frame).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20240029300 A1) in view of Holzschneider (US 20210397838 A1), Fuhao (US 20220345621 A1) and Filip (F. Radenović, G. Tolias and O. Chum, "Fine-Tuning CNN Image Retrieval with No Human Annotation," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 7, pp. 1655-1668, 1 July 2019, doi: 10.1109/TPAMI.2018.2846566).
Regarding Claim 4, Shi combination teaches the system of claim 3. However, none of the Shi combination explicitly teach the remaining limitation of Claim 4. -Filip teaches wherein the query general descriptor is calculated using generalized mean pooling ([Section 3.2, paragraph 1]: generalized mean pooling and image descriptor…we exploit the generalized mean [55] and propose to use generalized-mean (GeM) pooling).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the teachings of Shi combination to include the teachings of Filip by substituting Shi’s model for extracting and comparing global features for image retrieval by Filip’s model that includes a generalized mean pooing layer. Doing so would improve the accuracy of image retrieval performance (Filip [abstract]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20240029300 A1) in view of Holzschneider (US 20210397838 A1), Fuhao (US 20220345621 A1), and Sarlin (US 20210150252 A1).
Regarding Claim 10, the Shi combination teaches the system of claim 6. However, none of the Shi combination explicitly teach the remaining limitations of Claim 10. Sarlin teaches wherein the database local descriptors are compared to the query local descriptors using an attentional graphical neural network algorithm ([abstract]: description relates the feature matching. Our approach establishes pointwise correspondences between challenging image pairs. It takes off-the-shelf local features as input and uses an attentional graph neural network to solve an assignment optimization problem, [0004]: neural network configured to match two sets of local features by jointly finding correspondences and rejecting non-matchable points).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present invention to have modified the teachings of the Shi combination by Sarlin by substituting Shi’s teaching of local feature matching for Sarlin’s teaching of local feature matching through a graph neural network. Doing so would provide the predictable result of a matching operation between two local feature sets of two images.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20240029300 A1) in view of Holzschneider (US 20210397838 A1), Fuhao (US 20220345621 A1), and Bai (CN 112419374 B).
Regarding Claim 13, the Shi combination teach the system of claim 1. However, none of the Shi combination explicitly teach the remaining limitations of Claim 13.
Bai teaches further comprising one or more additional sensors, wherein the one or more additional sensors provide navigation measurements of heading and altitude ([pg. 5, paragraph 3]: obtaining the flight height of the unmanned aerial vehicle from the height sensor carried by the unmanned aerial vehicle…obtaining the flight direction of the unmanned aerial vehicle from the heading sensor carried by the unmanned aerial vehicle), wherein the at least one processor performs the coarse-matching algorithm for the data in the image database at an orientation associated with the heading and a scale associated with the altitude ([pg. 5, paragraph 3]: performing rotation conversion and scale conversion on the unmanned aerial vehicle shooting image, making it have the same direction and scale with the map image, [abstract]: matching the feature of the unmanned aerial vehicle shooting image with the map image, obtaining the corresponding relation of the key point image coordinate in the two images).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present invention to have modified the teachings of the Shi combination to include the teachings of Bai. Doing so would improve a matching operation by correcting for misalignment from rotation and height variations between the captured image and database image.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Shi (US 20240029300 A1) in view of Datta (US 12254548 B1), Holzschneider (US 20210397838 A1), and Fuhao (US 20220345621 A1).
Regarding Claim 20, Shi teaches a system comprising:
a central repository ([0069]: computing system 700 that can implement a method for re-localization …may include a processor 702 in communication with a memory 704. The memory 704 can include any device, combination of devices, circuitry, and the like that is capable of storing, accessing, organizing and/or retrieving data. Non-limiting examples include SANs (Storage Area Network), cloud storage networks) comprising:
a plurality of processors ([0072] processor 702 may be a single processor or multiple processors, and the memory 704 may be a single memory or multiple memories); and
an image database storing a repository of image data acquired from a third party ([0022]: the apparatus may receive Red Green Blue Depth (RGBD) images from a visual system of the robot and acquire keyframes from the RGBD images. Then the apparatus may perform a feature extraction process for each keyframe to extract image features of the keyframe and then save the image features of the keyframe into a keyframe database);
wherein the plurality of processors executes a plurality of machine learning models using a portion of the repository of image data to create a plurality of three-dimensional images, local descriptors, and general descriptors for images in the plurality of three-dimensional images ([0022]: the apparatus may receive Red Green Blue Depth (RGBD) images from a visual system of the robot and acquire keyframes from the RGBD images. Then the apparatus may perform a feature extraction process for each keyframe to extract image features of the keyframe and then save the image features of the keyframe into a keyframe database, [0031]: extract the image features of the keyframes via the HF-Net model); and
a navigation system comprising:
an image sensor mounted to an object, the image sensor configured to acquire a query frame of an environment containing the navigation system ([0029]: re-localization of a robot, [0032] At operation 220, the re-localization apparatus may extract image features of a current frame captured by the robot);
a memory device configured to store the three-dimensional images, the local descriptors, and the general descriptors received from the central repository ([0022]: the apparatus may receive Red Green Blue Depth (RGBD) images from a visual system of the robot and acquire keyframes from the RGBD images, [0031]: apparatus may retrieve the image features and the poses of the keyframes from the keyframe database, [0030] apparatus may retrieve the image features and the pose of the keyframe. The image features of the keyframe may include a global descriptor and local descriptors of the keyframe); and
at least one processor configured to execute computer-readable instructions that direct the at least one processor to ([0072] processor 702 may be a single processor or multiple processors):
perform a coarse-matching algorithm to identify a set of coarsely matched frames of data in the three-dimensional images that coarsely match the query frame ([0033] At operation 230, the re-localization apparatus may determine one or more rough matching frames from the keyframes based on comparison between the global descriptor of each keyframe and the global descriptor of the current frame);
perform a fine-matching algorithm to identify a candidate image in the set of coarsely matched candidates that match the query frame ([0058]: At the operation 520, for each rough matching frame, the matching between the local descriptors of the rough matching frame and the local descriptors of the current frame may be performed. For each local descriptor in the current frame, the re-localization apparatus may calculate the Euclidean distance from each local descriptor of the rough matching frame. For example, when the minimum Euclidean distance is less than 0.9 times the second smallest Euclidean distance, the matching may be considered as a successful matching. Examiner interpreting an “identified candidate image” to be a rough matching frame with a successful matching pair of local descriptors);
designate the candidate image as a matching image based on whether the candidate image satisfies a validity check ([0060] apparatus may calculate a reprojection error between the keypoint in the current frame and the matching point in the rough matching frame based on the fundamental matrix or the homography matrix, and may determine the matching point as an inline point when the reprojection error is smaller than a predetermined threshold, [0061] When the number of inline points in the rough matching frame is not greater than a predetermined threshold, the rough matching frame may be discarded… When the number of inline points in the rough matching frame is greater than the predetermined threshold, the re-localization apparatus may…calculate the pose); wherein execution of the validity check comprises:
calculate a homography matrix based on correspondences between query local descriptors for the query frame and database local descriptors for the candidate image [0059] The operations 530 and 540 may correspond to the operation 330 in FIG. 3 and may calculate a fundamental matrix and a homography matrix between the current frame and the rough matching frame based on the one or more matching pairs);
perform a pose-solving algorithm based on the acquired query frame, the matching image, and parameters for the image sensor to estimate a pose of the object in six degrees of freedom ([0005]: the system may use a perspective-n-point (PNP) algorithm between the keypoints of the current frame and the 3D landmarks to directly calculate a 6-Degree-of-Freedom (DoF) camera pose of the current frame).
Shi does not explicitly teach wherein execution of the fine-matching algorithm causes the at least one processor to:
receive confidence metrics for putative matches between local descriptors for the query frame and local descriptors for each coarsely matched frame in the set of coarsely matched frames;
sum the confidence metrics for each of the coarsely matched frames; and
designate a coarsely matched frame with the highest sum as the candidate image.
Datta teaches wherein execution of the fine-matching algorithm causes the at least one processor to:
receive confidence metrics for putative matches between local descriptors for the query frame and local descriptors for each coarsely matched frame in the set of coarsely matched frames ([0148]: the above processes of image comparing feature points and performing motion estimation across putative matching images may be performed multiple times for a particular query image to compare the query image to multiple potential matches among the stored database images. Dozens of comparisons may be performed before one (or more) satisfactory matches that exceed the relevant thresholds (for both matching feature points and motion estimation) may be found. The thresholds may also include a confidence threshold);
sum the confidence metrics for each of the coarsely matched frames ([0148]: Dozens of comparisons may be performed before one (or more) satisfactory matches that exceed the relevant thresholds (for both matching feature points and motion estimation) may be found. The thresholds may also include a confidence threshold, [0148]: system may continue attempting to match an image until a certain number of potential matches are identified, a certain confidence score is reached (either individually with a single potential match or among multiple matches)); and
designate a coarsely matched frame with the highest sum as the candidate image ([0148]: the system may stop processing additional candidate matches and simply select the high confidence match as the final match).
Further, Shi does not explicitly teach wherein execution of the validity check causes the at least one processor to:
map boundaries for the query frame onto the candidate image; and
determine that the candidate image is the matching image when at least one of the query local descriptors corresponds to the database local descriptors within the mapped boundaries and the mapped boundaries are associated with a valid mapping on the candidate image; and
Holzschneider teaches wherein execution of the validity check causes the at least one processor to:
map boundaries for the ([0074]: In some embodiments, step 446 is executed in a similar manner as described in FIG. 2B, where a homographic transformation is applied to the reference image); and
determine that the candidate image is the matching image when at least one of the query local descriptors corresponds to the database local descriptors within the mapped boundaries and the mapped boundaries are associated with a valid mapping on the candidate image ([0074]: homographic transformation is applied to the reference image, and then a determination is made whether at least part of the query image, including the consensus set of features, fit into the transformed reference image, from which it is determined whether the two images match).
Although Holzhneider teaches mapping boundaries for a reference frame onto a query frame rather than the claimed mapping of a query frame onto a reference frame/candidate image, it is known in the art that a query frame can be mapped to a reference frame in an image matching operation as taught by Fuhuo ([0080]: Different image matching methods may be applied in different examples, as described herein, such as direct image correlation-based matching and image feature based homography transform estimation. In some examples, translation estimator 364 may estimate two-dimensional translation by comparing (e.g., features of) current images of a current frame to (e.g., features of) reference images of a reference frame, and estimating the two-dimensional translation of the current images relative to the reference images based on the comparison, [0081] use information from …translation estimator 364, to map…a real-world scene of current frame data to the images of a reference frame).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the teachings of Shi to include the teachings of Datta by including confidence metrics to select the closest matching image. Doing so would improve the accuracy of selecting a matching image after local descriptor matching. Further, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the present invention to have modified the Shi and Datta combination to include the teachings of Holzschneider. Shi teaches a validity check by selecting a final frame based on reprojection error of keypoints associated with matched local descriptors. Shi does not explicitly teach that these keypoints/features also need to be within boundaries of the current frame mapped onto the reference frame. Holzschneider teaches an image matching method involving mapping the boundaries of a reference frame onto a query frame, and considering a match to have occurred based on correspondences/features fitting into the transformed frame/within the mapped bounds. The modification of the Shi and Datta combination with Holzschneider would improve the accuracy of selecting a matching frame. Lastly, it would have been obvious to one of ordinary skill to have modified the Shi, Datta, and Holzshneider combination to include the teachings of Fuhao by substituting the mapping of a reference frame onto query frame to identify a matching frame as taught by Holzschneider, for Fuhao’s mapping of a query frame onto a reference frame. Doing so would provide the predictable result of a mapping between two frames to determine if the frames match.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JANICE VAZ whose telephone number is (703)756-4685. The examiner can normally be reached Monday-Friday 9:00-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JANICE E. VAZ/Examiner, Art Unit 2667 /MICHAEL ROBERT CAMMARATA/Primary Examiner, Art Unit 2667
Prosecution Insights