DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 11/12/2024, 11/17/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are considered by examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 9-14, 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al (Topological Spatial Verification for Instance Search) in view of Williams et al (Real-Time SLAM Relocalisation).
Regarding Claim 1, Zhang et al teach a method (topological spatial matching using a triangulated-graph based technique; Fig 2 and III. Topological Spatial Verification ¶ 1) comprising:
obtaining a set of keypoint matches (matches of keypoints (visual words) from query and reference image based on Sketch-and-Match; Fig 2 and III.A. Topological Spatial Verification, Spatial Configuration ¶ 1, III.B. Sketch-and-Match ¶ 5-6) comprising a set of query image keypoints from a query image (keypoints (visual words) from query image Q; Fig 2 and III.A. Topological Spatial Verification, Spatial Configuration ¶ 1) and a corresponding set of reference image keypoints from a reference image (keypoints (visual words) from reference image R; Fig 2 and III.A. Topological Spatial Verification, Spatial Configuration ¶ 1) to which each of the set of query image keypoints are matched (matches of keypoints (visual words) from query and reference image based on Q=HR, triangulation and weighting factor; Fig 2 and III.A. Topological Spatial Verification, Spatial Configuration ¶ 1, III.B. Sketch-and-Match ¶ 5-6);
determining a first spatial relationship (first spatial relationship of keypoints within the query image Q is identified with a triangulated graph; Fig 2 and III.B. Sketch-and-Match ¶ 3-4) between a first query image keypoint and one or more additional query image keypoints of the set of query image keypoints (spatial topology within the query image Q is determined with Delaunay Triangulation to sketch the triangulated graph; Fig 2 and III.B. Sketch-and-Match ¶ 3-4);
identifying a first reference image keypoint from the set of reference image keypoints matched to the first query image keypoint (matches of keypoints (visual words) from query and reference image based on Q=HR using a homography matrix; Fig 2 and III.A. Topological Spatial Verification, Spatial Configuration ¶ 1); and
determining a second spatial relationship (second spatial relationship of keypoints within the reference image R is identified with a triangulated graph; Fig 2 and III.B. Sketch-and-Match ¶ 3-4) between the first reference image keypoint and one or more additional reference image keypoints of the set of reference image keypoints (spatial topology within the reference image R is determined with Delaunay Triangulation to sketch the triangulated graph; Fig 2 and III.B. Sketch-and-Match ¶ 3-4).
Zhang et al does not teach determining relocalization data based on a comparison of the first spatial relationship and the second spatial relationship.
Williams et al is analogous art pertinent to the technological problem addressed in the current application and teaches determining relocalization data based on a comparison of the first spatial relationship and the second spatial relationship (relocalization is performed to calculate pose and predict the location of the image to reinitialize the SLAM system based on feature correspondence; Fig 3 and 4.2 Randomised Lists for SLAM recovery ¶ 5. Relocalisation Using the Randomised Lists Classifier ¶ 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Zhang et al with Williams et al including determining relocalization data based on a comparison of the first spatial relationship and the second spatial relationship. By applying a relocalization, real-time tracking can be used for image matching and resetting in systems such as SLAM, thereby overcoming issues associated with rapid camera motions, motion blurs and similar pose losses, thereby improving resets through re-estimations rather than from scratch resets, thereby reducing computation and preserving map integrity, as recognized by Williams et al (Abstract, 1. Introduction ¶ 3-5).
Regarding Claim 2, Zhang et al in view of Williams et al teach the method of claim 1 (as described above), wherein determining the relocalization data (Williams et al, relocalization based on image geometry; Fig 5 and 6.2 SLAM with Recovery ¶ 3-5, Table 3) comprises: comparing a geometry of the first spatial relationship to a geometry of the second spatial relationship (Williams et al, geometry within the image is used to map the (geometry of) features of the first image to a second image (for tracking) with features mapping (comparing geometry) for relocalising; Fig 5 and 6.2 SLAM with Recovery ¶ 3-8, Table 3; examiner notes Zhang et al teaches a spatial triangulation technique used for comparing the geometry relationships between keypoints in the query and reference images based on graph matching; Fig 2 and III.B. Sketch-and-Match ¶ 3-6).
Regarding Claim 3, Zhang et al in view of Williams et al teach the method of claim 2 (as described above), wherein comparing the geometry of the first spatial relationship to the geometry of the second spatial relationship (Zhang et al, the triangulated graph data between the identical keypoints in the first image and second image are matched with graph matching; Fig 2 and III.B. Sketch-and-Match ¶ 3-6) comprises: calculating a graph distance between the geometry of the first spatial relationship and the geometry of the second spatial relationship (Zhang et al, the triangulated graph data between the identical keypoints in the first image and second image are matched with graph matching with each point indexed based on a Hamming distance measurement; Fig 2 and III.B. Sketch-and-Match ¶ 5).
Regarding Claim 4, Zhang et al in view of Williams et al teach the method of claim 3 (as described above), further comprising: assigning a weight to the first query image keypoint based on the graph distance (Zhang et al, a weighting strategy is assigned to estimate the spatial consistency between common edges in the query and reference image keypoints; III.C. Weighting Strategy).
Regarding Claim 5, Zhang et al in view of Williams et al teach the method of claim 1 (as described above), wherein determining the relocalization data (Williams et al, relocalization based on image geometry; Fig 5 and 6.2 SLAM with Recovery ¶ 3-5, Table 3) comprises: comparing visual characteristics of the query image keypoints comprised in the first spatial relationship to visual characteristics of the reference image keypoints comprised in the second spatial relationship (Williams et al, keypoints are used to perform SLAM implementation between frames (query and reference) and keypoints represent feature locations, used to map the (geometry of) features of the first image to a second image (for tracking) with features mapping (comparing geometry) for relocalising; Fig 5 and 4.2 Randomised Lists for SLAM recovery, 6.2 SLAM with Recovery ¶ 3-8, Table 3; examiner notes Zhang et al likewise teaches a spatial triangulation technique used for comparing relationships between keypoints in the query and reference images based on graph matching; Fig 2 and III.B. Sketch-and-Match ¶ 3-6).
Regarding Claim 6, Zhang et al in view of Williams et al teach the method of claim 1 (as described above), further comprising: selecting the one or more additional query image keypoints by identifying spatial neighbors to the first query image keypoint from the set of query image keypoints (Zhang et al, spatial nearness of keypoints are identified in a given (query image) triangulated graph and the full set of edges (triangles) between the points are used to create the sketch; Fig 2 and III.B. Sketch-and-Match ¶ 3-4).
Regarding Claim 9, Zhang et al teach a non-transitory computer readable medium comprising computer readable code executable by one or more processors (Computing machine with 8-core 2.67 GHz processor and 128 memory were used to store instructions and perform the steps for the Instance Search; III.F. Speed Efficiency) to: perform steps identical to claim 1 (as discussed above).
Regarding Claim 10, Zhang et al in view of Williams et al teach the non-transitory computer readable medium of claim 9 (as described above), with further steps identical to claim 2 (as described above).
Regarding Claim 11, Zhang et al in view of Williams et al teach the non-transitory computer readable medium of claim 10 (as described above), with further steps identical to claim 3 (as described above).
Regarding Claim 12, Zhang et al in view of Williams et al teach the non-transitory computer readable medium of claim 11 (as described above), with further steps identical to claim 4 (as described above).
Regarding Claim 13, Zhang et al in view of Williams et al teach the non-transitory computer readable medium of claim 9 (as described above), with further steps identical to claim 5 (as described above).
Regarding Claim 14, Zhang et al in view of Williams et al teach the non-transitory computer readable medium of claim 9 (as described above), with further steps identical to claim 6 (as described above).
Regarding Claim 17, Zhang et al teach a system (Computing machine; III.F. Speed Efficiency) comprising: one or more processors (Computing machine with 8-core 2.67 GHz processor; III.F. Speed Efficiency); and one or more computer readable media comprising computer readable code executable by the one or more processors (Computing machine with 8-core 2.67 GHz processor and 128 memory were used to store instructions and perform the steps for the Instance Search; III.F. Speed Efficiency) to: perform steps identical to claim 1 (as discussed above).
Regarding Claim 18, Zhang et al in view of Williams et al teach the system of claim 17 (as described above), with further steps identical to claim 2 (as described above).
Regarding Claim 19, Zhang et al in view of Williams et al teach the system of claim 17 (as described above), with further steps identical to claim 5 (as described above).
Regarding Claim 20, Zhang et al in view of Williams et al teach the system of claim 17 (as described above), with further steps identical to claim 6 (as described above).
Claims 7, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al (Topological Spatial Verification for Instance Search) in view of Williams et al (Real-Time SLAM Relocalisation) and Gonzalez-Diaz et al (Neighborhood Matching for Image Retrieval).
Regarding Claim 7, Zhang et al in view of Williams et al teach the method of claim 6 (as described above).
Zhang et al in view of Williams et al does not teach filtering a set of initial keypoint matches by applying a threshold constraint between a nearest neighbor distance and a next nearest neighbor distance among the set of initial keypoint matches to obtain the set of keypoint matches.
Gonzalez-Diaz et al is analogous art pertinent to the technological problem addressed in the current application and teaches filtering a set of initial keypoint matches by applying a threshold constraint between a nearest neighbor distance and a next nearest neighbor distance among the set of initial keypoint matches to obtain the set of keypoint matches (nearest neighborhood matching is perform with the SIFT feature points contained within a threshold distance of the identified neighborhood with a strong matched point as the center of the neighborhood; Fig 1 and III. Neighborhood Matching, IV.A.2. Neighborhood Matching for Image Retrieval – Geometric Verification ¶ 1-2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Zhang et al in view of Williams et al with Gonzalez-Diaz et al including filtering a set of initial keypoint matches by applying a threshold constraint between a nearest neighbor distance and a next nearest neighbor distance among the set of initial keypoint matches to obtain the set of keypoint matches. By performing the neighborhood matching, false matches are filtered while maintaining strong matches, thereby avoiding iterative and time-consuming estimation of points, leading to stronger reliability in the identified matches, as recognized by Gonzalez-Diaz et al (II. Related Work ¶ 7, IV.B Neighborhood Matching for Image Retrieval – Geometrical Verification ¶ 5).
Regarding Claim 15, Zhang et al in view of Williams et al teach the non-transitory computer readable medium of claim 14 (as described above), with further steps identical to claim 7 (as described above).
Claims 8, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al (Topological Spatial Verification for Instance Search) in view of Williams et al (Real-Time SLAM Relocalisation) and Li et al (EMOVIS – An Efficient Mobile Visual Search System for Landmark Recognition).
Regarding Claim 8, Zhang et al in view of Williams et al teach the method of claim 1 (as described above), including obtaining the set of keypoint matches (matches of keypoints (visual words) from query and reference image based on Sketch-and-Match; Fig 2 and III.A. Topological Spatial Verification, Spatial Configuration ¶ 1, III.B. Sketch-and-Match ¶ 5-6).
Zhang et al in view of Williams et al does not teach obtaining a set of initial query image keypoints from the query image; cropping one or more patches from the query image according to the set of initial query image keypoints; and obtaining the set of keypoint matches based on the set of initial query image keypoints within the one or more patches.
Li et al is analogous art pertinent to the technological problem addressed in the current application and teaches obtaining a set of initial query image keypoints from the query image (the image query image are extracted from a query image; Fig 2, 6 and IV.C. Query Image Preprocessing ¶ 1-2); cropping one or more patches from the query image according to the set of initial query image keypoints (the query image is cropped to remove irrelevant regions (interpreted the cropped ROI is the patch of interest); Fig 2, 6 and IV.C. Query Image Preprocessing ¶ 1-3); and obtaining the set of keypoint matches based on the set of initial query image keypoints within the one or more patches (matching is performed with the keypoints in the cropped query and database (reference) image; Fig 2, 6 and IV.C. Query Image Preprocessing ¶ 3-4).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Zhang et al in view of Williams et al with Li et al including obtaining a set of initial query image keypoints from the query image; cropping one or more patches from the query image according to the set of initial query image keypoints; and obtaining the set of keypoint matches based on the set of initial query image keypoints within the one or more patches. By using a cropping function on the query image, the unnecessary regions not of interest are removed and the computing is focused only on the selected keypoints in the region of interest, thereby reducing computational requirements and improving analysis speed without changing matching outcomes, as recognized by Li et al (IV.C. Query Image Preprocessing ¶ 4).
Regarding Claim 16, Zhang et al in view of Williams et al teach the non-transitory computer readable medium of claim 9 (as described above), with further steps identical to claim 8 (as described above).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Huang et al (US 2022/0092334, application 17/410,630), from the same assignee/co-inventors for analyzing query image keypoints and matched to reference keypoints but does not teach the same invention as the current invention with application ‘630 including a feature graph comprising a geometric spatial relationship between first keypoints and second keypoints and performing feature graph comparisons with reference feature graphs.
Tang et al (US 2021/0326601) teach a method and system for keypoint matching between a first and second image based on geometrical similarities between the points.
Taguchi et al (Point-Plane SLAM for Hand-Held 3D Sensors) teach a method and system for SLAM analysis using both keypoints and key planes to perform matching in a 3D model.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHLEEN M BROUGHTON whose telephone number is (571)270-7380. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATHLEEN M BROUGHTON/Primary Examiner, Art Unit 2661
Prosecution Insights