DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for priority under 35 U.S.C. 119 (e). The provisional Application No. 63/599,527, filed on 11/15/2023.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/14/2024 was filed and has been considered by the examiner.
Drawings
The drawings that were filed on 11/14/2024 have been considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-4, 6-9, and 12-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kimia et al. (US 20230392944 A1), and herein after will be referred to as Kimia, in view of Tang et al. (US 20210318140 A1), herein after will be referred to as Tang.
Regarding Claim 1, Kimia teaches a navigation system comprising (A navigation system employing techniques to determine location and instructions for outdoor and indoor environments; [0029]):
one or more processing circuits configured to: process vision data of a space (The system comprises at least one processor receiving and processing image data from cameras; [0006]);
generate, based on the vision data, a map of the space (Generating a map by utilizing movement data and imagery acquired while navigating the space; [0038] [0104]);
compare a feature of the…image with one or more features of the vision data to generate a comparison between the…image and the vision data (Identifying features of the image data and comparing them to previous acquired data to generate a match result; [0049] [0083]);
determine, based on the comparison between the…image and the vision data, a location and a direction associated with the…image within the space (Determining the location and/or orientation from matching the image data to the acquired image data; [0049]);
associate the location and the direction associated with the…image with a position and an orientation within the map (Correlating the determined physical location of the user to a position onto the map; [0102]); and
compute, based on the position and the orientation within the map, a path through the space to a destination (The navigation module generates a path from the position/orientation to guide the user to a destination; [0064]).
Kimia does not explicitly teach the terminology of receiving a query image of the space.
However, Tang discloses a visual localization system by an agent that receives a query image of the current environment and matches a set of keypoints to a target image with the query image ([0004] [0006]). This teaching is equivalent to the claimed limitation because the system captures an image of the current environment to identify and match keypoints of an image for localization.
Kimia and Tang are considered to be analogous to the claim invention because they are in the same field of vision-based navigation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to modify Kimia to incorporate the teachings of the query image reception as taught by Tang based on a reasonable expectation of success and the motivation to improve the captured images for comparison against a target dataset for the feature comparison and best match determination disclosed by Kim ([0032]).
Regarding Claim 2, Kimia and Tang remains as applied above in claim 1. Kimia further teaches the vision data includes a plurality of reference images stored in a reference image database, each reference image of the plurality of reference images associated with a location within the space (A repository containing previously acquired image data associated with spatial locations; [0051]).
Regarding Claim 3, Kimia and Tang remains as applied above in claim 2. Kimia further teaches the one or more processing circuits are configured to generate a three dimensional map of the space based on the plurality of reference images and the locations associated therewith (A map includes 3-dimensional data based on nodes that represent the location and image data associated with the node; [0063]).
Regarding Claim 4, Kimia and Tang remains as applied above in claim 3. Kimia does not explicitly teach the one or more processing circuits are configured to extract descriptors from the plurality of reference images and calculate a direction of each reference image of the plurality of reference images to generate the three dimensional map.
However, Tang discloses a visual mapping system that utilizes learned descriptors and keypoints that is used to generate a globally consistent map where each keypoint respective descriptor is a string determined based on the features of the image ([0028] [0034]). Tang further teaches utilizing a robust estimator to calculate a 6-DoF rigid pose transformation between the target and context views then performs pose estimation by lifting the 2D keypoints from the target image 3D with the associated depth map variable ([0048]). These teachings are equivalent to the claimed limitation because the system uses the keypoints and descriptors from the images and calculates the pose (direction) of reference images to construct the 3D map. It would have been obvious to one having ordinary skill in the art at the time the invention was made to modify Kimia to incorporate the teachings of the descriptor extraction and pose calculation to construct a 3D map as taught by Tang based on the motivation to build accurate 3D maps from 2D images by determining the orientation of each frame relative to other frames. This provides the benefit of improving localization when reference images are captured from different viewpoints.
Regarding Claim 6, Kimia and Tang remains as applied above in claim 5. Kimia further teaches the query image is added to the reference image database as a reference image (Newly acquired image data is stored in the repository to expand the reference image data available to the system; [0052] [0104]).
Regarding Claim 7, Kimia and Tang remains as applied above in claim 1. Kimia further teaches the comparison between the query image and the vision data is generated by calculating…a similarity score between the feature of the query image and the one or more features of the vision data (Determining a measure of similarity or match score between the features detected in image data previously acquired; [0083]).
Kimia does not explicitly teach calculating a Euclidean distance.
However, Tang discloses a method for image localization using distance metrics by sorting images based on Euclidean distance between an image of the database and the quest image ([0032]). This teaching is equivalent to the claimed limitation because the process of matching the keypoints of the query image includes using Euclidean distance to sort the images for the nearest neighbor search. It would have been obvious to one having ordinary skill in the art at the time the invention was made to modify Kimia to incorporate the teachings of calculating the Euclidean distance as taught by Tang based on a reasonable expectation of success and the motivation to further enhance the best match or match score when comparing the images. This provides the benefit of improved retrieval of the closest match reference image from the database by sorting images based on a Euclidean distance with the query image.
Regarding Claim 8, Kimia and Tang remains as applied above in claim 7. Kimia further teaches the reference images of the vision data determined by the one or more processing circuits, based on the comparison, to be similar to the query image are selected as candidate images (Producing an N-best list of the closest matches based on the comparison; [0052]).
Regarding Claim 9, Kimia and Tang remains as applied above in claim 8. Kimia further teaches the one or more processing circuits are configured to determine, based on known positions and orientations of the candidate images, the location and the direction associated with the query image within the space (Using the stored associated information identifying the location and/or orientation of the multiple matches to interpolate and estimate the user’s current location and direction; [0051-0052]).
Regarding Claim 12, Kimia and Tang remains as applied above in claim 1. Kimia further teaches the query image is captured by a user device of a user, and wherein the location and the direction associated with the query image is indicative of a location and a direction of the user within the space (Newly acquired image data captured by the image acquisition module is used to determine the location and/or orientation of the user; [0049] [0051]).
Regarding Claim 13, Kimia and Tang remains as applied above in claim 12. Kimia further teaches the one or more processing circuits are configured to provide instructions, by the user device, regarding the path through the space to the destination, wherein the instructions include at least one of an audible feedback (Generating a path to a destination and providing navigation commands using the user device via audible directions and haptic gestures; [0064]).
Regarding Claim 14, Kimia and Tang remains as applied above in claim 1. Kimia further teaches the path includes a first section of the path to a location of a reference image that is closest to the location associated with the query image and a second section of the path from the location of the reference image to the destination (The path is generated in small increments as a sequence of nodes from the starting point identified via image matching corresponding to the first section of the path starting from the user’s localized position to subsequent sections (second section) of the path; [0064] [0102]).
Regarding Claim 15, Kimia and Tang remains as applied above in claim 1. Kimia further teaches the vision data is captured by a sensor of at least one of a user device (Previously acquired image data is captured by others navigating the space using image acquisition devices with sensors on their wearable systems; [0032]).
Regarding Claim 16, Kimia and Tang remains as applied above in claim 15. Kimia does not explicitly teach the vision agent includes an aerial drone, a ground vehicle, a robotic quadruped, or a robotic biped.
However, Tang discloses a visual positioning system where the localization system can be a component of agents such as a bus, boat, drone, or robot ([0073]). This teaching is equivalent to the claimed limitation because the image localization system can be implemented on a ground vehicle, drone, and a robot. It would have been obvious to one having ordinary skill in the art at the time the invention was made to modify Kimia to incorporate the teachings of the image localization system as a component of a bus, drone, or robot as taught by Tang based on the motivation to automate the data collection process for the map database allowing for mapping environments that may be hazardous or inaccessible for human users to navigate. This provides the benefit of enhancing the safety of the data collection process and expands the amount of data that can be collected by using an aerial drone.
Regarding Claim 17, Kimia and Tang remains as applied above in claim 15. Kimia further teaches the one or more processing circuits includes at least one of (i) a first processing circuit located on the user device (A system manager as a first processing circuit located on the wearable user device; [0043-0044]).
Regarding Claim 18, Kimia and Tang remains as applied above in claim 1. Kimia further teaches the space includes at least one of an indoor space or an outdoor space (The navigation system is designed for use in both indoor and outdoor environments; [0029] [0040]).
Regarding Claim 19, Kimia teaches a navigation system comprising (A navigation system employing techniques to determine location and instructions for outdoor and indoor environments; [0029]):
a first sensor configured to acquire vision data of a space (The stored vision data used for the map is acquired by image acquisition devices during their prior or initial visits; [0032]);
a user device including a second sensor configured to acquire a…image associated with a location and a direction that are indicative of a location and a direction of a user within the space (A wearable user device with mounted cameras that captures images and determines the user’s current location and/or orientation; [0044] [0049]);
one or more processing circuits configured to: generate, based on the vision data, a map of the space (Map module configured to create and update maps based on acquired data; [0062]);
compare a feature of the…image with features of one or more reference images of the vision data to generate a comparison between the…image and the one or more reference images (Identifying features of the image data and comparing them to previous acquired data to generate a match result; [0049] [0083]);
determine, based on the comparison between the…image and the one or more reference images, the location and the direction associated with the…image within the space (Determining the location and/or orientation from matching the image data to the acquired image data; [0049]);
associate the location and the direction associated with the…image with a position and an orientation within the map (Correlating the determined physical location of the user to a position onto the map; [0102]); and
compute, based on the position and the orientation within the map, a path through the space to a destination (The navigation module generates a path from the position/orientation to guide the user to a destination; [0064]).
Kimia does not explicitly teach receive, from the user device, the query image of the space.
However, Tang discloses a visual localization system by an agent that receives a query image of the current environment and matches a set of keypoints to a target image with the query image ([0004] [0006]). Tang further teaches that the modules of the system can be obtained by a user terminal ([0109]). This teaching is equivalent to the claimed limitation because the user terminal captures an image of the current environment to query a database for localization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to modify Kimia to incorporate the teachings of the query image reception as taught by Tang based on a reasonable expectation of success and the motivation to improve the captured images for comparison against a target dataset for the feature comparison and best match determination disclosed by Kim ([0032]).
Regarding Claim 20, Kimia teaches a method for environment mapping and navigation, the method comprising (A navigation system employing techniques to determine location and instructions for outdoor and indoor environments; [0029]):
processing vision data of a space (The system comprises at least one processor receiving and processing image data from cameras; [0006]);
generating, based on the vision data, a map of the space (Map module configured to create and update maps based on acquired data; [0062]);
comparing a feature of a…image associated with a location and a direction within the space with features of a reference image of the vision data to generate a comparison between the…image and the reference image (Features of the image data are identified and compared to previously acquired image data which location and/or orientation of acquisition is known; [0049] [0083]);
determining, based on the comparison between the…image and the reference image, the location and the direction associated with the…image within the space (Determining the location and/or orientation from matching the image data to the acquired image data; [0049]);
associating the location and the direction associated with the…image with a position and an orientation within the map (Correlating the determined physical location of the user to a position onto the map; [0102]); and
computing, based on the position and the orientation within the map, a path through the space to a destination (The navigation module generates a path from the position/orientation to guide the user to a destination; [0064]).
Kimia does not explicitly use or teach a query image.
However, Tang discloses a visual localization system by an agent that explicitly receives a query image of the current environment and matches a set of keypoints to a target image with the query image ([0004] [0006]). This teaching is equivalent to the claimed limitation because the system captures an image of the current environment to identify and match keypoints of an image for localization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to modify Kimia to incorporate the teachings of the query image reception as taught by Tang based on a reasonable expectation of success and the motivation to improve the captured images for comparison against a target dataset for the feature comparison and best match determination disclosed by Kim ([0032]).
Claim(s) 5, 10, and 11 is rejected under 35 U.S.C. 103 as being unpatentable over Kimia in view of Tang, as applied in claim 4, and in further view of Zhao et al. (US 20210110615 A1), herein after will be referred to as Zhao.
Regarding Claim 5, Kimia and Tang remains as applied above in claim 4. Kimia and Tang does not explicitly teach the map is a topometric map, and wherein the one or more processing circuits are configured to correspond three dimensional coordinates associated with the descriptors to two dimensional coordinates of a floor plan of the space to generate the topometric map of the space.
However, Zhao discloses a cross reality system that supports multiple types with localization capabilities. Zhao teaches that the localization process where accuracy of matching features can be improved by projecting 3D cluster of features into a plane and performing a 3D mapping within that plane ([0544]. Zhao further teaches that the tracking map provides a floor plan of physical objects that correspond to the physical world ([0210]). These teachings are equivalent to the claimed limitation because it describes the process of taking 3D data (3D cluster of features) and projecting it onto a 3D surface (plane) to create the floor plan representation for the map (FIG. 7).
Kimia, Tang, and Zhao are considered to be analogous to the claim invention because they are in the same field of vision-based navigation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to modify Kimia and Tang to incorporate the teachings of the 2D projection from 3D data and floor plan generation as taught by Zhao based on the motivation to improve the matching features extracted from devices with a monocular camera. This provides the benefit of improving the localization for devices with only a single lens to capture images.
Regarding Claim 10, Kimia and Tang remains as applied above in claim 1. Kimia and Tang does not explicitly teach identify a network characteristic associated with the query image, the network characteristic indicative of the location and the direction associated with the query image within the space; and determine, based on (i) the comparison between the query image and the vision data and (ii) the network characteristic, the location and the direction associated with the query image within the space.
However, Zhao discloses a localization system that utilizes multiple data sources. Zhao teaches that the native framework provides information of the device’s camera images alongside the position, movement and/or orientation used for localization and data on detected network access points within the 3d environment of the device ([0518]). This teaching is equivalent to the claimed limitation of identify a network characteristic associated with the query image, the network characteristic indicative of the location and the direction associated with the query image within the space because the network characteristic (data on access points) is processed by the native AR framework to assist in determining the device’s orientation and localization. Zhao teaches that the system performs a comparison (match to a feature) and uses the matches to compute a relative transformation required to localize the device ([0491]). Zhao further teaches using network characteristics, such as Wi-Fi signal data or Wi-Fi finger print information, to filter the maps and limit the analysis to only relevant maps for localization ([0459]).
These teachings are equivalent to the claimed limitation of determine, based on (i) the comparison between the query image and the vision data and (ii) the network characteristic, the location and the direction associated with the query image within the space because the localization is derived from the visual match (comparison) and is performed on the specific set of maps restricted by the Wi-Fi finger print. It would have been obvious to one having ordinary skill in the art at the time the invention was made to modify Kimia and Tang to incorporate the teachings of identifying network access points to assist with localization as taught by Zhao based on the motivation to improve the efficiency and accuracy of the localization by limiting analysis of descriptors to only relevant maps and avoid searching irrelevant data.
Regarding Claim 11, Kimia and Tang remains as applied above in claim 10. Kimia and Tang does not explicitly teach the one or more processing circuits are configured to disregard a reference image of the vision data determined to be similar to the query image when the one or more processing circuits determine, based on the network characteristic, that a location of the reference image is geographically dissimilar from the location of the query image within the space.
However, Zhao discloses a localization system that filters map data based on network characteristics. Zhao teaches geographic data can be used to limit a search for matching and localization purpose by area attributes such as Wi-Fi signal data and Wi-Fi fingerprint information ([0459]). This teaching is equivalent to the claimed limitation because the system uses network characteristics to filter and disregard reference images from locations that are geographically different. It would have been obvious to one having ordinary skill in the art at the time the invention was made to modify Kimia and Tang to incorporate the teachings of using Wi-Fi characteristics to filter the geographic location of reference images as taught by Zhao based on the motivation to prevent false positives in visual matching where images contain similar features and characteristics but are in different locations. This provides the benefit of improving the reliability of localization by ensuring visual matches are matched to the physical location.
Prior Art
The prior art made of record and not relied upon is considered pertinent, most relevant, to applicant's disclosure.
Streem (US 20230400327 A1)
Spiegel (US 20200401617 A1)
Sheng (US 20250137813 A1)
Senthamil (US 20180005393 A1)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EDWARD ANDREW IZON DIZON whose telephone number is (571)272-4834. The examiner can normally be reached M-F 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Angela Ortiz can be reached at (571) 272-1206. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EDWARD ANDREW IZON DIZON/Examiner, Art Unit 3663
/ANGELA Y ORTIZ/Supervisory Patent Examiner, Art Unit 3663