DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 29 January 2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Status of Application
Claims 1-24 are pending.
Claims 1, 7, 13, and 19 are independent.
Claims 1, 2, 4-8, 10, 11, 13, 14, 16-20, and 22-24 have been amended
This FINAL action is in response to “Amendments and Remarks” received on 29 January 2026.
Response to Amendment/Remarks
With respect to Applicant’s remarks filed 29 January 2026, Applicant’s “Amendments and Remarks” have been fully considered and were not wholly persuasive. Applicant’s remarks will be addressed in sequential order as they were presented.
With respect to objection of the claims, Applicant’s “Amendments and Remarks” have been fully considered and are persuasive. Therefore, the objections to the Specification have been withdrawn.
With respect to claim interpretations under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, Applicant’s “Amendments and Remarks” have been fully considered and are persuasive. Therefore, the interpretation is withdrawn. Applicant is correct that structures identified in the Office Action mailed 04 November 2025 must also include equivalents thereof. However, claims 13-18 contain “means for” language which invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. See below for analysis.
With respect to claim rejections under 35 U.S.C. 102 and/or 35 U.S.C. 103, Applicant’s “Amendments and Remarks” have been fully considered and are persuasive. Therefore, the rejection is withdrawn. However, upon further consideration, there is a new ground(s) of rejection made in view of newly found prior art.
Final Office Action
Claim Interpretation
During examination, claims are given the broadest reasonable interpretation consistent with the specification and limitations in the specification are not read into the claims. See MPEP §2111, MPEP §2111.01 and In re Yamamoto et al., 222 USPQ 934 10 (Fed. Cir. 1984). Under a broadest reasonable interpretation, words of the claim must be given their plain meaning, unless such meaning is inconsistent with the specification. See MPEP 2111.01 (I). It is further noted it is improper to import claim limitations from the specification, i.e., a particular embodiment appearing in the written description may not be read into a claim when the claim language is broader than the embodiment. See 15 MPEP 2111.01 (II).
A first exception to the prohibition of reading limitations from the specification into the claims is when the Applicant for patent has provided a lexicographic definition for the term. See MPEP §2111.01 (IV). Following a review of the claims in view of the specification herein, the Office has found that Applicant has not provided any lexicographic definitions, either expressly or implicitly, for any claim terms or phrases with any reasonable clarity, deliberateness and precision. Accordingly, the Office concludes that Applicant has not acted as his/her own lexicographer.
A second exception to the prohibition of reading limitations from the specification into the claims is when the claimed feature is written as a means-plus-function. See 35 U.S.C. §112(f) and MPEP §2181-2183. As noted in MPEP §2181, a three-prong test is used to determine the scope of a means-plus-function limitation in a claim:
(A) the claim limitation uses the term "means" or "step" or a term used as a substitute for "means" that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function
(B) the term "means" or "step" or the generic placeholder is modified by functional language, typically, but not always linked by the transition word "for" (e.g., "means for") or another linking word or phrase, such as "configured to" or "so that"
(C) the term "means" or "step" or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
The Office has found herein that the claims contain limitations of means or means type language that must be analyzed under 35 U.S.C. §112 (f).
Claim limitation “means for maneuvering” in claim 13 has been evaluated under the three-prong test set forth in MPEP § 2181, subsection 1. The limitation invokes 35 U.S.C. 112(f) and is interpreted in view of [0032] of the Specification, wherein “Example robotic device 200 is illustrated as a ground vehicle design that utilizes one or more wheels 202 driven by corresponding motors to provide locomotion to the robotic device 200. The illustration of robotic device 200 is not intended to imply or require that various embodiments are limited to ground robotic devices. For example, various embodiments may be used with rotorcraft or winged robotic devices, water-borne robotic devices, and space-based robotic devices,” or equivalents thereof.
Claim limitation “means for determining a location” in claim 13 has been evaluated under the three-prong test set forth in MPEP § 2181, subsection 1. The limitation invokes 35 U.S.C. 112(f) and is interpreted in view of [0039] of the Specification, wherein The robotic device 200 may be controlled through control of the individual motors of the rotors 202 as the robotic device 200 progresses toward a destination. The processor 220 may receive data from the navigation unit 222 and use such data in order to determine the present position and orientation of the robotic device 200, as well as the appropriate course towards the destination or intermediate sites. In various embodiments, the navigation unit 222 may include a GNSS receiver system (e.g., one or more global positioning system (GPS) receivers) enabling the robotic device 200 to navigate using GNSS signals. Alternatively or in addition, the navigation unit 222 may be equipped with radio navigation receivers for receiving navigation beacons or other signals from radio nodes, such as navigation beacons (e.g., very high frequency (VHF) omni-directional range (VOR) beacons), Wi-Fi access points, cellular network sites, radio station, remote computing devices, other robotic devices, etc.,” or equivalents thereof.
Claim limitation “means for performing visual loop closure detection” in claim 13 has been evaluated under the three-prong test set forth in MPEP § 2181, subsection 1. The limitation invokes 35 U.S.C. 112(f) and is interpreted in view of [0050] of the Specification wherein “The processing system 400 may be configured by machine-readable or processor-executable instructions that may include one or more instruction modules. The instruction modules may include computer program modules. The instruction modules may include one or more of a robotic device maneuvering module 402, an R TT measurement module 404, a location determining module 406, a camera module 408, and a visual loop closure detection module 410, as well as other instruction modules.” And in view of [0053] of the Specification wherein “The location determining module 406 may be configured to determine a location of the robotic device, for example, based on an R TT measurement of a signal sent by the robotic device to an access point. The location determining module 408 may be configured to determine whether a location of the robotic device is within a threshold distance of a previously-visited one of the plurality of locations,” or equivalents thereof.
Claim limitation “means for selecting a plurality of keyframes” in claim 14 has been evaluated under the three-prong test set forth in MPEP § 2181, subsection 1. The limitation invokes 35 U.S.C. 112(f) and is interpreted in view of [0054] of the Specification wherein “The camera module 408 may be configured to capture and process images using a camera of the robotic device. The camera module 408 may be configured to select keyframes from among images captured using the camera,” or equivalents thereof.
Claim limitation “means for determining the location of the robotic device coincident with each of the plurality of keyframes” in claim 14 has been evaluated under the three-prong test set forth in MPEP § 2181, subsection 1. The limitation invokes 35 U.S.C. 112(f) and is interpreted in view of [0070] of the Specification wherein “the processor may determine the location of the robotic device coincident with each of the plurality of keyframes,” or equivalents thereof.
Claim limitation “means for associating with each respective keyframe the determining location” in claim 15 has been evaluated under the three-prong test set forth in MPEP § 2181, subsection 1. The limitation invokes 35 U.S.C. 112(f) and is interpreted in view of [0072] of the Specification wherein “the processor may determine whether a location of the robotic device that is associated with a keyframe is within a threshold distance of a previously-visited one of the plurality of locations associated with another keyframe,” or equivalents thereof.
Claim limitation “means for determining whether a location of the robotic device associated with a keyframe is within a threshold distance” in claim 16 has been evaluated under the three-prong test set forth in MPEP § 2181, subsection 1. The limitation invokes 35 U.S.C. 112(f) and is interpreted in view of [0072] of the Specification wherein “the processor may determine whether a location of the robotic device that is associated with a keyframe is within a threshold distance of a previously-visited one of the plurality of locations associated with another keyframe,” or equivalents thereof.
Claim limitation “means for comparing a determined location of the robotic device with every previously-determined location” in claim 17 has been evaluated under the three-prong test set forth in MPEP § 2181, subsection 1. The limitation invokes 35 U.S.C. 112(f) and is interpreted in view of [0065] of the Specification wherein “the processor may compare the determined location of the robotic device with every previously-determined location of the robotic device while maneuvering the robotic device through the plurality of locations,” or equivalents thereof.
Claim limitation “means performing a Wi-Fi RTT measurement” in claim 18 has been evaluated under the three-prong test set forth in MPEP § 2181, subsection 1. The limitation invokes 35 U.S.C. 112(f) and is interpreted in view of [0064] of the Specification wherein “the processor may perform a Wi-Fi RTT measurement to determine the location of the robotic device,” or equivalents thereof.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-24 are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. (US 20190094027 A1), hereinafter Xu, in view of Brunner et al. (US 20150092048 A1), hereinafter Brunner, and further in view of Nowakowski et al. (US 20200236504 A1), hereinafter Nowakowski.
Regarding claim 1, Xu discloses:
A method performed by a processing device of a robotic device for loop closure detection, comprising (Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 7B; [0028], Also during such movement in such a learning mode, a series of pathways may be derived that, together, define the map. Some aspects of the map, including locations of pathway intersections where loops in the map are closed, may be derived through use of key frames taken from the two separate sets ofkey frames as those sets are still being generated. As the pathways are derived, each captured frame that is selected to become a key frame in one or both of the sets of key frames may be correlated to a location along one of the pathways of the map. Other aspects of the map, including directions and/or lengths of pathways, may be derived through use of the one or more location sensors based on any of a variety of sensing technologies. Where possible, in some embodiments, locations along one or more of the pathways of the map may be associated with an identifier of a physical location within the defined area. Overall, in generating the map, it may be deemed desirable to make use of both key frames and the location sensors in generating the map. This may be due to limitations in the degree of positioning accuracy that may be achievable through use of the location sensors, alone):
maneuvering the robotic device through a plurality of locations ([0026], The mobile device may be self propelled (e.g., motorized with wheels to move itself about) or may be carried about by another device and/or by a person. The mobile device may be equipped with one or more cameras to recurringly capture images of the surroundings of the mobile device as the mobile device is moved about within the defined area. Various approaches may be used to incorporate the one or more cameras into the mobile device to enable capturing images in more than one direction from the current position of the mobile device. In some embodiments, the mobile device may also be equipped with any of a variety of types of location sensors, such as one or more radio frequency (RF) receivers that may receive signals from multiple positioning satellites orbiting the Earth and/or multiple stationary position devices (e.g., wireless access points), one or more accelerometers and/or one or more gyroscopes to provide inertial guidance, and/or readers of barcodes and/or RFID tags that may be positioned at various locations within the defined area. In some embodiments, the mobile device may be equipped with a wireless network interface to enable communication between the mobile);
and performing visual loop closure detection in response to determining that the location of the robotic device associated with the first keyframe is within the threshold distance of the previously-visited one of the plurality of locations that is associated with the second keyframe ([0026], The mobile device may be self propelled (e.g., motorized with wheels to move itself about) or may be carried about by another device and/or by a person. The mobile device may be equipped with one or more cameras to recurringly capture images of the surroundings of the mobile device as the mobile device is moved about within the defined area. Various approaches may be used to incorporate the one or more cameras into the mobile device to enable capturing images in more than one direction from the current position of the mobile device. In some embodiments, the mobile device may also be equipped with any of a variety of types of location sensors, such as one or more radio frequency (RF) receivers that may receive signals from multiple positioning satellites orbiting the Earth and/or multiple stationary position devices (e.g., wireless access points), one or more accelerometers and/or one or more gyroscopes to provide inertial guidance, and/or readers of barcodes and/or RFID tags that may be positioned at various locations within the defined area. In some embodiments, the mobile device may be equipped with a wireless network interface to enable communication between the mobile).
However, Xu does not specifically state:
determining, using a round trip time (RTT) measurement of at least one signal sent to an access point, a location of the robotic device at each of the plurality of locations coincident with each of a plurality of keyframes from images using a camera of the robotic device;
determining whether the location of the robotic device associated with a first keyframe is within a threshold distance of a previously-visited one of the plurality of locations that is associated with a second keyframe;
Brunner teaches:
determining, using a round trip time (RTT) measurement of at least one signal sent to an access point, a location of the robotic device at each of the plurality of locations coincident with each of a plurality of keyframes from images using a camera of the robotic device (Abstract, A Visual Inertial Tracker (VIT), such as a Simultaneous Localization And Mapping (SLAM) system based on an Extended Kalman Filter (EKF) framework (EKF-SLAM) can provide drift correction in calculations of a pose (translation and orientation) of a mobile device by obtaining location information regarding a target, obtaining an image of the target, estimating, from the image of the target, measurements relating to a pose of the mobile device based on the image and location information, and correcting a pose determination of the mobile device using an EKF, based, at least in part, on the measurements relating to the pose of the mobile device; [0041], The process can start by receiving a camera image at block 310. The type of camera image can vary in resolution, color, and/or other characteristics, depending on desired functionality, camera hardware, and/or other factors. Moreover, the camera image may be a discrete still image or may be one of several frames of video captured by the camera. In some embodiments, the image may be processed to a degree before it is received by a VIT, to facilitate further image processing by the VIT; [0042], At block 320, the VIT optionally receives WiFi signals, which can facilitate the determination of which targets may be included in the received image. For example, wireless signals may be utilized together with a map of a venue that includes the identity and locations of WiFi access points. If the locations of certain access points can be determined from the map, the VIT can get a rough estimate of where in the venue the VIT (and any mobile device associated therewith) is. The VIT can do this by measuring WiFi signals received from the WiFi access points by the mobile device (e.g., measuring received signal strength (RSSI), round-trip time (RTT), and/or other measurements) to determine a proximity of the access points--including which access points may be closest. This can then be compared with the map to determine a region in the venue in which the mobile device is located.);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Brunner into the invention of Xu to include collect WiFi signal data as well as images captured by a camera to determine a pose and position of a mobile device as Brunner discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can corroborate RTT measurements from known access points with image data to perform visual loop closure detection. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu and using RTT measurements to localize a device as taught by Brunner. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
However, Xu in view of Brunner does not specifically state:
determining whether the location of the robotic device associated with a first keyframe is within a threshold distance of a previously-visited one of the plurality of locations that is associated with a second keyframe;
Nowakowski teaches:
determining whether the location of the robotic device associated with a first keyframe is within a threshold distance of a previously-visited one of the plurality of locations that is associated with a second keyframe ([0081], The distance between visual and radio observations can be calculated for example by odometry, or by comparing timestamps of observations, and adding to a radio observation vector a visual observation vector which is the closest in time; [0087], In order for the visual and radio observations to be consistent, the visual and radio observations need to be performed within the same time frame. In a number of embodiments of the invention, an observation vector is formed from visual and radio observations if their capture time is below a predefined threshold. Usually, visual observations are performed at a higher rate than radio observations. In a number of embodiments of the invention, radio observations indicating the receivable radio transmitters are performed periodically by the adaptation 242, and an observation vector is created, at each time step, from the radio observation and the visual observation which has been captured by the one or more image sensors 210 at the closest time. The determination of the visual observation that has been captured at the closest time can be performed for example by comparing timestamps of visual and radio observations.);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Nowakowski into the invention of Xu in view of Brunner to include measuring distance between visual and radio measurement within a threshold time to measure distance between measurements as Nowakowski discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can measure distance traveled between datapoints. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu in view of Brunner and comparing visual and radio observations as taught by Nowakowski. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 2, Xu in view of Brunner and Nowakowski teaches:
selecting the plurality of keyframes from among images captured using the camera of the robotic device (Xu: Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 3A; Fig. 3B);
Regarding claim 3, Xu in view of Brunner and Nowakowski teaches:
associating with each respective keyframe the determined location of the robotic device coincident with each of the plurality of keyframes (Xu: Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 3A; Fig. 3B).
Regarding claim 4, Xu in view of Brunner and Nowakowski teaches:
wherein determining whether the location of the robotic device is within the threshold distance of a previously-visited one of the plurality of locations comprises determining whether a location of the robotic device that is associated with a keyframe is within a threshold distance of a previously-visited one of the plurality of locations associated with another keyframe (Nowakowski: [0081], The distance between visual and radio observations can be calculated for example by odometry, or by comparing timestamps of observations, and adding to a radio observation vector a visual observation vector which is the closest in time; [0087], In order for the visual and radio observations to be consistent, the visual and radio observations need to be performed within the same time frame. In a number of embodiments of the invention, an observation vector is formed from visual and radio observations if their capture time is below a predefined threshold. Usually, visual observations are performed at a higher rate than radio observations. In a number of embodiments of the invention, radio observations indicating the receivable radio transmitters are performed periodically by the adaptation 242, and an observation vector is created, at each time step, from the radio observation and the visual observation which has been captured by the one or more image sensors 210 at the closest time. The determination of the visual observation that has been captured at the closest time can be performed for example by comparing timestamps of visual and radio observations.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Nowakowski into the invention of Xu in view of Brunner and Nowakowski to include measuring distance between visual and radio measurement within a threshold time to measure distance between measurements as Nowakowski discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can measure distance traveled between datapoints. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu in view of Brunner and Nowakowski and comparing visual and radio observations as taught by Nowakowski. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 5, Xu in view of Brunner and Nowakowski teaches:
wherein determining whether the location of the robotic device is within a threshold distance of a previously-visited one of the plurality of locations comprises comparing a determined location of the robotic device with every previously-determined location of the robotic device while maneuvering the robotic device through the plurality of locations (Xu: Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 7B; [0028], Also during such movement in such a learning mode, a series of pathways may be derived that, together, define the map. Some aspects of the map, including locations of pathway intersections where loops in the map are closed, may be derived through use of key frames taken from the two separate sets of key frames as those sets are still being generated. As the pathways are derived, each captured frame that is selected to become a key frame in one or both of the sets of key frames may be correlated to a location along one of the pathways of the map. Other aspects of the map, including directions and/or lengths of pathways, may be derived through use of the one or more location sensors based on any of a variety of sensing technologies. Where possible, in some embodiments, locations along one or more of the pathways of the map may be associated with an identifier of a physical location within the defined area. Overall, in generating the map, it may be deemed desirable to make use of both key frames and the location sensors in generating the map. This may be due to limitations in the degree of positioning accuracy that may be achievable through use of the location sensors, alone).
Regarding claim 6, Xu in view of Brunner and Nowakowski teaches:
wherein determining the location of the robotic device at each of the plurality of locations using a round trip time (RTT) measurement of a signal sent to an access point comprises performing a Wi-Fi RTT measurement to determine the location of the robotic device (Brunner: Abstract, A Visual Inertial Tracker (VIT), such as a Simultaneous Localization And Mapping (SLAM) system based on an Extended Kalman Filter (EKF) framework (EKF-SLAM) can provide drift correction in calculations of a pose (translation and orientation) of a mobile device by obtaining location information regarding a target, obtaining an image of the target, estimating, from the image of the target, measurements relating to a pose of the mobile device based on the image and location information, and correcting a pose determination of the mobile device using an EKF, based, at least in part, on the measurements relating to the pose of the mobile device; [0041], The process can start by receiving a camera image at block 310. The type of camera image can vary in resolution, color, and/or other characteristics, depending on desired functionality, camera hardware, and/or other factors. Moreover, the camera image may be a discrete still image or may be one of several frames of video captured by the camera. In some embodiments, the image may be processed to a degree before it is received by a VIT, to facilitate further image processing by the VIT; [0042], At block 320, the VIT optionally receives WiFi signals, which can facilitate the determination of which targets may be included in the received image. For example, wireless signals may be utilized together with a map of a venue that includes the identity and locations of WiFi access points. If the locations of certain access points can be determined from the map, the VIT can get a rough estimate of where in the venue the VIT (and any mobile device associated therewith) is. The VIT can do this by measuring WiFi signals received from the WiFi access points by the mobile device (e.g., measuring received signal strength (RSSI), round-trip time (RTT), and/or other measurements) to determine a proximity of the access points--including which access points may be closest. This can then be compared with the map to determine a region in the venue in which the mobile device is located.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Brunner into the invention of Xu in view of Brunner and Nowakowski to include collecting WiFi signal data as well as images captured by a camera to determine a pose and position of a mobile device as Brunner discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can corroborate RTT measurements from known access points with image data to perform visual loop closure detection. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu and using RTT measurements to localize a device as taught by Brunner. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 7, Xu discloses:
A robotic device, comprising (Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 7B; [0028], Also during such movement in such a learning mode, a series of pathways may be derived that, together, define the map. Some aspects of the map, including locations of pathway intersections where loops in the map are closed, may be derived through use of key frames taken from the two separate sets ofkey frames as those sets are still being generated. As the pathways are derived, each captured frame that is selected to become a key frame in one or both of the sets of key frames may be correlated to a location along one of the pathways of the map. Other aspects of the map, including directions and/or lengths of pathways, may be derived through use of the one or more location sensors based on any of a variety of sensing technologies. Where possible, in some embodiments, locations along one or more of the pathways of the map may be associated with an identifier of a physical location within the defined area. Overall, in generating the map, it may be deemed desirable to make use of both key frames and the location sensors in generating the map. This may be due to limitations in the degree of positioning accuracy that may be achievable through use of the location sensors, alone; [0026], The mobile device may be self propelled (e.g., motorized with wheels to move itself about) or may be carried about by another device and/or by a person. The mobile device may be equipped with one or more cameras to recurringly capture images of the surroundings of the mobile device as the mobile device is moved about within the defined area. Various approaches may be used to incorporate the one or more cameras into the mobile device to enable capturing images in more than one direction from the current position of the mobile device. In some embodiments, the mobile device may also be equipped with any of a variety of types of location sensors, such as one or more radio frequency (RF) receivers that may receive signals from multiple positioning satellites orbiting the Earth and/or multiple stationary position devices (e.g., wireless access points), one or more accelerometers and/or one or more gyroscopes to provide inertial guidance, and/or readers of barcodes and/or RFID tags that may be positioned at various locations within the defined area. In some embodiments, the mobile device may be equipped with a wireless network interface to enable communication between the mobile):
a processor configured with processor-executable instructions to (Fig. 1A; Fig. 1B; Fig. 3A; [0037], In various embodiments, the mobile device 100 may incorporate a processor circuit 150, a storage 160, one or more cameras 110, and/or a network interface 190 to couple the mobile device 100 to the network 999. In embodiments in which the locating system 1000 includes the one or more transmitting devices 319, the mobile device 100 may additionally incorporate one or more location sensors 119 to receive signals transmitted by the one or more transmitting devices 319. In embodiments in which the mobile device 100 is self-propelled such that the mobile device 100 is capable of moving about within the defined area 300 under its own power, the mobile device 100 may incorporate one or more motors 170 to effect such movement. The storage 160 may store a control routine 140, captured image data 131 and/or captured location data 137. The control routine 140 may each incorporate a sequence of instructions operative on the processor circuit 150 to implement logic to perform various functions.):
maneuver the robotic device through a plurality of locations ([0026], The mobile device may be self propelled (e.g., motorized with wheels to move itself about) or may be carried about by another device and/or by a person. The mobile device may be equipped with one or more cameras to recurringly capture images of the surroundings of the mobile device as the mobile device is moved about within the defined area. Various approaches may be used to incorporate the one or more cameras into the mobile device to enable capturing images in more than one direction from the current position of the mobile device. In some embodiments, the mobile device may also be equipped with any of a variety of types of location sensors, such as one or more radio frequency (RF) receivers that may receive signals from multiple positioning satellites orbiting the Earth and/or multiple stationary position devices (e.g., wireless access points), one or more accelerometers and/or one or more gyroscopes to provide inertial guidance, and/or readers of barcodes and/or RFID tags that may be positioned at various locations within the defined area. In some embodiments, the mobile device may be equipped with a wireless network interface to enable communication between the mobile);
and perform visual loop closure detection in response to determining that the location of the robotic device associated with the first keyframe is within the threshold distance of the previously-visited one of the plurality of locations that is associated with the second keyframe ([0026], The mobile device may be self propelled (e.g., motorized with wheels to move itself about) or may be carried about by another device and/or by a person. The mobile device may be equipped with one or more cameras to recurringly capture images of the surroundings of the mobile device as the mobile device is moved about within the defined area. Various approaches may be used to incorporate the one or more cameras into the mobile device to enable capturing images in more than one direction from the current position of the mobile device. In some embodiments, the mobile device may also be equipped with any of a variety of types of location sensors, such as one or more radio frequency (RF) receivers that may receive signals from multiple positioning satellites orbiting the Earth and/or multiple stationary position devices (e.g., wireless access points), one or more accelerometers and/or one or more gyroscopes to provide inertial guidance, and/or readers of barcodes and/or RFID tags that may be positioned at various locations within the defined area. In some embodiments, the mobile device may be equipped with a wireless network interface to enable communication between the mobile).
However, Xu does not specifically state:
determine, using a round trip time (RTT) measurement of at least one signal sent to an access point, a location of the robotic device at each of the plurality of locations coincident with each of a plurality of keyframes from images captured using a camera of the robotic device;
determine whether the location of the robotic device associated with a first keyframe is within a threshold distance of a previously-visited one of the plurality of locations that is associated with a second keyframe;
Brunner teaches:
determine, using a round trip time (RTT) measurement of at least one signal sent to an access point, a location of the robotic device at each of the plurality of locations coincident with each of a plurality of keyframes from images captured using a camera of the robotic device (Abstract, A Visual Inertial Tracker (VIT), such as a Simultaneous Localization And Mapping (SLAM) system based on an Extended Kalman Filter (EKF) framework (EKF-SLAM) can provide drift correction in calculations of a pose (translation and orientation) of a mobile device by obtaining location information regarding a target, obtaining an image of the target, estimating, from the image of the target, measurements relating to a pose of the mobile device based on the image and location information, and correcting a pose determination of the mobile device using an EKF, based, at least in part, on the measurements relating to the pose of the mobile device; [0041], The process can start by receiving a camera image at block 310. The type of camera image can vary in resolution, color, and/or other characteristics, depending on desired functionality, camera hardware, and/or other factors. Moreover, the camera image may be a discrete still image or may be one of several frames of video captured by the camera. In some embodiments, the image may be processed to a degree before it is received by a VIT, to facilitate further image processing by the VIT; [0042], At block 320, the VIT optionally receives WiFi signals, which can facilitate the determination of which targets may be included in the received image. For example, wireless signals may be utilized together with a map of a venue that includes the identity and locations of WiFi access points. If the locations of certain access points can be determined from the map, the VIT can get a rough estimate of where in the venue the VIT (and any mobile device associated therewith) is. The VIT can do this by measuring WiFi signals received from the WiFi access points by the mobile device (e.g., measuring received signal strength (RSSI), round-trip time (RTT), and/or other measurements) to determine a proximity of the access points--including which access points may be closest. This can then be compared with the map to determine a region in the venue in which the mobile device is located.);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Brunner into the invention of Xu to include collect WiFi signal data as well as images captured by a camera to determine a pose and position of a mobile device as Brunner discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can corroborate RTT measurements from known access points with image data to perform visual loop closure detection. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu and using RTT measurements to localize a device as taught by Brunner. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
However, Xu in view of Brunner does not specifically state:
determine whether the location of the robotic device associated with a first keyframe is within a threshold distance of a previously-visited one of the plurality of locations that is associated with a second keyframe;
Nowakowski teaches:
determine whether the location of the robotic device associated with a first keyframe is within a threshold distance of a previously-visited one of the plurality of locations that is associated with a second keyframe ([0081], The distance between visual and radio observations can be calculated for example by odometry, or by comparing timestamps of observations, and adding to a radio observation vector a visual observation vector which is the closest in time; [0087], In order for the visual and radio observations to be consistent, the visual and radio observations need to be performed within the same time frame. In a number of embodiments of the invention, an observation vector is formed from visual and radio observations if their capture time is below a predefined threshold. Usually, visual observations are performed at a higher rate than radio observations. In a number of embodiments of the invention, radio observations indicating the receivable radio transmitters are performed periodically by the adaptation 242, and an observation vector is created, at each time step, from the radio observation and the visual observation which has been captured by the one or more image sensors 210 at the closest time. The determination of the visual observation that has been captured at the closest time can be performed for example by comparing timestamps of visual and radio observations.);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Nowakowski into the invention of Xu in view of Brunner to include measuring distance between visual and radio measurement within a threshold time to measure distance between measurements as Nowakowski discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can measure distance traveled between datapoints. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu in view of Brunner and comparing visual and radio observations as taught by Nowakowski. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 8, Xu in view of Brunner and Nowakowski teaches:
select the plurality of keyframes from among images captured using the camera of the robotic device (Xu: Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 3A; Fig. 3B);
Regarding claim 9, Xu in view of Brunner and Nowakowski teaches:
wherein the processor is further configured with processor-executable instructions to associate with each respective keyframe the determined location of the robotic device coincident with each of the plurality of keyframes (Xu: Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 3A; Fig. 3B).
Regarding claim 10, Xu in view of Brunner and Nowakowski teaches:
wherein the processor is further configured with processor-executable instructions to determine whether the location of the robotic device that is associated with a keyframe is within a threshold distance of a previously-visited one of the plurality of locations associated with another keyframe (Nowakowski: [0081], The distance between visual and radio observations can be calculated for example by odometry, or by comparing timestamps of observations, and adding to a radio observation vector a visual observation vector which is the closest in time; [0087], In order for the visual and radio observations to be consistent, the visual and radio observations need to be performed within the same time frame. In a number of embodiments of the invention, an observation vector is formed from visual and radio observations if their capture time is below a predefined threshold. Usually, visual observations are performed at a higher rate than radio observations. In a number of embodiments of the invention, radio observations indicating the receivable radio transmitters are performed periodically by the adaptation 242, and an observation vector is created, at each time step, from the radio observation and the visual observation which has been captured by the one or more image sensors 210 at the closest time. The determination of the visual observation that has been captured at the closest time can be performed for example by comparing timestamps of visual and radio observations.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Nowakowski into the invention of Xu in view of Brunner and Nowakowski to include measuring distance between visual and radio measurement within a threshold time to measure distance between measurements as Nowakowski discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can measure distance traveled between datapoints. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu in view of Brunner and Nowakowski and comparing visual and radio observations as taught by Nowakowski. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 11, Xu in view of Brunner and Nowakowski teaches:
wherein the processor is further configured with processor-executable instructions to compare the determined location of the robotic device with every previously-determined location of the robotic device while maneuvering the robotic device through the plurality of locations (Xu: Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 7B; [0028], Also during such movement in such a learning mode, a series of pathways may be derived that, together, define the map. Some aspects of the map, including locations of pathway intersections where loops in the map are closed, may be derived through use of key frames taken from the two separate sets of key frames as those sets are still being generated. As the pathways are derived, each captured frame that is selected to become a key frame in one or both of the sets of key frames may be correlated to a location along one of the pathways of the map. Other aspects of the map, including directions and/or lengths of pathways, may be derived through use of the one or more location sensors based on any of a variety of sensing technologies. Where possible, in some embodiments, locations along one or more of the pathways of the map may be associated with an identifier of a physical location within the defined area. Overall, in generating the map, it may be deemed desirable to make use of both key frames and the location sensors in generating the map. This may be due to limitations in the degree of positioning accuracy that may be achievable through use of the location sensors, alone).
Regarding claim 12, Xu in view of Brunner and Nowakowski teaches:
wherein the processor is further configured with processor-executable instructions to perform a Wi-Fi RTT measurement to determine the location of the robotic device (Brunner: Abstract, A Visual Inertial Tracker (VIT), such as a Simultaneous Localization And Mapping (SLAM) system based on an Extended Kalman Filter (EKF) framework (EKF-SLAM) can provide drift correction in calculations of a pose (translation and orientation) of a mobile device by obtaining location information regarding a target, obtaining an image of the target, estimating, from the image of the target, measurements relating to a pose of the mobile device based on the image and location information, and correcting a pose determination of the mobile device using an EKF, based, at least in part, on the measurements relating to the pose of the mobile device; [0041], The process can start by receiving a camera image at block 310. The type of camera image can vary in resolution, color, and/or other characteristics, depending on desired functionality, camera hardware, and/or other factors. Moreover, the camera image may be a discrete still image or may be one of several frames of video captured by the camera. In some embodiments, the image may be processed to a degree before it is received by a VIT, to facilitate further image processing by the VIT; [0042], At block 320, the VIT optionally receives WiFi signals, which can facilitate the determination of which targets may be included in the received image. For example, wireless signals may be utilized together with a map of a venue that includes the identity and locations of WiFi access points. If the locations of certain access points can be determined from the map, the VIT can get a rough estimate of where in the venue the VIT (and any mobile device associated therewith) is. The VIT can do this by measuring WiFi signals received from the WiFi access points by the mobile device (e.g., measuring received signal strength (RSSI), round-trip time (RTT), and/or other measurements) to determine a proximity of the access points--including which access points may be closest. This can then be compared with the map to determine a region in the venue in which the mobile device is located.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Brunner into the invention of Xu in view of Brunner and Nowakowski to include collecting WiFi signal data as well as images captured by a camera to determine a pose and position of a mobile device as Brunner discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can corroborate RTT measurements from known access points with image data to perform visual loop closure detection. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu and using RTT measurements to localize a device as taught by Brunner. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 13, Xu discloses:
A robotic device, comprising (Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 7B; [0028], Also during such movement in such a learning mode, a series of pathways may be derived that, together, define the map. Some aspects of the map, including locations of pathway intersections where loops in the map are closed, may be derived through use of key frames taken from the two separate sets ofkey frames as those sets are still being generated. As the pathways are derived, each captured frame that is selected to become a key frame in one or both of the sets of key frames may be correlated to a location along one of the pathways of the map. Other aspects of the map, including directions and/or lengths of pathways, may be derived through use of the one or more location sensors based on any of a variety of sensing technologies. Where possible, in some embodiments, locations along one or more of the pathways of the map may be associated with an identifier of a physical location within the defined area. Overall, in generating the map, it may be deemed desirable to make use of both key frames and the location sensors in generating the map. This may be due to limitations in the degree of positioning accuracy that may be achievable through use of the location sensors, alone; [0026], The mobile device may be self propelled (e.g., motorized with wheels to move itself about) or may be carried about by another device and/or by a person. The mobile device may be equipped with one or more cameras to recurringly capture images of the surroundings of the mobile device as the mobile device is moved about within the defined area. Various approaches may be used to incorporate the one or more cameras into the mobile device to enable capturing images in more than one direction from the current position of the mobile device. In some embodiments, the mobile device may also be equipped with any of a variety of types of location sensors, such as one or more radio frequency (RF) receivers that may receive signals from multiple positioning satellites orbiting the Earth and/or multiple stationary position devices (e.g., wireless access points), one or more accelerometers and/or one or more gyroscopes to provide inertial guidance, and/or readers of barcodes and/or RFID tags that may be positioned at various locations within the defined area. In some embodiments, the mobile device may be equipped with a wireless network interface to enable communication between the mobile):
means for maneuvering the robotic device through a plurality of locations ([0026], The mobile device may be self propelled (e.g., motorized with wheels to move itself about) or may be carried about by another device and/or by a person. The mobile device may be equipped with one or more cameras to recurringly capture images of the surroundings of the mobile device as the mobile device is moved about within the defined area. Various approaches may be used to incorporate the one or more cameras into the mobile device to enable capturing images in more than one direction from the current position of the mobile device. In some embodiments, the mobile device may also be equipped with any of a variety of types of location sensors, such as one or more radio frequency (RF) receivers that may receive signals from multiple positioning satellites orbiting the Earth and/or multiple stationary position devices (e.g., wireless access points), one or more accelerometers and/or one or more gyroscopes to provide inertial guidance, and/or readers of barcodes and/or RFID tags that may be positioned at various locations within the defined area. In some embodiments, the mobile device may be equipped with a wireless network interface to enable communication between the mobile);
and means for performing visual loop closure detection in response to determining that the location of the robotic device associated with the first keyframe is within the threshold distance of the previously-visited one of the plurality of locations that is associated with the second keyframe ([0026], The mobile device may be self propelled (e.g., motorized with wheels to move itself about) or may be carried about by another device and/or by a person. The mobile device may be equipped with one or more cameras to recurringly capture images of the surroundings of the mobile device as the mobile device is moved about within the defined area. Various approaches may be used to incorporate the one or more cameras into the mobile device to enable capturing images in more than one direction from the current position of the mobile device. In some embodiments, the mobile device may also be equipped with any of a variety of types of location sensors, such as one or more radio frequency (RF) receivers that may receive signals from multiple positioning satellites orbiting the Earth and/or multiple stationary position devices (e.g., wireless access points), one or more accelerometers and/or one or more gyroscopes to provide inertial guidance, and/or readers of barcodes and/or RFID tags that may be positioned at various locations within the defined area. In some embodiments, the mobile device may be equipped with a wireless network interface to enable communication between the mobile).
However, Xu does not specifically state:
means for determining, using a round trip time (RTT) measurement of at least one signal sent to an access point, a location of the robotic device at each of the plurality of locations coincident with each of a plurality of keyframes from images captured using a camera of the robotic device;
means for determining whether the location of the robotic device associated with a first keyframe is within a threshold distance of a previously-visited one of the plurality of locations that is associated with a second keyframe;
Brunner teaches:
means for determining, using a round trip time (RTT) measurement of at least one signal sent to an access point, a location of the robotic device at each of the plurality of locations coincident with each of a plurality of keyframes from images captured using a camera of the robotic device (Abstract, A Visual Inertial Tracker (VIT), such as a Simultaneous Localization And Mapping (SLAM) system based on an Extended Kalman Filter (EKF) framework (EKF-SLAM) can provide drift correction in calculations of a pose (translation and orientation) of a mobile device by obtaining location information regarding a target, obtaining an image of the target, estimating, from the image of the target, measurements relating to a pose of the mobile device based on the image and location information, and correcting a pose determination of the mobile device using an EKF, based, at least in part, on the measurements relating to the pose of the mobile device; [0041], The process can start by receiving a camera image at block 310. The type of camera image can vary in resolution, color, and/or other characteristics, depending on desired functionality, camera hardware, and/or other factors. Moreover, the camera image may be a discrete still image or may be one of several frames of video captured by the camera. In some embodiments, the image may be processed to a degree before it is received by a VIT, to facilitate further image processing by the VIT; [0042], At block 320, the VIT optionally receives WiFi signals, which can facilitate the determination of which targets may be included in the received image. For example, wireless signals may be utilized together with a map of a venue that includes the identity and locations of WiFi access points. If the locations of certain access points can be determined from the map, the VIT can get a rough estimate of where in the venue the VIT (and any mobile device associated therewith) is. The VIT can do this by measuring WiFi signals received from the WiFi access points by the mobile device (e.g., measuring received signal strength (RSSI), round-trip time (RTT), and/or other measurements) to determine a proximity of the access points--including which access points may be closest. This can then be compared with the map to determine a region in the venue in which the mobile device is located.);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Brunner into the invention of Xu to include collect WiFi signal data as well as images captured by a camera to determine a pose and position of a mobile device as Brunner discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can corroborate RTT measurements from known access points with image data to perform visual loop closure detection. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu and using RTT measurements to localize a device as taught by Brunner. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
However, Xu in view of Brunner does not specifically state:
means for determining whether the location of the robotic device associated with a first keyframe is within a threshold distance of a previously-visited one of the plurality of locations that is associated with a second keyframe;
Nowakowski teaches:
means for determining whether the location of the robotic device associated with a first keyframe is within a threshold distance of a previously-visited one of the plurality of locations that is associated with a second keyframe ([0081], The distance between visual and radio observations can be calculated for example by odometry, or by comparing timestamps of observations, and adding to a radio observation vector a visual observation vector which is the closest in time; [0087], In order for the visual and radio observations to be consistent, the visual and radio observations need to be performed within the same time frame. In a number of embodiments of the invention, an observation vector is formed from visual and radio observations if their capture time is below a predefined threshold. Usually, visual observations are performed at a higher rate than radio observations. In a number of embodiments of the invention, radio observations indicating the receivable radio transmitters are performed periodically by the adaptation 242, and an observation vector is created, at each time step, from the radio observation and the visual observation which has been captured by the one or more image sensors 210 at the closest time. The determination of the visual observation that has been captured at the closest time can be performed for example by comparing timestamps of visual and radio observations.);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Nowakowski into the invention of Xu in view of Brunner to include measuring distance between visual and radio measurement within a threshold time to measure distance between measurements as Nowakowski discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can measure distance traveled between datapoints. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu in view of Brunner and comparing visual and radio observations as taught by Nowakowski. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 14, Xu in view of Brunner and Nowakowski teaches:
means for selecting a plurality of keyframes from among images captured using the camera of the robotic device (Xu: Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 3A; Fig. 3B);
Regarding claim 15, Xu in view of Brunner and Nowakowski teaches:
further comprising means for associating with each respective keyframe the determined location of the robotic device coincident with each of the plurality of keyframes (Xu: Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 3A; Fig. 3B).
Regarding claim 16, Xu in view of Brunner and Nowakowski teaches:
wherein means for determining whether the location of the robotic device is within the threshold distance of a previously-visited one of the plurality of locations comprises means for determining whether the location of the robotic device that is associated with a keyframe is within the threshold distance of a previously-visited one of the plurality of locations associated with another keyframe (Nowakowski: [0081], The distance between visual and radio observations can be calculated for example by odometry, or by comparing timestamps of observations, and adding to a radio observation vector a visual observation vector which is the closest in time; [0087], In order for the visual and radio observations to be consistent, the visual and radio observations need to be performed within the same time frame. In a number of embodiments of the invention, an observation vector is formed from visual and radio observations if their capture time is below a predefined threshold. Usually, visual observations are performed at a higher rate than radio observations. In a number of embodiments of the invention, radio observations indicating the receivable radio transmitters are performed periodically by the adaptation 242, and an observation vector is created, at each time step, from the radio observation and the visual observation which has been captured by the one or more image sensors 210 at the closest time. The determination of the visual observation that has been captured at the closest time can be performed for example by comparing timestamps of visual and radio observations.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Nowakowski into the invention of Xu in view of Brunner and Nowakowski to include measuring distance between visual and radio measurement within a threshold time to measure distance between measurements as Nowakowski discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can measure distance traveled between datapoints. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu in view of Brunner and Nowakowski and comparing visual and radio observations as taught by Nowakowski. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 17, Xu in view of Brunner and Nowakowski teaches:
wherein means for determining whether a location of the robotic device is within the threshold distance of a previously-visited one of the plurality of locations comprises means for comparing the determined location of the robotic device with every previously-determined location of the robotic device while maneuvering the robotic device through the plurality of locations (Xu: Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 7B; [0028], Also during such movement in such a learning mode, a series of pathways may be derived that, together, define the map. Some aspects of the map, including locations of pathway intersections where loops in the map are closed, may be derived through use of key frames taken from the two separate sets of key frames as those sets are still being generated. As the pathways are derived, each captured frame that is selected to become a key frame in one or both of the sets of key frames may be correlated to a location along one of the pathways of the map. Other aspects of the map, including directions and/or lengths of pathways, may be derived through use of the one or more location sensors based on any of a variety of sensing technologies. Where possible, in some embodiments, locations along one or more of the pathways of the map may be associated with an identifier of a physical location within the defined area. Overall, in generating the map, it may be deemed desirable to make use of both key frames and the location sensors in generating the map. This may be due to limitations in the degree of positioning accuracy that may be achievable through use of the location sensors, alone).
Regarding claim 18, Xu in view of Brunner and Nowakowski teaches:
wherein means for determining the location of the robotic device at each of the plurality of locations using a round trip time (RTT) measurement of a signal sent to an access point comprises means for performing a Wi-Fi RTT measurement to determine the location of the robotic device (Brunner: Abstract, A Visual Inertial Tracker (VIT), such as a Simultaneous Localization And Mapping (SLAM) system based on an Extended Kalman Filter (EKF) framework (EKF-SLAM) can provide drift correction in calculations of a pose (translation and orientation) of a mobile device by obtaining location information regarding a target, obtaining an image of the target, estimating, from the image of the target, measurements relating to a pose of the mobile device based on the image and location information, and correcting a pose determination of the mobile device using an EKF, based, at least in part, on the measurements relating to the pose of the mobile device; [0041], The process can start by receiving a camera image at block 310. The type of camera image can vary in resolution, color, and/or other characteristics, depending on desired functionality, camera hardware, and/or other factors. Moreover, the camera image may be a discrete still image or may be one of several frames of video captured by the camera. In some embodiments, the image may be processed to a degree before it is received by a VIT, to facilitate further image processing by the VIT; [0042], At block 320, the VIT optionally receives WiFi signals, which can facilitate the determination of which targets may be included in the received image. For example, wireless signals may be utilized together with a map of a venue that includes the identity and locations of WiFi access points. If the locations of certain access points can be determined from the map, the VIT can get a rough estimate of where in the venue the VIT (and any mobile device associated therewith) is. The VIT can do this by measuring WiFi signals received from the WiFi access points by the mobile device (e.g., measuring received signal strength (RSSI), round-trip time (RTT), and/or other measurements) to determine a proximity of the access points--including which access points may be closest. This can then be compared with the map to determine a region in the venue in which the mobile device is located.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Brunner into the invention of Xu in view of Brunner and Nowakowski to include collecting WiFi signal data as well as images captured by a camera to determine a pose and position of a mobile device as Brunner discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can corroborate RTT measurements from known access points with image data to perform visual loop closure detection. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu and using RTT measurements to localize a device as taught by Brunner. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 19, Xu discloses:
A non-transitory processor-readable medium having stored thereon processor-executable instructions configured to cause a processor of a robotic device to perform operations comprising (Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 7B; [0028], Also during such movement in such a learning mode, a series of pathways may be derived that, together, define the map. Some aspects of the map, including locations of pathway intersections where loops in the map are closed, may be derived through use of key frames taken from the two separate sets ofkey frames as those sets are still being generated. As the pathways are derived, each captured frame that is selected to become a key frame in one or both of the sets of key frames may be correlated to a location along one of the pathways of the map. Other aspects of the map, including directions and/or lengths of pathways, may be derived through use of the one or more location sensors based on any of a variety of sensing technologies. Where possible, in some embodiments, locations along one or more of the pathways of the map may be associated with an identifier of a physical location within the defined area. Overall, in generating the map, it may be deemed desirable to make use of both key frames and the location sensors in generating the map. This may be due to limitations in the degree of positioning accuracy that may be achievable through use of the location sensors, alone; [0026], The mobile device may be self propelled (e.g., motorized with wheels to move itself about) or may be carried about by another device and/or by a person. The mobile device may be equipped with one or more cameras to recurringly capture images of the surroundings of the mobile device as the mobile device is moved about within the defined area. Various approaches may be used to incorporate the one or more cameras into the mobile device to enable capturing images in more than one direction from the current position of the mobile device. In some embodiments, the mobile device may also be equipped with any of a variety of types of location sensors, such as one or more radio frequency (RF) receivers that may receive signals from multiple positioning satellites orbiting the Earth and/or multiple stationary position devices (e.g., wireless access points), one or more accelerometers and/or one or more gyroscopes to provide inertial guidance, and/or readers of barcodes and/or RFID tags that may be positioned at various locations within the defined area. In some embodiments, the mobile device may be equipped with a wireless network interface to enable communication between the mobile; [0099], As previously discussed, the storage 960 (which may correspond to the storage 460) may be made up of one or more distinct storage devices based on any of a wide variety of technologies or combinations of technologies. More specifically, as depicted, the storage 960 may include one or more of a volatile storage 961 ( e.g., solid state storage based on one or more forms of RAM technology), a nonvolatile storage 962 (e.g., solid state, ferromagnetic or other storage not requiring a constant provision of electric power to preserve their contents), and a removable media storage 963 ( e.g., removable disc or solid state memory card storage by which information may be conveyed between devices). This depiction of the storage 960 as possibly including multiple distinct types of storage is in recognition of the commonplace use of more than one type of storage device in devices in which one type provides relatively rapid reading and writing capabilities enabling more rapid manipulation of data by the processor circuit 950 (but possibly using a "volatile" technology constantly requiring electric power) while another type provides relatively high density of non-volatile storage (but likely provides relatively slow reading and writing capabilities)):
maneuvering the robotic device through a plurality of locations ([0026], The mobile device may be self propelled (e.g., motorized with wheels to move itself about) or may be carried about by another device and/or by a person. The mobile device may be equipped with one or more cameras to recurringly capture images of the surroundings of the mobile device as the mobile device is moved about within the defined area. Various approaches may be used to incorporate the one or more cameras into the mobile device to enable capturing images in more than one direction from the current position of the mobile device. In some embodiments, the mobile device may also be equipped with any of a variety of types of location sensors, such as one or more radio frequency (RF) receivers that may receive signals from multiple positioning satellites orbiting the Earth and/or multiple stationary position devices (e.g., wireless access points), one or more accelerometers and/or one or more gyroscopes to provide inertial guidance, and/or readers of barcodes and/or RFID tags that may be positioned at various locations within the defined area. In some embodiments, the mobile device may be equipped with a wireless network interface to enable communication between the mobile);
and performing visual loop closure detection in response to determining that the location of the robotic device associated with the first keyframe is within the threshold distance of the previously-visited one of the plurality of locations that is associated with the second keyframe ([0026], The mobile device may be self propelled (e.g., motorized with wheels to move itself about) or may be carried about by another device and/or by a person. The mobile device may be equipped with one or more cameras to recurringly capture images of the surroundings of the mobile device as the mobile device is moved about within the defined area. Various approaches may be used to incorporate the one or more cameras into the mobile device to enable capturing images in more than one direction from the current position of the mobile device. In some embodiments, the mobile device may also be equipped with any of a variety of types of location sensors, such as one or more radio frequency (RF) receivers that may receive signals from multiple positioning satellites orbiting the Earth and/or multiple stationary position devices (e.g., wireless access points), one or more accelerometers and/or one or more gyroscopes to provide inertial guidance, and/or readers of barcodes and/or RFID tags that may be positioned at various locations within the defined area. In some embodiments, the mobile device may be equipped with a wireless network interface to enable communication between the mobile).
However, Xu does not specifically state:
determining, using a round trip time (RTT) measurement of at least one signal sent to an access point, a location of the robotic device at each of the plurality of locations using a round trip time (RTT) measurement of a signal sent to an access point coincident with each of a plurality of keyframes from images captured using a camera of the robotic device;
determining whether the location of the robotic device associated with a first keyframe is within a threshold distance of a previously-visited one of the plurality of locations that is associated with a second keyframe;
Brunner teaches:
determining, using a round trip time (RTT) measurement of at least one signal sent to an access point, a location of the robotic device at each of the plurality of locations using a round trip time (RTT) measurement of a signal sent to an access point coincident with each of a plurality of keyframes from images captured using a camera of the robotic device (Abstract, A Visual Inertial Tracker (VIT), such as a Simultaneous Localization And Mapping (SLAM) system based on an Extended Kalman Filter (EKF) framework (EKF-SLAM) can provide drift correction in calculations of a pose (translation and orientation) of a mobile device by obtaining location information regarding a target, obtaining an image of the target, estimating, from the image of the target, measurements relating to a pose of the mobile device based on the image and location information, and correcting a pose determination of the mobile device using an EKF, based, at least in part, on the measurements relating to the pose of the mobile device; [0041], The process can start by receiving a camera image at block 310. The type of camera image can vary in resolution, color, and/or other characteristics, depending on desired functionality, camera hardware, and/or other factors. Moreover, the camera image may be a discrete still image or may be one of several frames of video captured by the camera. In some embodiments, the image may be processed to a degree before it is received by a VIT, to facilitate further image processing by the VIT; [0042], At block 320, the VIT optionally receives WiFi signals, which can facilitate the determination of which targets may be included in the received image. For example, wireless signals may be utilized together with a map of a venue that includes the identity and locations of WiFi access points. If the locations of certain access points can be determined from the map, the VIT can get a rough estimate of where in the venue the VIT (and any mobile device associated therewith) is. The VIT can do this by measuring WiFi signals received from the WiFi access points by the mobile device (e.g., measuring received signal strength (RSSI), round-trip time (RTT), and/or other measurements) to determine a proximity of the access points--including which access points may be closest. This can then be compared with the map to determine a region in the venue in which the mobile device is located.);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Brunner into the invention of Xu to include collect WiFi signal data as well as images captured by a camera to determine a pose and position of a mobile device as Brunner discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can corroborate RTT measurements from known access points with image data to perform visual loop closure detection. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu and using RTT measurements to localize a device as taught by Brunner. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
However, Xu in view of Brunner does not specifically state:
determining whether the location of the robotic device associated with a first keyframe is within a threshold distance of a previously-visited one of the plurality of locations that is associated with a second keyframe;
Nowakowski teaches:
determining whether the location of the robotic device associated with a first keyframe is within a threshold distance of a previously-visited one of the plurality of locations that is associated with a second keyframe ([0081], The distance between visual and radio observations can be calculated for example by odometry, or by comparing timestamps of observations, and adding to a radio observation vector a visual observation vector which is the closest in time; [0087], In order for the visual and radio observations to be consistent, the visual and radio observations need to be performed within the same time frame. In a number of embodiments of the invention, an observation vector is formed from visual and radio observations if their capture time is below a predefined threshold. Usually, visual observations are performed at a higher rate than radio observations. In a number of embodiments of the invention, radio observations indicating the receivable radio transmitters are performed periodically by the adaptation 242, and an observation vector is created, at each time step, from the radio observation and the visual observation which has been captured by the one or more image sensors 210 at the closest time. The determination of the visual observation that has been captured at the closest time can be performed for example by comparing timestamps of visual and radio observations.);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Nowakowski into the invention of Xu in view of Brunner to include measuring distance between visual and radio measurement within a threshold time to measure distance between measurements as Nowakowski discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can measure distance traveled between datapoints. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu in view of Brunner and comparing visual and radio observations as taught by Nowakowski. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 20, Xu in view of Brunner and Nowakowski teaches:
select the plurality of keyframes from among images captured using the camera of the robotic device (Xu: Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 3A; Fig. 3B);
Regarding claim 21, Xu in view of Brunner and Nowakowski teaches:
wherein the stored processor-executable instructions are further configured to cause the processor of the robotic device to perform operations further comprising associating with each respective keyframe the determined location of the robotic device coincident with each of the plurality of keyframes (Xu: Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 3A; Fig. 3B).
Regarding claim 22, Xu in view of Brunner and Nowakowski teaches:
wherein the stored processor-executable instructions are further configured to cause the processor of the robotic device to perform operations such that determining whether the location of the robotic device is within the threshold distance of a previously-visited one of the plurality of locations comprises determining whether the location of the robotic device that is associated with a keyframe is within the threshold distance of a previously-visited one of the plurality of locations associated with another keyframe (Nowakowski: [0081], The distance between visual and radio observations can be calculated for example by odometry, or by comparing timestamps of observations, and adding to a radio observation vector a visual observation vector which is the closest in time; [0087], In order for the visual and radio observations to be consistent, the visual and radio observations need to be performed within the same time frame. In a number of embodiments of the invention, an observation vector is formed from visual and radio observations if their capture time is below a predefined threshold. Usually, visual observations are performed at a higher rate than radio observations. In a number of embodiments of the invention, radio observations indicating the receivable radio transmitters are performed periodically by the adaptation 242, and an observation vector is created, at each time step, from the radio observation and the visual observation which has been captured by the one or more image sensors 210 at the closest time. The determination of the visual observation that has been captured at the closest time can be performed for example by comparing timestamps of visual and radio observations.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Nowakowski into the invention of Xu in view of Brunner and Nowakowski to include measuring distance between visual and radio measurement within a threshold time to measure distance between measurements as Nowakowski discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can measure distance traveled between datapoints. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu in view of Brunner and Nowakowski and comparing visual and radio observations as taught by Nowakowski. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 23, Xu in view of Brunner and Nowakowski teaches:
wherein the stored processor-executable instructions are further configured to cause the processor of the robotic device to perform operations such that determining whether a location of the robotic device is within a threshold distance of a previously-visited one of the plurality of locations comprises comparing the determined location of the robotic device with every previously-determined location of the robotic device while maneuvering the robotic device through the plurality of locations (Xu: Abstract, Various embodiments are directed to techniques for determining a current location of a mobile device. An apparatus includes a SLAM candidate component to identify a first candidate key frame matching a current captured frame by a first degree from an interval-based key frame set with key frames selected on a recurring interval from multiple earlier captured frames captured by mobile device camera of surroundings within a defined area, a CNN candidate component to identify a second candidate key frame matching the current captured frame by a second degree from a difference- based key frame set with key frames selected from the multiple earlier captured frames based on a degree of difference from all key frames already therein, and a position estimation component to determine a current location of the mobile device from estimates of differences between the current location and locations of the first and second candidate key frames; Fig. 7B; [0028], Also during such movement in such a learning mode, a series of pathways may be derived that, together, define the map. Some aspects of the map, including locations of pathway intersections where loops in the map are closed, may be derived through use of key frames taken from the two separate sets of key frames as those sets are still being generated. As the pathways are derived, each captured frame that is selected to become a key frame in one or both of the sets of key frames may be correlated to a location along one of the pathways of the map. Other aspects of the map, including directions and/or lengths of pathways, may be derived through use of the one or more location sensors based on any of a variety of sensing technologies. Where possible, in some embodiments, locations along one or more of the pathways of the map may be associated with an identifier of a physical location within the defined area. Overall, in generating the map, it may be deemed desirable to make use of both key frames and the location sensors in generating the map. This may be due to limitations in the degree of positioning accuracy that may be achievable through use of the location sensors, alone).
Regarding claim 24, Xu in view of Brunner and Nowakowski teaches:
wherein the stored processor-executable instructions are further configured to cause the processor of the robotic device to perform operations such that determining the location of the robotic device at each of the plurality of locations using a round trip time (RTT) measurement of a signal sent to an access point comprises performing a Wi-Fi RTT measurement to determine the location of the robotic device (Brunner: Abstract, A Visual Inertial Tracker (VIT), such as a Simultaneous Localization And Mapping (SLAM) system based on an Extended Kalman Filter (EKF) framework (EKF-SLAM) can provide drift correction in calculations of a pose (translation and orientation) of a mobile device by obtaining location information regarding a target, obtaining an image of the target, estimating, from the image of the target, measurements relating to a pose of the mobile device based on the image and location information, and correcting a pose determination of the mobile device using an EKF, based, at least in part, on the measurements relating to the pose of the mobile device; [0041], The process can start by receiving a camera image at block 310. The type of camera image can vary in resolution, color, and/or other characteristics, depending on desired functionality, camera hardware, and/or other factors. Moreover, the camera image may be a discrete still image or may be one of several frames of video captured by the camera. In some embodiments, the image may be processed to a degree before it is received by a VIT, to facilitate further image processing by the VIT; [0042], At block 320, the VIT optionally receives WiFi signals, which can facilitate the determination of which targets may be included in the received image. For example, wireless signals may be utilized together with a map of a venue that includes the identity and locations of WiFi access points. If the locations of certain access points can be determined from the map, the VIT can get a rough estimate of where in the venue the VIT (and any mobile device associated therewith) is. The VIT can do this by measuring WiFi signals received from the WiFi access points by the mobile device (e.g., measuring received signal strength (RSSI), round-trip time (RTT), and/or other measurements) to determine a proximity of the access points--including which access points may be closest. This can then be compared with the map to determine a region in the venue in which the mobile device is located.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Brunner into the invention of Xu in view of Brunner and Nowakowski to include collecting WiFi signal data as well as images captured by a camera to determine a pose and position of a mobile device as Brunner discloses with a reasonable expectation of success. One would be motivated to incorporate aspects of the cited prior art to create a more robust system that can corroborate RTT measurements from known access points with image data to perform visual loop closure detection. Additionally, the claimed invention is merely a combination of old, well-known elements of performing visual loops closure detection as disclosed by Xu and using RTT measurements to localize a device as taught by Brunner. The combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Documents Considered but Not Relied Upon
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Yang et al. (US 9288633 B2) discloses a method of detecting a revisit position includes receiving at a computing system a plurality of position data points, each of the plurality of position data points including a signal scan measurement. The method farther includes calculating a first signal distance between a first signal scan measurement corresponding to a first position data point of the plurality of position data points and a second signal scan measurement corresponding to a second position data point of the plurality of position data points. The method further includes determining that the first signal distance is less than a first threshold, that the first signal distance is a local minimum for the first position data point, and the first signal distance is a local minimum for the second position data point. The method further includes, based on the determining, identifying the first and second position data points as revisit points.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IZCALLI ANDRE RIOS-AGUIRRE whose telephone number is (571)272-0790. The examiner can normally be reached Monday through Friday 8:30 - 17:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Scott A. Browne can be reached at (571) 270-0151. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/I.A.R./Examiner, Art Unit 3666
/SCOTT A BROWNE/Supervisory Patent Examiner, Art Unit 3666
Prosecution Insights