DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a Final Office Action on the merits. Claims 1-23 are currently pending and are addressed below.
Response to Amendment
The specification was objected to due to minor informalities. Applicant amended the specification accordingly; therefore, the specification objection is withdrawn.
Claims 11-12 were objected to due to minor informalities. Applicant amended the claims accordingly; therefore, the objection is withdrawn.
Claims 2-9 and 14-21 were rejected under 35 U.S.C. 112 as being indefinite. Applicant amended the claims accordingly; therefore, the rejection is withdrawn.
Response to Arguments
Applicant’s arguments on pages 15-16 of the response, with respect to the rejection(s) of claim(s) 1-22 under 35 U.S.C. 101 have been fully considered but they are not persuasive. Applicant argues that claim 1 is directed to a technological improvement in the functioning of a computer-implemented vision system of a UAV and is meaningfully integrated into a particular machine and a specific technological environment. Examiner respectfully disagrees. The limitations “…mapping correspondences between the reference aerial image and the current aerial image [using a homography estimating tool executing onboard the UAV]…” and “…validating…a detection of the first object [by the object detection model] based on the mapping of the correspondences” are abstract ideas that encompass mentally mapping correspondences between two images and mentally validating a detection of an object, respectively. The limitation “…informing a detection of the first object by the object detection model based on the mapping of the correspondences” is recited at a high level of generality such that it amounts to mere post-solution activity or displaying, which is a form of insignificant extra-solution activity. The additional elements “object detection model” and “homography estimating tool executing onboard the UAV” are also recited at a high level of generality such that they merely describe how to “apply” the otherwise mental judgments in a technological environment using generic computer components.
Furthermore, any software or hardware that aids in the process of mapping correspondences, such as the camera of the UAV, can be considered a “homography estimating tool”, and labeling the tool as such does not require the entire process to be executed onboard the UAV. Lastly, the functions of storing an asset map, acquiring aerial images, mapping correspondences, analyzing aerial images, and validating or informing a detection of an object based on the mapping are recited at a high level of generality such that they can be performed by any generic computer.
Applicant’s arguments on pages 16-19 of the response, with respect to the rejection(s) of claim(s) 1, 8, 13, and 20 under 35 U.S.C. 102 and claim(s) 2-7, 9-12, 14-19, and 21-22 under 35 U.S.C. 103 have been fully considered but they are not persuasive. Applicant argues that Price fails to disclose validating or informing a detection of a first object by an object detection model based on the mapping of correspondences using a homography estimating tool, since the “detection/identification is an upstream input to the homography-based correlation, not something informed by it”. Examiner respectfully disagrees. [0030] of Price recites “At 208, method 200 includes, based on the above-described automatic correlation, determining a geospatial coordinate for the visual target in the search region. For example, the RANSAC homography computation may be configured to match the aerial photograph to the satellite photograph and to use a scale and displacement in the matching of the photographs to determine a coordinate of a pixel location in the aerial photography”, which clearly shows correlating the aerial photograph to the satellite photograph using the RANSAC homography computation to then determine a pixel location. Price additionally recites in [0029] using RANSAC to align landmarks based on image features, such that “a visual target in an aerial image may be correlated against a satellite image even when partially occluded in one or more of the aerial image and the satellite image”. This shows that occluded targets are detected based on the correlation, and, thus, after performing the correlation.
Applicant further argues there is “no suggestion that homography results are fed back into an object detection model to “validate” or “inform” detection of the visual target”, that “Price does not describe that detection reliability of a visual target”, and “Price does not disclose that homography is used to validate, confirm, rejected, or adjust confidence in any manner that the visual target has been correctly detected or identified”. However, these arguments are not directed to claim 1 as written since the claim does not recite feeding homography results into the object detection model, a detection reliability of a visual target, or a confidence that the visual target has been correctly detected.
Applicant further argues that “Price makes no reference to a homography estimating tool executing onboard a UAV”. Examiner respectfully disagrees. Any software or hardware that aids in the process of mapping correspondences, such as the camera of the UAV, can be considered a “homography estimating tool”, and labeling the tool as such does not require the entire process to be executed onboard the UAV.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Independent Claim 1:
Step 1: Claim 1 is directed to a method performed by a UAV (i.e., a process). Therefore, claim 1 is within at least one of the four statutory categories.
Step 2A Prong 1: Regarding Prong 1 of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity and/or c) mental processes.
Independent claim 1 includes limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection. Claim 1 recites:
A method performed by an unmanned aerial vehicle (UAV), the method comprising:
storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image;
acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area;
mapping correspondences between the reference aerial image and the current aerial image using a homography estimating tool executing onboard the UAV;
analyzing the current aerial image with an object detection model to detect a first object positioned at the ground area; and
validating or informing a detection of the first object by the object detection model based on the mapping of the correspondences.
The examiner submits that the foregoing bolded limitations constitute a “mental process” because under its broadest reasonable interpretation, the claim covers performance of the limitations in the human mind. For example, the limitation “mapping correspondences between the reference aerial image and the current aerial image” in the context of this claim encompasses mentally mapping correspondences between two images. The limitation “analyzing the current aerial image…to detect a first object positioned at the ground area” encompasses mentally analyzing an image to identify an object in the image. Lastly, the limitation “validating…a detection of the first object…based on the mapping of the correspondences” encompasses mentally validating the detection of an object based on the mapping.
Step 2A Prong 2: Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A method performed by an unmanned aerial vehicle (UAV), the method comprising:
storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image;
acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area;
mapping correspondences between the reference aerial image and the current aerial image using a homography estimating tool executing onboard the UAV;
analyzing the current aerial image with an object detection model to detect a first object positioned at the ground area; and
validating or informing a detection of the first object by the object detection model based on the mapping of the correspondences.
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
The additional limitation(s) “storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image” and “acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area” are recited at a high level of generality and amount to mere data storage and data gathering, which is a form of insignificant extra-solution activity. The additional limitation “…informing a detection of the first object….” is also recited at a high level of generality and amounts to mere post-solution activity or displaying, which is another form of insignificant extra-solution activity. Lastly, the additional limitations “an unmanned aerial vehicle (UAV)”, “using a homography estimating tool executing onboard the UAV”, and “object detection model” merely describe how to generally “apply” the otherwise mental judgements in a technological environment using generic computer components.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B: Regarding Step 2B of the 2019 PEG, representative independent claim 1 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements “unmanned aerial vehicle (UAV)”, “using a homography estimating tool executing onboard the UAV”, and “object detection model” are recited at a high-level of generality and amount to nothing more than applying the exception to a technological environment using generic computer components. The examiner also submits that the additional limitations “storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image”, “acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area”, and “…informing a detection of the first object…” are insignificant extra-solution activity.
Further, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B to determine if they are more than what is well-understood, routine, conventional activity in the field. The additional limitations “storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image” and “acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area” is well-understood, routine, and conventional activity in light of MPEP 2106.05(d)(II) and the cases cited therein, including Intellectual Ventures I, LLC v. Symantec Corp., 838 F.3d 1307, 1321 (Fed. Cir. 2016), TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610 (Fed. Cir. 2016), and OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363 (Fed. Cir. 2015), which indicate that mere collection or receipt of data over a network is a well‐understood, routine, and conventional function when it is claimed in a merely generic manner. The additional limitation “…informing a detection of the first object…” is well-understood, routine, and conventional activity in light of MPEP 2106.05(g) and the cases cited therein, including OIP Techs., 788 F.3d at 1362-63, 115 USPQ2d at 1092-93, which indicate that “presenting offers” is a well-understood, routine, and conventional function when it is claimed at a high level of generality.
Therefore, claim 1 is ineligible under 35 U.S.C §101.
Regarding Independent Claim 1:
Step 1: Claim 13 is directed to a method performed by a UAV (i.e., a process). Therefore, claim 1 is within at least one of the four statutory categories.
Step 2A Prong 1: Regarding Prong 1 of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity and/or c) mental processes.
Independent claim 13 includes limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection. Claim 13 recites:
A method performed by an unmanned aerial vehicle (UAV), the method comprising:
storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image;
acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area;
mapping correspondences between the reference aerial image and the current aerial image using a homography estimating tool executing onboard the UAV;
analyzing the current aerial image with an object detection model to detect a first object positioned at the ground area; and
validating or informing a detection of the first object by the object detection model based on the mapping of the correspondences.
At least one non-transitory machine-accessible storage medium that provides instructions that, when executed by an unmanned aerial vehicle (UAV), will cause the UAV to perform operations comprising:
storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image;
acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area;
mapping correspondences between the reference aerial image and the current aerial image using a homography estimating tool executing onboard the UAV;
analyzing the current aerial image with an object detection model to detect a first object positioned at the ground area; and
validating or informing a detection of the first object by the object detection model based on the mapping of the correspondences.
The examiner submits that the foregoing bolded limitations constitute a “mental process” because under its broadest reasonable interpretation, the claim covers performance of the limitations in the human mind. For example, the limitation “mapping correspondences between the reference aerial image and the current aerial image” in the context of this claim encompasses mentally mapping correspondences between two images. The limitation “analyzing the current aerial image…to detect a first object positioned at the ground area” encompasses mentally analyzing an image to identify an object in the image. Lastly, the limitation “validating…a detection of the first object…based on the mapping of the correspondences” encompasses mentally validating the detection of an object based on the mapping.
Step 2A Prong 2: Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
At least one non-transitory machine-accessible storage medium that provides instructions that, when executed by an unmanned aerial vehicle (UAV), will cause the UAV to perform operations comprising:
storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image;
acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area;
mapping correspondences between the reference aerial image and the current aerial image using a homography estimating tool executing onboard the UAV;
analyzing the current aerial image with an object detection model to detect a first object positioned at the ground area; and
validating or informing a detection of the first object by the object detection model based on the mapping of the correspondences.
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
The additional limitation(s) “storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image” and “acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area” are recited at a high level of generality and amount to mere data storage and data gathering, which is a form of insignificant extra-solution activity. The additional limitation “…informing a detection of the first object….” is also recited at a high level of generality and amounts to mere post-solution activity or displaying, which is another form of insignificant extra-solution activity. Lastly, the additional limitations “an unmanned aerial vehicle (UAV)”, “using a homography estimating tool executing onboard the UAV”, and “object detection model” merely describe how to generally “apply” the otherwise mental judgements in a technological environment using generic computer components.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B: Regarding Step 2B of the 2019 PEG, representative independent claim 13 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements “unmanned aerial vehicle (UAV)”, “using a homography estimating tool executing onboard the UAV”, and “object detection model” are recited at a high-level of generality and amount to nothing more than applying the exception to a technological environment using generic computer components. The examiner also submits that the additional limitations “storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image”, “acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area”, and “…informing a detection of the first object…” are insignificant extra-solution activity.
Further, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B to determine if they are more than what is well-understood, routine, conventional activity in the field. The additional limitations “storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image” and “acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area” is well-understood, routine, and conventional activity in light of MPEP 2106.05(d)(II) and the cases cited therein, including Intellectual Ventures I, LLC v. Symantec Corp., 838 F.3d 1307, 1321 (Fed. Cir. 2016), TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610 (Fed. Cir. 2016), and OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363 (Fed. Cir. 2015), which indicate that mere collection or receipt of data over a network is a well‐understood, routine, and conventional function when it is claimed in a merely generic manner. The additional limitation “…informing a detection of the first object…” is well-understood, routine, and conventional activity in light of MPEP 2106.05(g) and the cases cited therein, including OIP Techs., 788 F.3d at 1362-63, 115 USPQ2d at 1092-93, which indicate that “presenting offers” is a well-understood, routine, and conventional function when it is claimed at a high level of generality.
Therefore, claim 13 is ineligible under 35 U.S.C §101.
Dependent Claims
Dependent claim(s) 2-11 and 14-23 do not recite any further limitations that cause the claims to be patent eligible. Rather, the limitations of the dependent claim(s) are directed toward additional aspects of the judicial exception. Dependent claim(s) 2 and 14 are further directed to the abstract idea of annotating the current aerial image, dependent claim(s) 3 and 15 are further directed to the abstract idea of affirming or upweighting the detection of an object, dependent claim(s) 4 and 16 are further directed to the abstract idea of masking or downweighting the detection of an object, dependent claims 5-7 and 17-19 are further directed to the abstract idea of navigating (i.e., mentally planning a route for the UAV), and dependent claims 8 and 20 are further directed to the abstract ideas of making navigation decisions and decreasing an influence of the mapping on the navigation decisions. Dependent claims 9 and 21 are further directed to the abstract ideas of identifying a gravity aligned pixel, matching the gravity aligned pixel in one image to a corresponding geolocated pixel in another image, and localizing the UAV. Dependent claim 21 is further directed to the abstract idea of annotating the asset map with geolocation labels. Dependent claims 10 and 22 further describe the reference objects and the object detection model, dependent claims 11-12 further recite “storing a library of asset maps”, which is insignificant extra-solution activity (i.e., data gathering), and the abstract idea of selecting the asset map from the library of asset maps, and dependent claim 12 is further directed to the abstract idea of generating a current vector embedding based on an image. Dependent claim 23 is further directed to the abstract ideas of generating annotation data for an aerial image and validating the detection of an object by accepting, rejecting, or adjusting a confidence of the detection based on a spatial correspondence between detection and annotation data. Therefore, dependent claim(s) 2-12 and 14-23 is/are not patent eligible under the same rationale as provided for in the rejection of claims 1 and 13.
Therefore, claim(s) 1-23 is/are ineligible under 35 U.S.C. §101.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 8, 13, and 20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Price of US 20210383144 A1, published 12/09/2021, hereinafter “Price”.
Regarding claim 1, Price discloses:
A method performed by an unmanned aerial vehicle (UAV), the method comprising: (See at least Abstract: “A method of automatically geolocating a visual target. The method comprises operating a flying vehicle in a search region including the visual target. The method further includes affirmatively identifying a visual target in an aerial photograph of the search region captured by the flying vehicle…”)
storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image; (See at least [0017]: “…The geo-tagged photography may be labeled in any suitable fashion. For example, a satellite photo may be labeled with one or more coordinates associated with specific pixels (e.g., a coordinate associated with a corner or center of the image). Alternately or additionally, specific pixel locations in an image may have an associated label (e.g., indicating a geospatial coordinate, address, or landmark)…” & [0027]: “…The satellite photograph and/or labels may be stored on the assistive device (e.g., whether the assistive device is a head-mounted device, mobile device, or any other device), stored on the flying vehicle, and/or stored on an external server…”)
acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area; (See at least [0021]: “…assistive device 102 may communicate with a flying vehicle 104 including a camera, wherein flying vehicle 104 is configured to fly above a visual target to obtain an aerial picture of the visual target, and communicate the aerial picture of the visual target back to assistive device 102…”)
mapping correspondences between the reference aerial image and the current aerial image using a homography estimating tool executing onboard the UAV; (See at least [0027]: “At 206, method 200 includes automatically correlating the aerial photograph of the search region to a geo-tagged photograph of the search region. As a non-limiting example, the geo-tagged photograph may be a satellite photograph…” & [0029]: “In some examples, automatically correlating the aerial photograph of the search region to the satellite photograph of the search region includes automatically computer-assessing a homography between the aerial photograph and the satellite photograph. In some examples, the homography may be established via random sample consensus (RANSAC) or via any other suitable statistical and/or machine learning algorithm. For example, RANSAC may be able to align geographic landmarks based on image features…”)
analyzing the current aerial image with an object detection model to detect a first object positioned at the ground area; and (See at least [0026]: “At 204, method 200 includes affirmatively identifying a visual target in an aerial photograph of the search region captured by the flying vehicle. In some examples, identifying the visual target in the aerial photograph may be based on computer-identification using visual features in the photograph…For example, a machine learning system may be trained for object detection and/or object location, thereby permitting the machine learning system to affirmatively identify a visual target in a photograph, including determining a location of the visual target in the photograph…”)
validating or informing a detection of the first object by the object detection model based on the mapping of the correspondences. (See at least [0016]: “…For example, human 110b may be a human user of an assistive device 102 using the assistive device 102 to request geospatial coordinates of their current location…Using the methods of the present disclosure, the aerial photograph of the environment 110 may be correlated to the satellite photograph 112 to determine a location 112b corresponding to the human 110b. Based on the determined location 112b in the satellite photography, a geospatial coordinate of the user 110b may be determined and returned to the assistive device…” & [0030]: “At 208, method 200 includes, based on the above-described automatic correlation, determining a geospatial coordinate for the visual target in the search region. For example, the RANSAC homography computation may be configured to match the aerial photograph to the satellite photograph and to use a scale and displacement in the matching of the photographs to determine a coordinate of a pixel location in the aerial photography.”)
Regarding claim 8, Price discloses all the limitations of claim 1 as discussed above.
Price additionally discloses:
further comprising: making navigation decisions for the UAV based upon one or both of the detection of the first object by the object detection model or the mapping of the correspondences from the homography estimating tool; and decreasing an influence of the mapping of the correspondences from the homography estimating tool on the navigation decisions, relative to the detection of the first object by the object detection model, as an above ground level (AGL) altitude of the UAV decreases. (See at least [0023]: “In general, the flying vehicle may be configured to deploy to an aerial region substantially above the ground target in order to capture a suitable aerial photograph. For example, flying vehicle 104 may be a human-portable, autonomous flying vehicle configured to ascend to a suitable altitude to take a plan-view photograph of a nearby environment of the human user, then to descend and return to the human user. The aerial region substantially above the target may be at any location suitable for obtaining a photograph that is substantially in plan view so that a visual target in the environment may be identified without obstruction…”, [0026]: “…In general, the affirmative identification of the visual target in the aerial photograph may include determining a location of the visual target in the aerial photograph (e.g., as an X-Y pixel coordinate in the aerial photograph). For example, a machine learning system may be trained for object detection and/or object location, thereby permitting the machine learning system to affirmatively identify a visual target in a photograph, including determining a location of the visual target in the photograph…” & [0049]: “…the flying vehicle is an autonomous flying vehicle configured to automatically deploy to an aerial region substantially above the visual target for capturing the aerial photograph…”)
NOTE: Claim 8 recites the following contingent limitation: “…decreasing an influence of the mapping of the correspondences from the homography estimating tool on the navigation decisions, relative to the detection of the first object by the object detection model, as an above ground level (AGL) altitude of the UAV decreases”. This limitation is contingent because it recites steps that are only required to be performed if their conditions are met. For example, the contingent limitation only needs to be performed if the “mapping from the homography tool” is used for making navigation decisions for the UAV, since the claim only requires “one or both of” the “detection of the first object by the detection model” or the “mapping from the homography tool”. Therefore, the BRI of claim 8 only requires the contingent limitation if the “mapping” is used to make navigation decisions for the UAV.
Regarding claim 13, Price discloses:
At least one non-transitory machine-accessible storage medium that provides instructions that, when executed by an unmanned aerial vehicle (UAV), will cause the UAV to perform operations comprising: (See at least Abstract: “A method of automatically geolocating a visual target. The method comprises operating a flying vehicle in a search region including the visual target. The method further includes affirmatively identifying a visual target in an aerial photograph of the search region captured by the flying vehicle…”. See also [0034-0037].)
storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image; (See at least [0017]: “…The geo-tagged photography may be labeled in any suitable fashion. For example, a satellite photo may be labeled with one or more coordinates associated with specific pixels (e.g., a coordinate associated with a corner or center of the image). Alternately or additionally, specific pixel locations in an image may have an associated label (e.g., indicating a geospatial coordinate, address, or landmark)…” & [0027]: “…The satellite photograph and/or labels may be stored on the assistive device (e.g., whether the assistive device is a head-mounted device, mobile device, or any other device), stored on the flying vehicle, and/or stored on an external server…”)
acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area; (See at least [0021]: “…assistive device 102 may communicate with a flying vehicle 104 including a camera, wherein flying vehicle 104 is configured to fly above a visual target to obtain an aerial picture of the visual target, and communicate the aerial picture of the visual target back to assistive device 102…”)
mapping correspondences between the reference aerial image and the current aerial image using a homography estimating tool executing onboard the UAV; (See at least [0027]: “At 206, method 200 includes automatically correlating the aerial photograph of the search region to a geo-tagged photograph of the search region. As a non-limiting example, the geo-tagged photograph may be a satellite photograph…” & [0029]: “In some examples, automatically correlating the aerial photograph of the search region to the satellite photograph of the search region includes automatically computer-assessing a homography between the aerial photograph and the satellite photograph. In some examples, the homography may be established via random sample consensus (RANSAC) or via any other suitable statistical and/or machine learning algorithm. For example, RANSAC may be able to align geographic landmarks based on image features…”)
analyzing the current aerial image with an object detection model to detect a first object positioned at the ground area; and (See at least [0026]: “At 204, method 200 includes affirmatively identifying a visual target in an aerial photograph of the search region captured by the flying vehicle. In some examples, identifying the visual target in the aerial photograph may be based on computer-identification using visual features in the photograph…For example, a machine learning system may be trained for object detection and/or object location, thereby permitting the machine learning system to affirmatively identify a visual target in a photograph, including determining a location of the visual target in the photograph…”)
validating or informing a detection of the first object by the object detection model based on the mapping of the correspondences. (See at least [0016]: “…For example, human 110b may be a human user of an assistive device 102 using the assistive device 102 to request geospatial coordinates of their current location…Using the methods of the present disclosure, the aerial photograph of the environment 110 may be correlated to the satellite photograph 112 to determine a location 112b corresponding to the human 110b. Based on the determined location 112b in the satellite photography, a geospatial coordinate of the user 110b may be determined and returned to the assistive device…” & [0030]: “At 208, method 200 includes, based on the above-described automatic correlation, determining a geospatial coordinate for the visual target in the search region. For example, the RANSAC homography computation may be configured to match the aerial photograph to the satellite photograph and to use a scale and displacement in the matching of the photographs to determine a coordinate of a pixel location in the aerial photography.”)
Regarding claim 20, Price discloses all the limitations of claim 13 as discussed above.
Price additionally discloses:
wherein the operations further comprise: making navigation decisions for the UAV based upon one or both of the detection of the first object by the object detection model or the mapping of the correspondences from the homography estimating tool; and decreasing an influence of the mapping of the correspondences from the homography estimating tool on the navigation decisions, relative to the detection of the first object by the object detection model, as an above ground level (AGL) altitude of the UAV decreases. (See at least [0023]: “In general, the flying vehicle may be configured to deploy to an aerial region substantially above the ground target in order to capture a suitable aerial photograph. For example, flying vehicle 104 may be a human-portable, autonomous flying vehicle configured to ascend to a suitable altitude to take a plan-view photograph of a nearby environment of the human user, then to descend and return to the human user. The aerial region substantially above the target may be at any location suitable for obtaining a photograph that is substantially in plan view so that a visual target in the environment may be identified without obstruction…”, [0026]: “…In general, the affirmative identification of the visual target in the aerial photograph may include determining a location of the visual target in the aerial photograph (e.g., as an X-Y pixel coordinate in the aerial photograph). For example, a machine learning system may be trained for object detection and/or object location, thereby permitting the machine learning system to affirmatively identify a visual target in a photograph, including determining a location of the visual target in the photograph…” & [0049]: “…the flying vehicle is an autonomous flying vehicle configured to automatically deploy to an aerial region substantially above the visual target for capturing the aerial photograph…”)
NOTE: Claim 20 recites the following contingent limitation: “…decreasing an influence of the mapping on the navigation decisions, relative to the detection of the first object by the object detection model, as an above ground level (AGL) altitude of the UAV decreases”. This limitation is contingent because it recites steps that are only required to be performed if their conditions are met. For example, the contingent limitation only needs to be performed if the “mapping from the homography tool” is used for making navigation decisions for the UAV, since the claim only requires “one or both of” the “detection of the first object by the detection model” or the “mapping from the homography tool”. Therefore, the BRI of claim 20 only requires the contingent limitation if the “mapping” is used to make navigation decisions for the UAV.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2-6, 10, 14-18, and 22-23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Price in view of Kerzner of US 20210110137 A1, filed 10/15/2020, hereinafter “Kerzner”.
Regarding claim 2, Price discloses all the limitations of claim 1 as discussed above.
Price does not explicitly teach:
further comprising: annotating the current aerial image with one or more annotations corresponding to one or more of the reference objects from the asset map based on the mapping of the correspondences between the reference aerial image and the current aerial image,
wherein validating or informing the detection of the first object by the object detection model is based on the annotating.
Kerzner teaches:
further comprising: annotating the current aerial image with one or more annotations corresponding to one or more of the reference objects from the asset map based on the mapping of the correspondences between the reference aerial image and the current aerial image, (See at least [0070]: “In analyzing the captured images, the monitoring server 130 may identify one or more visual landmarks from the captured images and/or the generated 3D environment map. In identifying one or more visual landmarks, the monitoring server 130 may employ segmentation to label areas within the captured images where physical objects are and/or what the physical objects are. Specifically, the monitoring server 130 may employ two dimensional (2D) scene segmentation, which maps each pixel to a different object or surface category, e.g., wall, floor, furniture, etc.…” & [0079]: “…In analyzing the captured images, the monitoring server 130 may identify one or more features of one or more of the identified visual landmarks. Typically, each identified feature will belong to a single visual landmark. Each of the features may be identified with a 3D representation. For example, once a visual landmark has been classified, the monitoring server 130 may refer to a lookup table for the classification or to a model for the classification to identify one or more features that are typical for the particular classification. For example, in regards to the table 126, the monitoring server 130 may refer to a lookup table for furniture tables. The lookup table may reveal that furniture tables typically have four legs and four corners…”)
wherein validating or informing the detection of the first object by the object detection model is based on the annotating. (See at least [0076]: “For example, if the scene segmentation labels a patch of pixels as “furniture”, that patch might be sent to a furniture classifier to gain more detail. The furniture classifier may then identify the patch of pixels as, for example, a couch, a table, a chair, etc. The furniture classifier may have been trained using one or more mathematical models of couches, tables, chairs, etc.…” & [0078]: “Alternatively, patches of pixels identified during segmentation may be sent by the monitoring server 130 to a binary classifier, trained only on good landmarks versus bad landmarks. As an example, the binary classifier may be trained to recognize that objects appearing similar to tables and couches should be classified as good landmarks, e.g., since tables and couches rarely change position. In this example, the binary classifier may be trained to recognize that objects appearing similar to chairs should be classified as bad landmarks, e.g., since chairs frequently change position.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Price’s method with Kerzner’s technique of annotating the current aerial image and validating or informing the detection of the first object based on the annotating. Doing so would be obvious so that “a robotic device that refers to the map, such as a drone, can more accurately and efficiently navigate through uncontrolled environments, such as residences, commercial properties, etc.” (See [0025] of Kerzner).
Regarding claim 3, Price and Kerzner in combination teach all the limitations of claim 2 as discussed above.
Kerzner additionally teaches:
wherein validating or informing the detection of the first object by the object detection model comprises: affirming or upweighting the detection of the first object when the detection of the first object overlaps with one of the one or more annotations annotating the current aerial image and sourced from the asset map. (See at least Figs. 2A-2B, [0059]: “The system 100 may collect specific images to be added to a subset of images 108 of images of previously identified visual landmarks, e.g., in order to later determine if the landmark is still present or to later determine if an appearance of the landmark has changed. For example, the drone 102 may refer to a previously generated environment map for the monitored property 120 and proceed to position itself and/or its camera 104 such that the FOV 106 encompasses a physical object that corresponds with a previously identified visual landmark…” & [0121]: “Once the drone 302 has positioned itself or its one or more onboard cameras towards where it expects the landmark 236 shown in FIGS. 2A-2B to be located, the drone 302 may capture one or more images including an image 308 (“Image X”). As shown, the image 308 includes the stairs 206. The drone 302 may compare the image 308 with the environment map 230. Specifically, the drone 302 may compare the image 308 with the landmark 236 in the environment map 230 and any features associated with the landmark 236. In comparing the image 308, the drone 302 may identify the landmark 236 within the image 308 and/or the features of the landmark 236 within the image 308. In comparing the image 308, the drone 302 may identify an expected location of the landmark 236 and/or one or more features of the landmark 236 with an actual location of the landmark 236 and/or one or more features of the landmark 236 based on the image 308...”)
Regarding claim 4, Price and Kerzner in combination teach all the limitations of claim 2 as discussed above.
Kerzner additionally teaches:
wherein validating or informing the detection of the first object by the object detection model comprises: masking or downweighting the detection of the first object when the detection of the first object does not overlap with one of the one or more annotations annotating the current aerial image and sourced from the asset map. (See at least [0071]: “In areas where the segmentation results differ, the monitoring server 130 may associate those areas with a lower confidence of the segmentation. The monitoring server 130 may actually calculate a confidence score for each area or may make a determination as to whether an area is acceptable, e.g., due to consistent segmentation results, or is unacceptable, e.g., due to differing segmentation results. Where an area is determined to have a confidence score below a first threshold level or is deemed unacceptable, the monitoring server 130 may label the area as having low confidence or as unacceptable. As an example, segmentation results may differ in areas of the monitored property 120 where appearance varies greatly by viewing angle, where a moving object is present, or where the segmentation algorithm is untrained for the surface(s) and/or physical object(s) present. In identifying one or more visual landmarks from the environment map, the monitoring server 130 may employ 3D scene segmentation. Each physical object identified through scene segmentation may be labelled as its own visual landmark. However, identified objects that were found within an area labelled as having low confidence or labelled as unacceptable may not be considered visual landmarks. This process may help to prevent moving objects such as people, animals, robots, etc. from being identified as visual landmarks.”)
Regarding claim 5, Price and Kerzner in combination teach all the limitations of claim 2 as discussed above.
Kerzner additionally teaches:
further comprising: navigating the UAV relative to the first object based upon the detection of the first object by the object detection model and (See at least [0289]: “The process 700 includes determining a path to navigate in the area based on the location (708). For example, as shown in FIG. 3, the drone 302 can generate a route through the floor 220 such that one or more landmarks, such as one or more planar landmarks, of the floor 220 will be observable and/or unobstructed along the entirety of the route or at certain points along the route. As shown in FIG. 3, the drone 302 generates a route from its current position 306b, to a second position 310 near the landmark 238, to a third position 312 near the landmark 234, and to a fourth position 314 near the landmark 232.”)
based upon the mapping of the correspondences between the reference aerial image and the current aerial image using the homography estimating tool. (See at least [0229]: “As another example, the monitoring server 130 can identify planar surfaces by detecting and matching features between the images, and identifying regions in the images which conform to a planar homography as predicted by the change in pose as given by the visual inertial odometry (VIO) which is generally accurate over relatively small distances. This process of identifying planar surfaces can be augmented by filtering for regions which exhibit a degree of continuity in motion flow, and/or using a previously generated 3D map to estimate the orientation of the camera to the candidate surface” & [0249]: “In some cases, the map includes indications of those landmarks that are planar landmarks. For example, the landmarks in the map can have corresponding metadata. This metadata can include an indication for each of the identified landmarks whether the landmark is planar or non-planar…”. See also [0246] regarding generating the environment map, which includes the visual landmarks.)
Regarding claim 6, Price and Kerzner in combination teach all the limitations of claim 5 as discussed above.
Kerzner additionally teaches:
wherein the UAV navigates with reference to the first object only after the detection of the first object by the object detection model registers to a corresponding one of the annotations sourced from the asset map. (See at least Fig. 7, [0189]: “The process 500 includes determining a path to navigate in the area based on the location (508). For example, as shown in FIG. 3, the drone 302 can generate a route through the floor 220 such that one or more landmarks of the floor 220 will be observable and/or unobstructed along the entirety of the route or at certain points along the route. Specifically, the drone 302 (or the monitoring server 130) may generate a path in order to keep the drone 302 within a threshold distance (e.g., 1.0 m, 1.5 m, 1.7 m, etc.) of at least a threshold number of landmarks at any given time (e.g., at least two landmarks, at least three landmarks, etc.)…” & [0289]: “The process 700 includes determining a path to navigate in the area based on the location (708). For example, as shown in FIG. 3, the drone 302 can generate a route through the floor 220 such that one or more landmarks, such as one or more planar landmarks, of the floor 220 will be observable and/or unobstructed along the entirety of the route or at certain points along the route. As shown in FIG. 3, the drone 302 generates a route from its current position 306b, to a second position 310 near the landmark 238, to a third position 312 near the landmark 234, and to a fourth position 314 near the landmark 232.”)
Regarding claim 10, Price discloses all the limitations of claim 1 as discussed above.
Price does not explicitly teach:
wherein the reference objects include at least one of a charging pad adapted for charging the UAV, a fiducial navigation marker adapted for visual navigation of the UAV, or an autoloader adapted to load a package onto the UAV, and wherein the object detection model comprises at least one of a machine learning (ML) charge pad detector, a ML fiducial marker detector, or a ML autoloader detector.
Kerzner teaches:
wherein the reference objects include at least one of a charging pad adapted for charging the UAV, a fiducial navigation marker adapted for visual navigation of the UAV, or an autoloader adapted to load a package onto the UAV, and wherein the object detection model comprises at least one of a machine learning (ML) charge pad detector, a ML fiducial marker detector, or a ML autoloader detector. (See at least [0049]: “As will be discussed in more detail below with respect to FIGS. 2A-2B, the environment map(s) generated and updated by the system 100 may include an indication of physical objects within the monitored property 120. These physical objects may be represented within the environment map(s) as landmarks. The landmarks may have or be associated with a location that indicates the location of the corresponding physical object within the monitored property 120…”, [0140]: “In some cases, identifying the landmarks in the area includes identifying an object or a surface of an object using one or more algorithms. The object or the surface of the object may be considered a landmark. For example, the drone 102 may use one or more machine learning algorithms to identify objects and/or particular surfaces of objects in the area of the property 120…” & [0146]: “…For example, if a surface of an object is classified as non-planar, the monitoring server 130 may determine that the surface and/or the object corresponding to the surface should not be used as a landmark (e.g., will not be used for navigation).”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it an obvious design choice combine Price’s method with Kerzner’s fiducial marker and ML fiducial marker detector. Doing so would be obvious so the UAV “can more accurately and efficiently navigate through uncontrolled environments, such as residences, commercial properties, etc.” (See [0025] of Kerzner).
Regarding claim 14, Price discloses all the limitations of claim 13 as discussed above.
Price does not explicitly teach:
wherein the operations further comprise: annotating the current aerial image with one or more annotations corresponding to one or more of the reference objects from the asset map based on the mapping of the correspondences between the reference aerial image and the current aerial image,
wherein validating or informing the detection of the first object by the object detection model is based on the annotating.
Kerzner teaches:
wherein the operations further comprise: annotating the current aerial image with one or more annotations corresponding to one or more of the reference objects from the asset map based on the mapping of the correspondences between the reference aerial image and the current aerial image, (See at least [0070]: “In analyzing the captured images, the monitoring server 130 may identify one or more visual landmarks from the captured images and/or the generated 3D environment map. In identifying one or more visual landmarks, the monitoring server 130 may employ segmentation to label areas within the captured images where physical objects are and/or what the physical objects are. Specifically, the monitoring server 130 may employ two dimensional (2D) scene segmentation, which maps each pixel to a different object or surface category, e.g., wall, floor, furniture, etc.…” & [0079]: “…In analyzing the captured images, the monitoring server 130 may identify one or more features of one or more of the identified visual landmarks. Typically, each identified feature will belong to a single visual landmark. Each of the features may be identified with a 3D representation. For example, once a visual landmark has been classified, the monitoring server 130 may refer to a lookup table for the classification or to a model for the classification to identify one or more features that are typical for the particular classification. For example, in regards to the table 126, the monitoring server 130 may refer to a lookup table for furniture tables. The lookup table may reveal that furniture tables typically have four legs and four corners…”)
wherein validating or informing the detection of the first object by the object detection model is based on the annotating. (See at least [0076]: “For example, if the scene segmentation labels a patch of pixels as “furniture”, that patch might be sent to a furniture classifier to gain more detail. The furniture classifier may then identify the patch of pixels as, for example, a couch, a table, a chair, etc. The furniture classifier may have been trained using one or more mathematical models of couches, tables, chairs, etc.…” & [0078]: “Alternatively, patches of pixels identified during segmentation may be sent by the monitoring server 130 to a binary classifier, trained only on good landmarks versus bad landmarks. As an example, the binary classifier may be trained to recognize that objects appearing similar to tables and couches should be classified as good landmarks, e.g., since tables and couches rarely change position. In this example, the binary classifier may be trained to recognize that objects appearing similar to chairs should be classified as bad landmarks, e.g., since chairs frequently change position.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Price’s method with Kerzner’s technique of annotating the current aerial image and validating or informing the detection of the first object based on the annotating. Doing so would be obvious so that “a robotic device that refers to the map, such as a drone, can more accurately and efficiently navigate through uncontrolled environments, such as residences, commercial properties, etc.” (See [0025] of Kerzner).
Regarding claim 15, Price and Kerzner in combination teach all the limitations of claim 14 as discussed above.
Kerzner additionally teaches:
wherein validating or informing the detection of the first object by the object detection model comprises: affirming or upweighting the detection of the first object when the detection of the first object overlaps with one of the one or more annotations annotating the current aerial image and sourced from the asset map. (See at least Figs. 2A-2B, [0059]: “The system 100 may collect specific images to be added to a subset of images 108 of images of previously identified visual landmarks, e.g., in order to later determine if the landmark is still present or to later determine if an appearance of the landmark has changed. For example, the drone 102 may refer to a previously generated environment map for the monitored property 120 and proceed to position itself and/or its camera 104 such that the FOV 106 encompasses a physical object that corresponds with a previously identified visual landmark…” & [0121]: “Once the drone 302 has positioned itself or its one or more onboard cameras towards where it expects the landmark 236 shown in FIGS. 2A-2B to be located, the drone 302 may capture one or more images including an image 308 (“Image X”). As shown, the image 308 includes the stairs 206. The drone 302 may compare the image 308 with the environment map 230. Specifically, the drone 302 may compare the image 308 with the landmark 236 in the environment map 230 and any features associated with the landmark 236. In comparing the image 308, the drone 302 may identify the landmark 236 within the image 308 and/or the features of the landmark 236 within the image 308. In comparing the image 308, the drone 302 may identify an expected location of the landmark 236 and/or one or more features of the landmark 236 with an actual location of the landmark 236 and/or one or more features of the landmark 236 based on the image 308...”)
Regarding claim 16, Price and Kerzner in combination teach all the limitations of claim 14 as discussed above.
Kerzner additionally teaches:
wherein validating or informing the detection of the first object by the object detection model comprises: masking or downweighting the detection of the first object when the detection of the first object does not overlap with one of the one or more annotations annotating the current aerial image and sourced from the asset map. (See at least [0071]: “In areas where the segmentation results differ, the monitoring server 130 may associate those areas with a lower confidence of the segmentation. The monitoring server 130 may actually calculate a confidence score for each area or may make a determination as to whether an area is acceptable, e.g., due to consistent segmentation results, or is unacceptable, e.g., due to differing segmentation results. Where an area is determined to have a confidence score below a first threshold level or is deemed unacceptable, the monitoring server 130 may label the area as having low confidence or as unacceptable. As an example, segmentation results may differ in areas of the monitored property 120 where appearance varies greatly by viewing angle, where a moving object is present, or where the segmentation algorithm is untrained for the surface(s) and/or physical object(s) present. In identifying one or more visual landmarks from the environment map, the monitoring server 130 may employ 3D scene segmentation. Each physical object identified through scene segmentation may be labelled as its own visual landmark. However, identified objects that were found within an area labelled as having low confidence or labelled as unacceptable may not be considered visual landmarks. This process may help to prevent moving objects such as people, animals, robots, etc. from being identified as visual landmarks.”)
Regarding claim 17, Price and Kerzner in combination teach all the limitations of claim 14 as discussed above.
Kerzner additionally teaches:
wherein the operations further comprise: navigating the UAV relative to the first object based upon the detection of the first object by the object detection model and (See at least [0289]: “The process 700 includes determining a path to navigate in the area based on the location (708). For example, as shown in FIG. 3, the drone 302 can generate a route through the floor 220 such that one or more landmarks, such as one or more planar landmarks, of the floor 220 will be observable and/or unobstructed along the entirety of the route or at certain points along the route. As shown in FIG. 3, the drone 302 generates a route from its current position 306b, to a second position 310 near the landmark 238, to a third position 312 near the landmark 234, and to a fourth position 314 near the landmark 232.”)
based upon the mapping of the correspondences between the reference aerial image and the current aerial image using the homography estimating tool. (See at least [0229]: “As another example, the monitoring server 130 can identify planar surfaces by detecting and matching features between the images, and identifying regions in the images which conform to a planar homography as predicted by the change in pose as given by the visual inertial odometry (VIO) which is generally accurate over relatively small distances. This process of identifying planar surfaces can be augmented by filtering for regions which exhibit a degree of continuity in motion flow, and/or using a previously generated 3D map to estimate the orientation of the camera to the candidate surface” & [0249]: “In some cases, the map includes indications of those landmarks that are planar landmarks. For example, the landmarks in the map can have corresponding metadata. This metadata can include an indication for each of the identified landmarks whether the landmark is planar or non-planar…”. See also [0246] regarding generating the environment map, which includes the visual landmarks.)
Regarding claim 18, Price and Kerzner in combination teach all the limitations of claim 17 as discussed above.
Kerzner additionally teaches:
wherein the UAV navigates with reference to the first object only after the detection of the first object by the object detection model registers to a corresponding one of the annotations sourced from the asset map. (See at least Fig. 7, [0189]: “The process 500 includes determining a path to navigate in the area based on the location (508). For example, as shown in FIG. 3, the drone 302 can generate a route through the floor 220 such that one or more landmarks of the floor 220 will be observable and/or unobstructed along the entirety of the route or at certain points along the route. Specifically, the drone 302 (or the monitoring server 130) may generate a path in order to keep the drone 302 within a threshold distance (e.g., 1.0 m, 1.5 m, 1.7 m, etc.) of at least a threshold number of landmarks at any given time (e.g., at least two landmarks, at least three landmarks, etc.)…” & [0289]: “The process 700 includes determining a path to navigate in the area based on the location (708). For example, as shown in FIG. 3, the drone 302 can generate a route through the floor 220 such that one or more landmarks, such as one or more planar landmarks, of the floor 220 will be observable and/or unobstructed along the entirety of the route or at certain points along the route. As shown in FIG. 3, the drone 302 generates a route from its current position 306b, to a second position 310 near the landmark 238, to a third position 312 near the landmark 234, and to a fourth position 314 near the landmark 232.”)
Regarding claim 22, Price discloses all the limitations of claim 13 as discussed above.
Price does not explicitly teach:
wherein: the reference objects include at least one of a charging pad adapted for charging the UAV, a fiducial navigation marker adapted for visual navigation of the UAV, or an autoloader adapted to load a package onto the UAV, and the object detection model comprises at least one of a machine learning (ML) charge pad detector, a ML fiducial marker detector, or a ML autoloader detector.
Kerzner teaches:
wherein: the reference objects include at least one of a charging pad adapted for charging the UAV, a fiducial navigation marker adapted for visual navigation of the UAV, or an autoloader adapted to load a package onto the UAV, and the object detection model comprises at least one of a machine learning (ML) charge pad detector, a ML fiducial marker detector, or a ML autoloader detector. (See at least [0049]: “As will be discussed in more detail below with respect to FIGS. 2A-2B, the environment map(s) generated and updated by the system 100 may include an indication of physical objects within the monitored property 120. These physical objects may be represented within the environment map(s) as landmarks. The landmarks may have or be associated with a location that indicates the location of the corresponding physical object within the monitored property 120…”, [0140]: “In some cases, identifying the landmarks in the area includes identifying an object or a surface of an object using one or more algorithms. The object or the surface of the object may be considered a landmark. For example, the drone 102 may use one or more machine learning algorithms to identify objects and/or particular surfaces of objects in the area of the property 120…” & [0146]: “…For example, if a surface of an object is classified as non-planar, the monitoring server 130 may determine that the surface and/or the object corresponding to the surface should not be used as a landmark (e.g., will not be used for navigation).”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it an obvious design choice combine Price’s method with Kerzner’s fiducial marker and ML fiducial marker detector. Doing so would be obvious so the UAV “can more accurately and efficiently navigate through uncontrolled environments, such as residences, commercial properties, etc.” (See [0025] of Kerzner).
Regarding claim 23, Price discloses all the limitations of claim 1 as discussed above.
Price does not explicitly teach:
further comprising: generating, based on the mapping of the correspondences, annotation data in the current aerial image corresponding to at least one of the reference objects labeled in the asset map,
wherein validating or informing the detection of the first object by the object detection model includes accepting, rejecting, or adjusting a confidence of the detection based on a spatial correspondence between the detection and the annotation data.
Kerzner teaches:
further comprising: generating, based on the mapping of the correspondences, annotation data in the current aerial image corresponding to at least one of the reference objects labeled in the asset map, (See at least Figs. 2A-2B, [0070]: “In analyzing the captured images, the monitoring server 130 may identify one or more visual landmarks from the captured images and/or the generated 3D environment map. In identifying one or more visual landmarks, the monitoring server 130 may employ segmentation to label areas within the captured images where physical objects are and/or what the physical objects are. Specifically, the monitoring server 130 may employ two dimensional (2D) scene segmentation, which maps each pixel to a different object or surface category, e.g., wall, floor, furniture, etc.…” & [0079]: “…In analyzing the captured images, the monitoring server 130 may identify one or more features of one or more of the identified visual landmarks. Typically, each identified feature will belong to a single visual landmark. Each of the features may be identified with a 3D representation. For example, once a visual landmark has been classified, the monitoring server 130 may refer to a lookup table for the classification or to a model for the classification to identify one or more features that are typical for the particular classification. For example, in regards to the table 126, the monitoring server 130 may refer to a lookup table for furniture tables. The lookup table may reveal that furniture tables typically have four legs and four corners…”)
wherein validating or informing the detection of the first object by the object detection model includes accepting, rejecting, or adjusting a confidence of the detection based on a spatial correspondence between the detection and the annotation data. (See at least [0071]: “In areas where the segmentation results differ, the monitoring server 130 may associate those areas with a lower confidence of the segmentation. The monitoring server 130 may actually calculate a confidence score for each area or may make a determination as to whether an area is acceptable, e.g., due to consistent segmentation results, or is unacceptable, e.g., due to differing segmentation results. Where an area is determined to have a confidence score below a first threshold level or is deemed unacceptable, the monitoring server 130 may label the area as having low confidence or as unacceptable. As an example, segmentation results may differ in areas of the monitored property 120 where appearance varies greatly by viewing angle, where a moving object is present, or where the segmentation algorithm is untrained for the surface(s) and/or physical object(s) present. In identifying one or more visual landmarks from the environment map, the monitoring server 130 may employ 3D scene segmentation. Each physical object identified through scene segmentation may be labelled as its own visual landmark. However, identified objects that were found within an area labelled as having low confidence or labelled as unacceptable may not be considered visual landmarks. This process may help to prevent moving objects such as people, animals, robots, etc. from being identified as visual landmarks.”)
Claim(s) 7 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Price in view of Kerzner and further in view of Shoeb of US 20230316740 A1, published 10/05/2023, hereinafter “Shoeb”.
Regarding claim 7, Price and Kerzner in combination teach all the limitations of claim 5 as discussed above.
Price and Kerzner in combination do not explicitly teach:
further comprising: transitioning from navigating the UAV based upon the mapping of the correspondences between the reference aerial image and the current aerial image to navigating the UAV based upon the detection of the first object by the object detection model as an above ground level (AGL) altitude of the UAV decreases.
Shoeb teaches:
further comprising: transitioning from navigating the UAV based upon the mapping of the correspondences between the reference aerial image and the current aerial image to navigating the UAV based upon the detection of the first object by the object detection model as an above ground level (AGL) altitude of the UAV decreases. (See at least [0124-0125]: “The operations at block 424 involve determining whether an obstacle was detected. If an obstacle is not detected, the operations repeat from block 405. If an obstacle is detected, the operations at block 430 are performed. The operations at block 430 involve controlling the UAV to avoid the obstacle. In some examples, the UAV is controlled to move in a lateral direction to avoid the obstacle during descent. For example, as illustrated in FIG. 9, the delivery zone is adjusted to avoid the obstacle (e.g., powerline 505F). Some examples of the UAV and/or ground control station determine the amount of lateral movement required to avoid the obstacle based in part on how far the obstacle extends within the delivery zone. Once a suitable delivery zone has been determined, the operations repeat from block 405. In some examples, the position of the UAV is controlled to avoid contact between a tethered payload of the UAV and the obstacle. After changing the UAV position, the tether of the UAV is controlled to deliver a tethered payload to the delivery zone.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Price and Kerzner’s method with Shoeb’s technique of transitioning from navigating the UAV based upon the mapping to navigating the UAV based upon the detection from the object detection model as an above ground level (AGL) altitude of the UAV decreases. Doing so would be obvious “to facilitate avoiding obstacles in a delivery zone or other environment of the UAV” (See [0004] of Shoeb).
Regarding claim 19, Price and Kerzner in combination teach all the limitations of claim 17 as discussed above.
Price and Kerzner in combination do not explicitly teach:
wherein the operations further comprise: transitioning from navigating the UAV based upon the mapping of the correspondences between the reference aerial image and the current aerial image to navigating the UAV based upon the detection of the first object by object detection model as an above ground level (AGL) altitude of the UAV decreases.
Shoeb teaches:
wherein the operations further comprise: transitioning from navigating the UAV based upon the mapping of the correspondences between the reference aerial image and the current aerial image to navigating the UAV based upon the detection of the first object by object detection model as an above ground level (AGL) altitude of the UAV decreases. (See at least [0124-0125]: “The operations at block 424 involve determining whether an obstacle was detected. If an obstacle is not detected, the operations repeat from block 405. If an obstacle is detected, the operations at block 430 are performed. The operations at block 430 involve controlling the UAV to avoid the obstacle. In some examples, the UAV is controlled to move in a lateral direction to avoid the obstacle during descent. For example, as illustrated in FIG. 9, the delivery zone is adjusted to avoid the obstacle (e.g., powerline 505F). Some examples of the UAV and/or ground control station determine the amount of lateral movement required to avoid the obstacle based in part on how far the obstacle extends within the delivery zone. Once a suitable delivery zone has been determined, the operations repeat from block 405. In some examples, the position of the UAV is controlled to avoid contact between a tethered payload of the UAV and the obstacle. After changing the UAV position, the tether of the UAV is controlled to deliver a tethered payload to the delivery zone.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Price and Kerzner’s method with Shoeb’s technique of transitioning from navigating the UAV based upon the mapping to navigating the UAV based upon the detection from the object detection model as an above ground level (AGL) altitude of the UAV decreases. Doing so would be obvious “to facilitate avoiding obstacles in a delivery zone or other environment of the UAV” (See [0004] of Shoeb).
Claim(s) 9 and 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Price in view of Shue of US 20200234043 A1, filed 10/13/2017, hereinafter “Shue”.
Regarding claim 9, Price discloses all the limitations of claim 1 as discussed above.
Price additionally teaches:
wherein the asset map is annotated with geolocation labels, (See at least [0018]: “…Accordingly, in some examples, the labelled satellite photograph(s) may be pre-loaded to assistive device 102 for use in such processing. For example, assistive device 102 may be pre-configured with global and/or regional satellite data with geospatial labels at any suitable resolution…”)
Price does not explicitly teach:
the method further comprising: identifying a gravity aligned pixel in the current aerial image based upon an attitude measurement output from a sensor disposed onboard the UAV, wherein the gravity aligned pixel corresponds to a portion of the current aerial image disposed immediately below the UAV along a gravity vector passing through the UAV; matching the gravity aligned pixel to a corresponding geolocated pixel in the reference aerial image of the asset map based upon the mapping of the correspondences between the reference aerial image and the current aerial image; and localizing the UAV based upon the matching.
However, Price teaches using the RANSAC homography computation to match an aerial photograph with a satellite photograph “and to use a scale and displacement in the matching of the photographs to determine a coordinate of a pixel location in the aerial photography” (See at least [0030]).
Furthermore, Shue teaches using orientation data from the UAV to rotate new images so they are at an angle consistent with historical images, such that, if the historical images were all taken “with a tilt of 0° relative to a plane orthogonal to a gravitational vector, but a gust of wind occurring when UAV 102 captured a new image caused the new image to be taken with a tilt of 5° relative to the plane”, then the new image can be rotated by -5° to align with the historical images (See at least [0041] & [0058]). Shue further teaches determining parameters such as yaw, attitude, tilt, and pitch based on orientation data generated by the UAV when the historical images were captured (See [0041] & [0058]). Therefore, since Price teaches determining a pixel coordinate by matching an aerial photograph with a satellite photograph and Shue teaches rotating the images so they are aligned, the combination of Price and Shue render obvious identifying a gravity aligned pixel corresponding with a gravity vector of the UAV in a current aerial image based on attitude of the UAV, matching the gravity aligned pixel to a corresponding geolocated pixel in a reference aerial image, and localizing the UAV based on the matching, which provides the benefit of “correct[ing] for yaw, attitude, and tilt differences” (See [0041] of Shue).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Price’s method with the teachings of Shue. Doing so would be obvious “to correct for yaw, attitude, and tilt differences” (See [0041] of Shue).
Regarding claim 21, Price discloses all the limitations of claim 13 as discussed above.
Price additionally teaches:
wherein the asset map is annotated with geolocation labels, (See at least [0018]: “…Accordingly, in some examples, the labelled satellite photograph(s) may be pre-loaded to assistive device 102 for use in such processing. For example, assistive device 102 may be pre-configured with global and/or regional satellite data with geospatial labels at any suitable resolution…”)
Price does not explicitly teach:
wherein the operations further comprise: identifying a gravity aligned pixel in the current aerial image based upon an attitude measurement output from a sensor disposed onboard the UAV, wherein the gravity aligned pixel corresponds to a portion of the current aerial image disposed immediately below the UAV along a gravity vector passing through the UAV; matching the gravity aligned pixel to a corresponding geolocated pixel in the reference aerial image of the asset map based upon the mapping of the correspondences; and localizing the UAV based upon the matching.
However, Price teaches using the RANSAC homography computation to match an aerial photograph with a satellite photograph “and to use a scale and displacement in the matching of the photographs to determine a coordinate of a pixel location in the aerial photography” (See at least [0030]).
Furthermore, Shue teaches using orientation data from the UAV to rotate new images so they are at an angle consistent with historical images, such that, if the historical images were all taken “with a tilt of 0° relative to a plane orthogonal to a gravitational vector, but a gust of wind occurring when UAV 102 captured a new image caused the new image to be taken with a tilt of 5° relative to the plane”, then the new image can be rotated by -5° to align with the historical images (See at least [0041] & [0058]). Shue further teaches determining parameters such as yaw, attitude, tilt, and pitch based on orientation data generated by the UAV when the historical images were captured (See [0041] & [0058]). Therefore, since Price teaches determining a pixel coordinate by matching an aerial photograph with a satellite photograph and Shue teaches rotating the images so they are aligned, the combination of Price and Shue render obvious identifying a gravity aligned pixel corresponding with a gravity vector of the UAV in a current aerial image based on attitude of the UAV, matching the gravity aligned pixel to a corresponding geolocated pixel in a reference aerial image, and localizing the UAV based on the matching, which provides the benefit of “correct[ing] for yaw, attitude, and tilt differences” (See [0041] of Shue).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Price’s method with the teachings of Shue. Doing so would be obvious “to correct for yaw, attitude, and tilt differences” (See [0041] of Shue).
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Price in view of Kugelmass of US 20160046374 A1, filed 05/29/2015, hereinafter “Kugelmass”, and further in view of Wang of CN 116105719 A, published 05/12/2023, hereinafter “Wang”.
Regarding claim 11, Price discloses all the limitations of claim 1 as discussed above.
Price additionally teaches:
further comprising: storing a library of asset maps including the asset map, (See at least [0018]: “…the labelled satellite photograph(s) may be pre-loaded to assistive device 102 for use in such processing. For example, assistive device 102 may be pre-configured with global and/or regional satellite data with geospatial labels at any suitable resolution…”)
Price does not explicitly teach:
wherein each of the asset maps corresponds to a different aerial image of the ground area captured from a different altitude or captured at a different time of day; and
Kugelmass teaches:
wherein each of the asset maps corresponds to a different aerial image of the ground area captured from a different altitude or captured at a different time of day; and (See at least [0101]: “…In this variation, Block S260 can retrieve one or more images stored from a previous mission, intersecting the ground area, and captured in a previous mission within a period of time from current that satisfies the time accuracy requirement, and Block S260 combines these stored images within images captured by the UAV during the current mission to generate the geospatial map. Block S260 can thus aggregate images captured over various missions or campaigns (i.e., in different periods of time) into the geospatial map of the selected ground area.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Price’s method with Kugelmass’s geospatial map corresponding to images captured over different periods of time. Doing so would be obvious so that “a user subsequently interfacing with the geospatial map can identify timeliness and geographic locations of visual data (i.e., pixels) within the geographic map by selecting corresponding pixels or groups of pixels” (See [0102] of Kugelmass).
Price and Kugelmass in combination do not explicitly teach:
selecting the asset map from the library of asset maps based upon at least one of a current altitude of the UAV or a current time of day when acquiring the current aerial image.
Wang teaches:
selecting the asset map from the library of asset maps based upon at least one of a current altitude of the UAV or a current time of day (See at least [0065]: “The local semantic map in this embodiment can be constructed online in real time based on the vehicle's driving process. For the vehicle's current location, it does not need to use all the previously constructed local semantic map data, but only a part of the local semantic map data close to the current time. Therefore, the optimized vehicle pose of the previous frame can be used as a reference to obtain the local semantic map corresponding to the optimized vehicle pose of the previous frame, that is, the local semantic map around the current vehicle.”)
Although Wang does not explicitly teach a current time of day when acquiring an aerial image, Wang does teach localizing a vehicle by obtaining image data of a current frame and performing semantic segmentation on the image to determine the relative transformation relationship between the semantic segmentation result of the current frame and the local semantic map, of which only the “part of the local semantic map data close to the current time” is used, as discussed above (See at least [0044] & [0065-0066]). Therefore, one having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to perform the teachings of Wang while taking into account the current time of day when acquiring an aerial image, such as the aerial image taught by Price (See at least [0021] of Price), since “it does not need to use all the previously constructed local semantic map data” (See [0065] of Wang).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Price and Kugelmass’s method with Wang’s technique of selecting the asset map from the library of asset maps based on a current time of day when acquiring an image. Doing so would be obvious since “it does not need to use all the previously constructed local semantic map data” (See [0065] of Wang).
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Price in view of Yan of CN 118427383 A, published 08/02/2024, hereinafter “Yan”.
Regarding claim 12, Price discloses all the limitations of claim 1 as discussed above.
Price additionally teaches:
further comprising: storing a library of asset maps including the asset map, wherein each of the asset maps corresponds to a different aerial image of the ground area, and (See at least [0018]: “…the labelled satellite photograph(s) may be pre-loaded to assistive device 102 for use in such processing. For example, assistive device 102 may be pre-configured with global and/or regional satellite data with geospatial labels at any suitable resolution. For example, assistive device 102 may be configured to download a regional satellite image database including a plurality of satellite photographs and pre-defined geospatial coordinates, so that photography of the nearby environment may be correlated against one or more of the plurality of satellite photographs…”)
Price does not explicitly teach:
wherein the asset maps are indexed to reference vector embeddings; and
generating a current vector embedding based on the current aerial image; and
selecting the asset map from the library of asset maps by comparing the current vector embedding to the reference vector embeddings.
Yan teaches:
wherein the asset maps are indexed to reference vector embeddings; and (See at least [0027]: “Phase 2 (Inference): Convert the query image into an embedding vector, retrieve the closest vector from the vector database obtained in the previous phase, complete the image matching, and obtain the location data.”)
generating a current vector embedding based on the current aerial image; and (See at least [0008]: “Further, step S1 includes: S11: generating key semantic features of the image using a visual language model; S12: constructing a natural language scene description from the key semantic features obtained in step S11; S13: converting the language scene description obtained in step S12 into an embedding vector and saving it” & [0011]: “Furthermore, in step S13, a word embedding model is used to convert the semantic description of each reference image into a corresponding embedding vector and save it for querying.”)
selecting the asset map from the library of asset maps by comparing the current vector embedding to the reference vector embeddings. (See at least [0014]: “Further, step S2 includes: S21: For the input query image, use the same method as in step S1 to generate an embedding vector; S22: In the vector corresponding to the reference image, find the vector that is closest to the input embedding vector through cosine similarity, thereby finding the corresponding reference image and completing image matching; S23: Read the corresponding vehicle location information from the label of the reference image to complete the localization of the vehicle”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Price’s method with Yan’s technique of the asset maps being indexed to reference vector embeddings, generating a current vector embedding based on the current aerial image, and selecting the asset map from the library of asset maps by comparing the current vector embedding to the reference vector embeddings. Doing so would be obvious for “improving the positioning accuracy” (See [0042] of Yan).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIKKI MARIE M MOLINA whose telephone number is (571)272-5180. The examiner can normally be reached M-F, 9am-6pm PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aniss Chad can be reached at 571-270-3832. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NIKKI MARIE M MOLINA/Examiner, Art Unit 3662
/ANISS CHAD/Supervisory Patent Examiner, Art Unit 3662
Prosecution Insights