DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a Non-final Office Action on the merits. Claims 1-4 and 6-23 are currently pending and are addressed below.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 08/05/2026 has been entered.
Information Disclosure Statement
The information disclosure statement(s) (IDS) dated 09/14/2026 is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement(s) is/are being considered by the examiner.
Response to Amendment
Claims 1-23 were rejected under 35 U.S.C. 101 as being allegedly directed to an abstract idea without significantly more. Applicant amended the claims accordingly; therefore, the rejection is withdrawn.
Response to Arguments
Applicant' s arguments on pages 11-14 of the response, with respect to the rejection(s) of claim(s) 1, 8, 13, and 20 under 35 U.S.C. 102 and claim(s) 2-7, 9-12, 14-19, and 21-23 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Nassar and Harris.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 13-17, and 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nassar of “A Deep CNN-Based Framework For Enhanced Aerial Imagery Registration with Applications to UAV Geolocalization”, published 06/18/2018, hereinafter “Nassar”, in view of Harris of US 20170090271 A1, published 03/30/2017, hereinafter “Harris”.
Regarding claim 1, Nassar teaches:
A method performed by an unmanned aerial vehicle (UAV), the method comprising: (See at least Abstract, lines 1-3: “In this paper we present a novel framework for geolocalizing Unmanned Aerial Vehicles (UAVs) using only their onboard camera.”)
storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image; (See at least Section 3.1, par. 3: “Reference Map Images (Satellite) The framework uses a reference map R with known GPS bounds”, Section 3.2, par. 1: “The Calibration component is responsible for computing an affine geometric transformation between the UAV image and the reference map image in order to map the UAV onto a reference map location, i.e. map S(i) with its field of view extents to corresponding position and covered region in R, taking into consideration the difference in scale and orientation between the two images. Calibration is an integral part of the framework, and is called upon on at S(1) and every three iterations (S(i+3)) frames to autocorrect any drift that might occur. To accomplish this task, SIFT feature points are extracted from the UAV image and the reference image and subsequently matched. Even though SIFT feature computation is time consuming, the calibration process is applied every several frames to maintain real-time performance. As previously stated, the initial GPS coordinates of the UAV are assumed to be known and so a region of interest r can be cropped from the reference map R where r =R(l,wS+b) is the coordinate of the center pixel” & Section 3.5, par. 1: “Applying the semantic segmentation step to S(i) and results in two images containing only buildings and roads as seen in Figure 6.”)
acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area; (See at least Section 3.1, par. 2: “The framework accepts a video sequence from the UAV denoted as S. From S, we can extract S(i) (video frame) which we compare to a reference map (R).”)
mapping correspondences between the reference aerial image and the current aerial image using a homography estimating tool executing onboard the UAV; (See at least Section 3.2, par. 2: “Using the created reduced search space, features are extracted from both r and S(i). Subsequently, the features are matched and a statistical estimator, such as RANSAC [14], is used to estimate a homography matrix that describes the transformation between UAV and reference image pairs. During UAV flight, the Calibration procedure is applied every several frames to sustain reasonably accurate translation, rotation and scale components between UAV images and reference map images.”)
analyzing the current aerial image with an object detection model to detect a first object positioned at the ground area; (See at least Section 3.4, par. 1: “Semantic segmentation is utilized to help extract meaningful shapes in UAV and satellite images, such as roads and buildings, and pass them to the SSM component for matching. To be able to accomplish this, S(i) and r are fed into U-Net [47] network. U-Net is an encoding and decoding model that was originally proposed for biomedical image segmentation. It is chosen due its performance and prominence in recent EO tasks [30] in comparison to FCN[33] and SegNet[5]. In fact adding skip connections to FCN [23]provides finer segmentation which is already implemented in U-Net. This enables the classification of our image pixels into many classes. However, for registration, we rely on building and road classes as shown in Figure 5.”)
validating or informing a detection of the first object by the object detection model based on the mapping of the correspondences including generating annotation data in the current aerial image corresponding to one of the reference objects labeled in the asset map and determining whether the first object detected by the object detection model is the same object as the one of the reference objects represented by the annotation data; and (See at least Figs. 3 and 6-7 & Section 3.5, par. 1-2: “Applying the semantic segmentation step to S(i) and r results in two images containing only buildings and roads as seen in Figure 6. Next, the two images are each split into two layers: building and road objects. Morphological techniques and contouring are applied to each layer to find the boundaries of different objects and extract shape descriptor information to be used in the matching process such as location, area, roundedness, etc. In fact, knowledge of object types provides information on the surroundings of the UAV which could be used recurrently to improve segmentation results and can also be beneficial in some applications such surveillance, parcel delivery, etc. The Semantic Shape Matching SSM pipeline accepts a classified image with a box drawn around each contour and executes the following process (all steps are applied similarly to both images): 1. Morphological operations such as dilation followed by erosion are used to fill gaps in blobs or shapes [18]. 2. Small shapes or blobs are filtered using the area of the shape found using area filtering, and using erosion followed by dilation which is another morphological operation that removes outlier pixels or noise. 3. A dictionary is built containing all the different shape features extracted. These features are the shapes’ area, location, radius, centroid, and orientation, which are all calculated using Hu moments [20]. 4. Using brute force matching, a scoring system is implemented to pick the matching pair. Each matching pair is awarded points based on if they have similar features within a certain tolerance. These features are also weighted due to their importance, for example matching shapes’ area is more important than their centroid. 5. Matches found are checked to see if a shape matched more than once, then these matches are filtered once more based on the scoring system devised by giving higher points to certain combinations of similar features. The scoring system gives higher points to certain combinations of similar features. For instance, more points are awarded to matched shapes if they share the same area, distance, and orientation, rather than sharing the same area, centroid, and radius. Afterwards, matches with the highest scores are selected. However, matches are eliminated if the score is near similar and they are within a short distance since this might be an unreliable match between two objects that are of the same size and within the same distance, as shown in Figure 7. Finally, matched shapes are used to calculate a homography that is then applied to adjust the current location of the UAV with improved accuracy.”)
Nassar does not explicitly teach:
autonomously navigating the UAV relative to the first object based upon the detection of the first object by the object detection model and based upon the mapping of the correspondences between the reference aerial image and the current aerial image using the homography estimating tool.
Harris teaches:
autonomously navigating the UAV relative to the first object based upon the detection of the first object by the object detection model and based upon the mapping of the correspondences between the reference aerial image and the current aerial image using the homography estimating tool. (See at least [0022-0023]: “For example, feature points of a first image captured at a first altitude may be paired with feature points of a second image captured at a second, different altitude. A homography may be computed for each paired feature point to confirm that the feature points are properly paired. A homography will exist if the two feature points are on the same plane (e.g., both feature points are on the ground). A homography will not exist between the paired feature points if the feature points are not on the same plane (e.g., the first feature point is on the ground and the second feature point is on a chair). Paired feature points that do not have a homography are discarded. Paired feature points that do have a homography are used to register the first image and the second image to form a registered image set. The registered image set is then used to generate a depth disparity map of the destination that provides an indication of objects at different distances from the UAV (i.e., objects in different planes). The depth disparity map may then be utilized to confirm whether objects are within the destination and/or whether there is a substantially planar surface available at the destination and of a size sufficient for the UAV to land, deliver a payload (e.g., package), etc.” & [0060]: “The example process 200 and corresponding example sub-process 300 may be performed each time a UAV descends toward a destination to survey the destination and determine whether it is safe to land and/or descend to a defined altitude at the destination without potentially harming any objects positioned at the destination. As discussed above, the example process 200 and corresponding example sub-process may be periodically and/or continually performed during descent of the UAV.” See also [0074] regarding the UAV control system.)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nassar’s method with Harris’s technique of autonomously navigating the UAV relative to the first object based upon the detection of the first object by the object detection model and based upon the mapping of the correspondences between the reference aerial image and the current aerial image using the homography estimating tool. Doing so would be obvious “To confirm that the destination is clear of objects and includes a safe landing or delivery location”, “for surveying a destination as an unmanned aerial vehicle (“UAV”) descends toward the destination”, and to “complete delivery of a payload” (See Abstract, [0021] & [0034] of Harris).
Regarding claim 2, Nassar and Harris in combination teach all the limitations of claim 1 as discussed above.
Nassar additionally teaches:
further comprising: annotating the current aerial image with one or more annotations corresponding to one or more of the reference objects from the asset map based on the mapping of the correspondences between the reference aerial image and the current aerial image, wherein validating or informing the detection of the first object by the object detection model is based on the annotating. (See at least Figs. 3 and 6-7 & Section 3.5, par. 1-2: “Applying the semantic segmentation step to S(i) and r results in two images containing only buildings and roads as seen in Figure 6. Next, the two images are each split into two layers: building and road objects. Morphological techniques and contouring are applied to each layer to find the boundaries of different objects and extract shape descriptor information to be used in the matching process such as location, area, roundedness, etc. In fact, knowledge of object types provides information on the surroundings of the UAV which could be used recurrently to improve segmentation results and can also be beneficial in some applications such surveillance, parcel delivery, etc. The Semantic Shape Matching SSM pipeline accepts a classified image with a box drawn around each contour and executes the following process (all steps are applied similarly to both images): 1. Morphological operations such as dilation followed by erosion are used to fill gaps in blobs or shapes [18]. 2. Small shapes or blobs are filtered using the area of the shape found using area filtering, and using erosion followed by dilation which is another morphological operation that removes outlier pixels or noise. 3. A dictionary is built containing all the different shape features extracted. These features are the shapes’ area, location, radius, centroid, and orientation, which are all calculated using Hu moments [20]. 4. Using brute force matching, a scoring system is implemented to pick the matching pair. Each matching pair is awarded points based on if they have similar features within a certain tolerance. These features are also weighted due to their importance, for example matching shapes’ area is more important than their centroid. 5. Matches found are checked to see if a shape matched more than once, then these matches are filtered once more based on the scoring system devised by giving higher points to certain combinations of similar features. The scoring system gives higher points to certain combinations of similar features. For instance, more points are awarded to matched shapes if they share the same area, distance, and orientation, rather than sharing the same area, centroid, and radius. Afterwards, matches with the highest scores are selected. However, matches are eliminated if the score is near similar and they are within a short distance since this might be an unreliable match between two objects that are of the same size and within the same distance, as shown in Figure 7. Finally, matched shapes are used to calculate a homography that is then applied to adjust the current location of the UAV with improved accuracy.”)
Regarding claim 3, Nassar and Harris in combination teach all the limitations of claim 2 as discussed above.
Nassar additionally teaches:
wherein validating or informing the detection of the first object by the object detection model comprises: affirming or upweighting the detection of the first object when the detection of the first object overlaps with one of the one or more annotations annotating the current aerial image and sourced from the asset map. (See at least Figs. 3 and 6-7 & Section 3.5, par. 1-2: “Applying the semantic segmentation step to S(i) and r results in two images containing only buildings and roads as seen in Figure 6. Next, the two images are each split into two layers: building and road objects. Morphological techniques and contouring are applied to each layer to find the boundaries of different objects and extract shape descriptor information to be used in the matching process such as location, area, roundedness, etc. In fact, knowledge of object types provides information on the surroundings of the UAV which could be used recurrently to improve segmentation results and can also be beneficial in some applications such surveillance, parcel delivery, etc. The Semantic Shape Matching SSM pipeline accepts a classified image with a box drawn around each contour and executes the following process (all steps are applied similarly to both images): 1. Morphological operations such as dilation followed by erosion are used to fill gaps in blobs or shapes [18]. 2. Small shapes or blobs are filtered using the area of the shape found using area filtering, and using erosion followed by dilation which is another morphological operation that removes outlier pixels or noise. 3. A dictionary is built containing all the different shape features extracted. These features are the shapes’ area, location, radius, centroid, and orientation, which are all calculated using Hu moments [20]. 4. Using brute force matching, a scoring system is implemented to pick the matching pair. Each matching pair is awarded points based on if they have similar features within a certain tolerance. These features are also weighted due to their importance, for example matching shapes’ area is more important than their centroid. 5. Matches found are checked to see if a shape matched more than once, then these matches are filtered once more based on the scoring system devised by giving higher points to certain combinations of similar features. The scoring system gives higher points to certain combinations of similar features. For instance, more points are awarded to matched shapes if they share the same area, distance, and orientation, rather than sharing the same area, centroid, and radius. Afterwards, matches with the highest scores are selected. However, matches are eliminated if the score is near similar and they are within a short distance since this might be an unreliable match between two objects that are of the same size and within the same distance, as shown in Figure 7. Finally, matched shapes are used to calculate a homography that is then applied to adjust the current location of the UAV with improved accuracy.”)
Regarding claim 4, Nassar and Harris in combination teach all the limitations of claim 2 as discussed above.
Harris additionally teaches:
wherein validating or informing the detection of the first object by the object detection model comprises: masking or downweighting the detection of the first object when the detection of the first object does not overlap with one of the one or more annotations annotating the current aerial image and sourced from the asset map. (See at least Figs. 3 and 6-7 & Section 3.5, par. 1-2: “Applying the semantic segmentation step to S(i) and r results in two images containing only buildings and roads as seen in Figure 6. Next, the two images are each split into two layers: building and road objects. Morphological techniques and contouring are applied to each layer to find the boundaries of different objects and extract shape descriptor information to be used in the matching process such as location, area, roundedness, etc. In fact, knowledge of object types provides information on the surroundings of the UAV which could be used recurrently to improve segmentation results and can also be beneficial in some applications such surveillance, parcel delivery, etc. The Semantic Shape Matching SSM pipeline accepts a classified image with a box drawn around each contour and executes the following process (all steps are applied similarly to both images): 1. Morphological operations such as dilation followed by erosion are used to fill gaps in blobs or shapes [18]. 2. Small shapes or blobs are filtered using the area of the shape found using area filtering, and using erosion followed by dilation which is another morphological operation that removes outlier pixels or noise. 3. A dictionary is built containing all the different shape features extracted. These features are the shapes’ area, location, radius, centroid, and orientation, which are all calculated using Hu moments [20]. 4. Using brute force matching, a scoring system is implemented to pick the matching pair. Each matching pair is awarded points based on if they have similar features within a certain tolerance. These features are also weighted due to their importance, for example matching shapes’ area is more important than their centroid. 5. Matches found are checked to see if a shape matched more than once, then these matches are filtered once more based on the scoring system devised by giving higher points to certain combinations of similar features. The scoring system gives higher points to certain combinations of similar features. For instance, more points are awarded to matched shapes if they share the same area, distance, and orientation, rather than sharing the same area, centroid, and radius. Afterwards, matches with the highest scores are selected. However, matches are eliminated if the score is near similar and they are within a short distance since this might be an unreliable match between two objects that are of the same size and within the same distance, as shown in Figure 7. Finally, matched shapes are used to calculate a homography that is then applied to adjust the current location of the UAV with improved accuracy.”)
Regarding claim 13, Nassar teaches:
At least one non-transitory machine-accessible storage medium that provides instructions that, when executed by an unmanned aerial vehicle (UAV), will cause the UAV to perform operations comprising: (See at least Abstract, lines 1-3: “In this paper we present a novel framework for geolocalizing Unmanned Aerial Vehicles (UAVs) using only their onboard camera.”)
storing an asset map of a ground area, wherein the asset map includes a reference aerial image of the ground area annotated with labels describing reference objects depicted in the reference aerial image; (See at least Section 3.1, par. 3: “Reference Map Images (Satellite) The framework uses a reference map R with known GPS bounds”, Section 3.2, par. 1: “The Calibration component is responsible for computing an affine geometric transformation between the UAV image and the reference map image in order to map the UAV onto a reference map location, i.e. map S(i) with its field of view extents to corresponding position and covered region in R, taking into consideration the difference in scale and orientation between the two images. Calibration is an integral part of the framework, and is called upon on at S(1) and every three iterations (S(i+3)) frames to autocorrect any drift that might occur. To accomplish this task, SIFT feature points are extracted from the UAV image and the reference image and subsequently matched. Even though SIFT feature computation is time consuming, the calibration process is applied every several frames to maintain real-time performance. As previously stated, the initial GPS coordinates of the UAV are assumed to be known and so a region of interest r can be cropped from the reference map R where r =R(l,wS+b) is the coordinate of the center pixel” & Section 3.5, par. 1: “Applying the semantic segmentation step to S(i) and results in two images containing only buildings and roads as seen in Figure 6.”)
acquiring a current aerial image of the ground area with an onboard camera system of the UAV while the UAV is flying above the ground area; (See at least Section 3.1, par. 2: “The framework accepts a video sequence from the UAV denoted as S. From S, we can extract S(i) (video frame) which we compare to a reference map (R).”)
mapping correspondences between the reference aerial image and the current aerial image using a homography estimating tool executing onboard the UAV; (See at least Section 3.2, par. 2: “Using the created reduced search space, features are extracted from both r and S(i). Subsequently, the features are matched and a statistical estimator, such as RANSAC [14], is used to estimate a homography matrix that describes the transformation between UAV and reference image pairs. During UAV flight, the Calibration procedure is applied every several frames to sustain reasonably accurate translation, rotation and scale components between UAV images and reference map images.”)
analyzing the current aerial image with an object detection model to detect a first object positioned at the ground area; and (See at least Section 3.4, par. 1: “Semantic segmentation is utilized to help extract meaningful shapes in UAV and satellite images, such as roads and buildings, and pass them to the SSM component for matching. To be able to accomplish this, S(i) and r are fed into U-Net [47] network. U-Net is an encoding and decoding model that was originally proposed for biomedical image segmentation. It is chosen due its performance and prominence in recent EO tasks [30] in comparison to FCN[33] and SegNet[5]. In fact adding skip connections to FCN [23]provides finer segmentation which is already implemented in U-Net. This enables the classification of our image pixels into many classes. However, for registration, we rely on building and road classes as shown in Figure 5.”)
validating or informing a detection of the first object by the object detection model based on the mapping of the correspondences including generating annotation data in the current aerial image corresponding to one of the reference objects labeled in the asset map and determining whether the first object detected by the object detection model is the same object as the one of the reference objects represented by the annotation data; and (See at least Figs. 3 and 6-7 & Section 3.5, par. 1-2: “Applying the semantic segmentation step to S(i) and r results in two images containing only buildings and roads as seen in Figure 6. Next, the two images are each split into two layers: building and road objects. Morphological techniques and contouring are applied to each layer to find the boundaries of different objects and extract shape descriptor information to be used in the matching process such as location, area, roundedness, etc. In fact, knowledge of object types provides information on the surroundings of the UAV which could be used recurrently to improve segmentation results and can also be beneficial in some applications such surveillance, parcel delivery, etc. The Semantic Shape Matching SSM pipeline accepts a classified image with a box drawn around each contour and executes the following process (all steps are applied similarly to both images): 1. Morphological operations such as dilation followed by erosion are used to fill gaps in blobs or shapes [18]. 2. Small shapes or blobs are filtered using the area of the shape found using area filtering, and using erosion followed by dilation which is another morphological operation that removes outlier pixels or noise. 3. A dictionary is built containing all the different shape features extracted. These features are the shapes’ area, location, radius, centroid, and orientation, which are all calculated using Hu moments [20]. 4. Using brute force matching, a scoring system is implemented to pick the matching pair. Each matching pair is awarded points based on if they have similar features within a certain tolerance. These features are also weighted due to their importance, for example matching shapes’ area is more important than their centroid. 5. Matches found are checked to see if a shape matched more than once, then these matches are filtered once more based on the scoring system devised by giving higher points to certain combinations of similar features. The scoring system gives higher points to certain combinations of similar features. For instance, more points are awarded to matched shapes if they share the same area, distance, and orientation, rather than sharing the same area, centroid, and radius. Afterwards, matches with the highest scores are selected. However, matches are eliminated if the score is near similar and they are within a short distance since this might be an unreliable match between two objects that are of the same size and within the same distance, as shown in Figure 7. Finally, matched shapes are used to calculate a homography that is then applied to adjust the current location of the UAV with improved accuracy.”)
Nassar does not explicitly teach:
autonomously navigating the UAV relative to the first object based upon the detection of the first object by the object detection model and based upon the mapping of the correspondences between the reference aerial image and the current aerial image using the homography estimating tool.
Harris teaches:
autonomously navigating the UAV relative to the first object based upon the detection of the first object by the object detection model and based upon the mapping of the correspondences between the reference aerial image and the current aerial image using the homography estimating tool. (See at least [0022-0023]: “For example, feature points of a first image captured at a first altitude may be paired with feature points of a second image captured at a second, different altitude. A homography may be computed for each paired feature point to confirm that the feature points are properly paired. A homography will exist if the two feature points are on the same plane (e.g., both feature points are on the ground). A homography will not exist between the paired feature points if the feature points are not on the same plane (e.g., the first feature point is on the ground and the second feature point is on a chair). Paired feature points that do not have a homography are discarded. Paired feature points that do have a homography are used to register the first image and the second image to form a registered image set. The registered image set is then used to generate a depth disparity map of the destination that provides an indication of objects at different distances from the UAV (i.e., objects in different planes). The depth disparity map may then be utilized to confirm whether objects are within the destination and/or whether there is a substantially planar surface available at the destination and of a size sufficient for the UAV to land, deliver a payload (e.g., package), etc.” & [0060]: “The example process 200 and corresponding example sub-process 300 may be performed each time a UAV descends toward a destination to survey the destination and determine whether it is safe to land and/or descend to a defined altitude at the destination without potentially harming any objects positioned at the destination. As discussed above, the example process 200 and corresponding example sub-process may be periodically and/or continually performed during descent of the UAV.” See also [0074] regarding the UAV control system.)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nassar’s method with Harris’s technique of autonomously navigating the UAV relative to the first object based upon the detection of the first object by the object detection model and based upon the mapping of the correspondences between the reference aerial image and the current aerial image using the homography estimating tool. Doing so would be obvious “To confirm that the destination is clear of objects and includes a safe landing or delivery location”, “for surveying a destination as an unmanned aerial vehicle (“UAV”) descends toward the destination”, and to “complete delivery of a payload” (See Abstract, [0021] & [0034] of Harris).
Regarding claim 14, Nassar and Harris in combination teach all the limitations of claim 13 as discussed above.
Nassar additionally teaches:
wherein the operations further comprise: annotating the current aerial image with one or more annotations corresponding to one or more of the reference objects from the asset map based on the mapping of the correspondences between the reference aerial image and the current aerial image, wherein validating or informing the detection of the first object by the object detection model is based on the annotating. (See at least Figs. 3 and 6-7 & Section 3.5, par. 1-2: “Applying the semantic segmentation step to S(i) and r results in two images containing only buildings and roads as seen in Figure 6. Next, the two images are each split into two layers: building and road objects. Morphological techniques and contouring are applied to each layer to find the boundaries of different objects and extract shape descriptor information to be used in the matching process such as location, area, roundedness, etc. In fact, knowledge of object types provides information on the surroundings of the UAV which could be used recurrently to improve segmentation results and can also be beneficial in some applications such surveillance, parcel delivery, etc. The Semantic Shape Matching SSM pipeline accepts a classified image with a box drawn around each contour and executes the following process (all steps are applied similarly to both images): 1. Morphological operations such as dilation followed by erosion are used to fill gaps in blobs or shapes [18]. 2. Small shapes or blobs are filtered using the area of the shape found using area filtering, and using erosion followed by dilation which is another morphological operation that removes outlier pixels or noise. 3. A dictionary is built containing all the different shape features extracted. These features are the shapes’ area, location, radius, centroid, and orientation, which are all calculated using Hu moments [20]. 4. Using brute force matching, a scoring system is implemented to pick the matching pair. Each matching pair is awarded points based on if they have similar features within a certain tolerance. These features are also weighted due to their importance, for example matching shapes’ area is more important than their centroid. 5. Matches found are checked to see if a shape matched more than once, then these matches are filtered once more based on the scoring system devised by giving higher points to certain combinations of similar features. The scoring system gives higher points to certain combinations of similar features. For instance, more points are awarded to matched shapes if they share the same area, distance, and orientation, rather than sharing the same area, centroid, and radius. Afterwards, matches with the highest scores are selected. However, matches are eliminated if the score is near similar and they are within a short distance since this might be an unreliable match between two objects that are of the same size and within the same distance, as shown in Figure 7. Finally, matched shapes are used to calculate a homography that is then applied to adjust the current location of the UAV with improved accuracy.”)
Regarding claim 15, Nassar and Harris in combination teach all the limitations of claim 14 as discussed above.
Nassar additionally teaches:
wherein validating or informing the detection of the first object by the object detection model comprises: affirming or upweighting the detection of the first object when the detection of the first object overlaps with one of the one or more annotations annotating the current aerial image and sourced from the asset map. (See at least Figs. 3 and 6-7 & Section 3.5, par. 1-2: “Applying the semantic segmentation step to S(i) and r results in two images containing only buildings and roads as seen in Figure 6. Next, the two images are each split into two layers: building and road objects. Morphological techniques and contouring are applied to each layer to find the boundaries of different objects and extract shape descriptor information to be used in the matching process such as location, area, roundedness, etc. In fact, knowledge of object types provides information on the surroundings of the UAV which could be used recurrently to improve segmentation results and can also be beneficial in some applications such surveillance, parcel delivery, etc. The Semantic Shape Matching SSM pipeline accepts a classified image with a box drawn around each contour and executes the following process (all steps are applied similarly to both images): 1. Morphological operations such as dilation followed by erosion are used to fill gaps in blobs or shapes [18]. 2. Small shapes or blobs are filtered using the area of the shape found using area filtering, and using erosion followed by dilation which is another morphological operation that removes outlier pixels or noise. 3. A dictionary is built containing all the different shape features extracted. These features are the shapes’ area, location, radius, centroid, and orientation, which are all calculated using Hu moments [20]. 4. Using brute force matching, a scoring system is implemented to pick the matching pair. Each matching pair is awarded points based on if they have similar features within a certain tolerance. These features are also weighted due to their importance, for example matching shapes’ area is more important than their centroid. 5. Matches found are checked to see if a shape matched more than once, then these matches are filtered once more based on the scoring system devised by giving higher points to certain combinations of similar features. The scoring system gives higher points to certain combinations of similar features. For instance, more points are awarded to matched shapes if they share the same area, distance, and orientation, rather than sharing the same area, centroid, and radius. Afterwards, matches with the highest scores are selected. However, matches are eliminated if the score is near similar and they are within a short distance since this might be an unreliable match between two objects that are of the same size and within the same distance, as shown in Figure 7. Finally, matched shapes are used to calculate a homography that is then applied to adjust the current location of the UAV with improved accuracy.”)
Regarding claim 16, Nassar and Harris in combination teach all the limitations of claim 14 as discussed above.
Nassar additionally teaches:
wherein validating or informing the detection of the first object by the object detection model comprises: masking or downweighting the detection of the first object when the detection of the first object does not overlap with one of the one or more annotations annotating the current aerial image and sourced from the asset map. (See at least Figs. 3 and 6-7 & Section 3.5, par. 1-2: “Applying the semantic segmentation step to S(i) and r results in two images containing only buildings and roads as seen in Figure 6. Next, the two images are each split into two layers: building and road objects. Morphological techniques and contouring are applied to each layer to find the boundaries of different objects and extract shape descriptor information to be used in the matching process such as location, area, roundedness, etc. In fact, knowledge of object types provides information on the surroundings of the UAV which could be used recurrently to improve segmentation results and can also be beneficial in some applications such surveillance, parcel delivery, etc. The Semantic Shape Matching SSM pipeline accepts a classified image with a box drawn around each contour and executes the following process (all steps are applied similarly to both images): 1. Morphological operations such as dilation followed by erosion are used to fill gaps in blobs or shapes [18]. 2. Small shapes or blobs are filtered using the area of the shape found using area filtering, and using erosion followed by dilation which is another morphological operation that removes outlier pixels or noise. 3. A dictionary is built containing all the different shape features extracted. These features are the shapes’ area, location, radius, centroid, and orientation, which are all calculated using Hu moments [20]. 4. Using brute force matching, a scoring system is implemented to pick the matching pair. Each matching pair is awarded points based on if they have similar features within a certain tolerance. These features are also weighted due to their importance, for example matching shapes’ area is more important than their centroid. 5. Matches found are checked to see if a shape matched more than once, then these matches are filtered once more based on the scoring system devised by giving higher points to certain combinations of similar features. The scoring system gives higher points to certain combinations of similar features. For instance, more points are awarded to matched shapes if they share the same area, distance, and orientation, rather than sharing the same area, centroid, and radius. Afterwards, matches with the highest scores are selected. However, matches are eliminated if the score is near similar and they are within a short distance since this might be an unreliable match between two objects that are of the same size and within the same distance, as shown in Figure 7. Finally, matched shapes are used to calculate a homography that is then applied to adjust the current location of the UAV with improved accuracy.”)
Regarding claim 17, Nassar and Harris in combination teach all the limitations of claim 14 as discussed above.
Harris additionally teaches:
wherein the operations further comprise: navigating the UAV relative to the first object based upon the detection of the first object by the object detection model and based upon the mapping of the correspondences between the reference aerial image and the current aerial image using the homography estimating tool. (See at least [0022-0023]: “For example, feature points of a first image captured at a first altitude may be paired with feature points of a second image captured at a second, different altitude. A homography may be computed for each paired feature point to confirm that the feature points are properly paired. A homography will exist if the two feature points are on the same plane (e.g., both feature points are on the ground). A homography will not exist between the paired feature points if the feature points are not on the same plane (e.g., the first feature point is on the ground and the second feature point is on a chair). Paired feature points that do not have a homography are discarded. Paired feature points that do have a homography are used to register the first image and the second image to form a registered image set. The registered image set is then used to generate a depth disparity map of the destination that provides an indication of objects at different distances from the UAV (i.e., objects in different planes). The depth disparity map may then be utilized to confirm whether objects are within the destination and/or whether there is a substantially planar surface available at the destination and of a size sufficient for the UAV to land, deliver a payload (e.g., package), etc.” & [0060]: “The example process 200 and corresponding example sub-process 300 may be performed each time a UAV descends toward a destination to survey the destination and determine whether it is safe to land and/or descend to a defined altitude at the destination without potentially harming any objects positioned at the destination. As discussed above, the example process 200 and corresponding example sub-process may be periodically and/or continually performed during descent of the UAV.” See also [0074] regarding the UAV control system.)
Regarding claim 23, Nassar and Harris in combination teach all the limitations of claim 1 as discussed above.
Nassar additionally teaches:
wherein validating or informing the detection of the first object by the object detection model includes accepting, rejecting, or adjusting a confidence of the detection based on a spatial correspondence between the detection and the annotation data. (See at least Section 3.5, par. 7: “The scoring system gives higher points to certain combinations of similar features. For instance, more points are awarded to matched shapes if they share the same area, distance, and orientation, rather than sharing the same area, centroid, and radius. Afterwards, matches with the highest scores are selected. However, matches are eliminated if the score is near similar and they are within a short distance since this might be an unreliable match between two objects that are of the same size and within the same distance, as shown in Figure 7. Finally, matched shapes are used to calculate a homography that is then applied to adjust the current location of the UAV with improved accuracy.”)
Claim(s) 6, 10, 18, and 22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nassar in view of Harris and further in view of Kerzner of US 20210110137 A1, filed 10/15/2020, hereinafter “Kerzner”.
Regarding claim 6, Nassar and Harris in combination teach all the limitations of claim 2 as discussed above.
Nassar and Harris in combination do not explicitly teach:
wherein the UAV navigates with reference to the first object only after the detection of the first object by the object detection model registers to a corresponding one of the annotations sourced from the asset map.
Kerzner teaches:
wherein the UAV navigates with reference to the first object only after the detection of the first object by the object detection model registers to a corresponding one of the annotations sourced from the asset map. (See at least Fig. 7, [0189]: “The process 500 includes determining a path to navigate in the area based on the location (508). For example, as shown in FIG. 3, the drone 302 can generate a route through the floor 220 such that one or more landmarks of the floor 220 will be observable and/or unobstructed along the entirety of the route or at certain points along the route. Specifically, the drone 302 (or the monitoring server 130) may generate a path in order to keep the drone 302 within a threshold distance (e.g., 1.0 m, 1.5 m, 1.7 m, etc.) of at least a threshold number of landmarks at any given time (e.g., at least two landmarks, at least three landmarks, etc.)…” & [0289]: “The process 700 includes determining a path to navigate in the area based on the location (708). For example, as shown in FIG. 3, the drone 302 can generate a route through the floor 220 such that one or more landmarks, such as one or more planar landmarks, of the floor 220 will be observable and/or unobstructed along the entirety of the route or at certain points along the route. As shown in FIG. 3, the drone 302 generates a route from its current position 306b, to a second position 310 near the landmark 238, to a third position 312 near the landmark 234, and to a fourth position 314 near the landmark 232.”)
Regarding claim 10, Nassar and Harris in combination teach all the limitations of claim 1 as discussed above.
Nassar and Harris in combination do not explicitly teach:
wherein the reference objects include at least one of a charging pad adapted for charging the UAV, a fiducial navigation marker adapted for visual navigation of the UAV, or an autoloader adapted to load a package onto the UAV, and wherein the object detection model comprises at least one of a machine learning (ML) charge pad detector, a ML fiducial marker detector, or a ML autoloader detector.
Kerzner teaches:
wherein the reference objects include at least one of a charging pad adapted for charging the UAV, a fiducial navigation marker adapted for visual navigation of the UAV, or an autoloader adapted to load a package onto the UAV, and wherein the object detection model comprises at least one of a machine learning (ML) charge pad detector, a ML fiducial marker detector, or a ML autoloader detector. (See at least [0049]: “As will be discussed in more detail below with respect to FIGS. 2A-2B, the environment map(s) generated and updated by the system 100 may include an indication of physical objects within the monitored property 120. These physical objects may be represented within the environment map(s) as landmarks. The landmarks may have or be associated with a location that indicates the location of the corresponding physical object within the monitored property 120…”, [0140]: “In some cases, identifying the landmarks in the area includes identifying an object or a surface of an object using one or more algorithms. The object or the surface of the object may be considered a landmark. For example, the drone 102 may use one or more machine learning algorithms to identify objects and/or particular surfaces of objects in the area of the property 120…” & [0146]: “…For example, if a surface of an object is classified as non-planar, the monitoring server 130 may determine that the surface and/or the object corresponding to the surface should not be used as a landmark (e.g., will not be used for navigation).”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it an obvious design choice combine Nassar and Harris’s method with Kerzner’s fiducial marker and ML fiducial marker detector. Doing so would be obvious so the UAV “can more accurately and efficiently navigate through uncontrolled environments, such as residences, commercial properties, etc.” (See [0025] of Kerzner).
Regarding claim 18, Nassar and Harris in combination teach all the limitations of claim 17 as discussed above.
Nassar and Harris in combination do not explicitly teach:
wherein the UAV navigates with reference to the first object only after the detection of the first object by the object detection model registers to a corresponding one of the annotations sourced from the asset map.
Kerzner teaches:
wherein the UAV navigates with reference to the first object only after the detection of the first object by the object detection model registers to a corresponding one of the annotations sourced from the asset map. (See at least Fig. 7, [0189]: “The process 500 includes determining a path to navigate in the area based on the location (508). For example, as shown in FIG. 3, the drone 302 can generate a route through the floor 220 such that one or more landmarks of the floor 220 will be observable and/or unobstructed along the entirety of the route or at certain points along the route. Specifically, the drone 302 (or the monitoring server 130) may generate a path in order to keep the drone 302 within a threshold distance (e.g., 1.0 m, 1.5 m, 1.7 m, etc.) of at least a threshold number of landmarks at any given time (e.g., at least two landmarks, at least three landmarks, etc.)…” & [0289]: “The process 700 includes determining a path to navigate in the area based on the location (708). For example, as shown in FIG. 3, the drone 302 can generate a route through the floor 220 such that one or more landmarks, such as one or more planar landmarks, of the floor 220 will be observable and/or unobstructed along the entirety of the route or at certain points along the route. As shown in FIG. 3, the drone 302 generates a route from its current position 306b, to a second position 310 near the landmark 238, to a third position 312 near the landmark 234, and to a fourth position 314 near the landmark 232.”)
Regarding claim 22, Nassar and Harris in combination teach all the limitations of claim 13 as discussed above.
Nassar and Harris in combination do not explicitly teach:
wherein: the reference objects include at least one of a charging pad adapted for charging the UAV, a fiducial navigation marker adapted for visual navigation of the UAV, or an autoloader adapted to load a package onto the UAV, and the object detection model comprises at least one of a machine learning (ML) charge pad detector, a ML fiducial marker detector, or a ML autoloader detector.
Kerzner teaches:
wherein: the reference objects include at least one of a charging pad adapted for charging the UAV, a fiducial navigation marker adapted for visual navigation of the UAV, or an autoloader adapted to load a package onto the UAV, and the object detection model comprises at least one of a machine learning (ML) charge pad detector, a ML fiducial marker detector, or a ML autoloader detector. (See at least [0049]: “As will be discussed in more detail below with respect to FIGS. 2A-2B, the environment map(s) generated and updated by the system 100 may include an indication of physical objects within the monitored property 120. These physical objects may be represented within the environment map(s) as landmarks. The landmarks may have or be associated with a location that indicates the location of the corresponding physical object within the monitored property 120…”, [0140]: “In some cases, identifying the landmarks in the area includes identifying an object or a surface of an object using one or more algorithms. The object or the surface of the object may be considered a landmark. For example, the drone 102 may use one or more machine learning algorithms to identify objects and/or particular surfaces of objects in the area of the property 120…” & [0146]: “…For example, if a surface of an object is classified as non-planar, the monitoring server 130 may determine that the surface and/or the object corresponding to the surface should not be used as a landmark (e.g., will not be used for navigation).”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it an obvious design choice combine Nassar and Harris’s method with Kerzner’s fiducial marker and ML fiducial marker detector. Doing so would be obvious so the UAV “can more accurately and efficiently navigate through uncontrolled environments, such as residences, commercial properties, etc.” (See [0025] of Kerzner).
Claim(s) 7 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nassar in view of Harris and further in view of Shoeb of US 20230316740 A1, published 10/05/2023, hereinafter “Shoeb”.
Regarding claim 7, Nassar and Harris in combination teach all the limitations of claim 2 as discussed above.
Nassar and Harris in combination do not explicitly teach:
further comprising: transitioning from navigating the UAV based upon the mapping of the correspondences between the reference aerial image and the current aerial image to navigating the UAV based upon the detection of the first object by the object detection model as an above ground level (AGL) altitude of the UAV decreases.
Shoeb teaches:
further comprising: transitioning from navigating the UAV based upon the mapping of the correspondences between the reference aerial image and the current aerial image to navigating the UAV based upon the detection of the first object by the object detection model as an above ground level (AGL) altitude of the UAV decreases. (See at least [0124-0125]: “The operations at block 424 involve determining whether an obstacle was detected. If an obstacle is not detected, the operations repeat from block 405. If an obstacle is detected, the operations at block 430 are performed. The operations at block 430 involve controlling the UAV to avoid the obstacle. In some examples, the UAV is controlled to move in a lateral direction to avoid the obstacle during descent. For example, as illustrated in FIG. 9, the delivery zone is adjusted to avoid the obstacle (e.g., powerline 505F). Some examples of the UAV and/or ground control station determine the amount of lateral movement required to avoid the obstacle based in part on how far the obstacle extends within the delivery zone. Once a suitable delivery zone has been determined, the operations repeat from block 405. In some examples, the position of the UAV is controlled to avoid contact between a tethered payload of the UAV and the obstacle. After changing the UAV position, the tether of the UAV is controlled to deliver a tethered payload to the delivery zone.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nassar and Harris’s method with Shoeb’s technique of transitioning from navigating the UAV based upon the mapping to navigating the UAV based upon the detection from the object detection model as an above ground level (AGL) altitude of the UAV decreases. Doing so would be obvious “to facilitate avoiding obstacles in a delivery zone or other environment of the UAV” (See [0004] of Shoeb).
Regarding claim 19, Nassar and Harris in combination teach all the limitations of claim 17 as discussed above.
Nassar and Harris in combination do not explicitly teach:
wherein the operations further comprise: transitioning from navigating the UAV based upon the mapping of the correspondences between the reference aerial image and the current aerial image to navigating the UAV based upon the detection of the first object by object detection model as an above ground level (AGL) altitude of the UAV decreases.
Shoeb teaches:
wherein the operations further comprise: transitioning from navigating the UAV based upon the mapping of the correspondences between the reference aerial image and the current aerial image to navigating the UAV based upon the detection of the first object by object detection model as an above ground level (AGL) altitude of the UAV decreases. (See at least [0124-0125]: “The operations at block 424 involve determining whether an obstacle was detected. If an obstacle is not detected, the operations repeat from block 405. If an obstacle is detected, the operations at block 430 are performed. The operations at block 430 involve controlling the UAV to avoid the obstacle. In some examples, the UAV is controlled to move in a lateral direction to avoid the obstacle during descent. For example, as illustrated in FIG. 9, the delivery zone is adjusted to avoid the obstacle (e.g., powerline 505F). Some examples of the UAV and/or ground control station determine the amount of lateral movement required to avoid the obstacle based in part on how far the obstacle extends within the delivery zone. Once a suitable delivery zone has been determined, the operations repeat from block 405. In some examples, the position of the UAV is controlled to avoid contact between a tethered payload of the UAV and the obstacle. After changing the UAV position, the tether of the UAV is controlled to deliver a tethered payload to the delivery zone.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nassar and Harris’s method with Shoeb’s technique of transitioning from navigating the UAV based upon the mapping to navigating the UAV based upon the detection from the object detection model as an above ground level (AGL) altitude of the UAV decreases. Doing so would be obvious “to facilitate avoiding obstacles in a delivery zone or other environment of the UAV” (See [0004] of Shoeb).
Claim(s) 9 and 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nassar in view of Harris and further in view of Shue of US 20200234043 A1, filed 10/13/2017, hereinafter “Shue”.
Regarding claim 9, Nassar and Harris in combination teach all the limitations of claim 1 as discussed above.
Nassar additionally teaches:
wherein the asset map is annotated with geolocation labels, (See at least Fig. 6, which shows “The contours found in a segmented image, along with the area, and radius calculated” & Section 3.5, par. 1: “Applying the semantic segmentation step to S(i) and r results in two images containing only buildings and roads as seen in Figure 6. Next, the two images are each split into two layers: building and road objects. Morphological techniques and contouring are applied to each layer to find the boundaries of different objects and extract shape descriptor information to be used in the matching process such as location, area, roundedness, etc. In fact, knowledge of object types provides information on the surroundings of the UAV which could be used recurrently to improve segmentation results and can also be beneficial in some applications such surveillance, parcel delivery, etc. The Semantic Shape Matching SSM pipeline accepts a classified image with a box drawn around each contour…”)
Nassar and Harris in combination do not explicitly teach:
the method further comprising: identifying a gravity aligned pixel in the current aerial image based upon an attitude measurement output from a sensor disposed onboard the UAV, wherein the gravity aligned pixel corresponds to a portion of the current aerial image disposed immediately below the UAV along a gravity vector passing through the UAV; matching the gravity aligned pixel to a corresponding geolocated pixel in the reference aerial image of the asset map based upon the mapping of the correspondences between the reference aerial image and the current aerial image; and localizing the UAV based upon the matching.
However, Nassar teaches a calibration component “for computing an affine geometric transformation between the UAV image and the reference map image in order to map the UAV onto a reference map location, i.e. map S(i) with its field of view extents to corresponding position and covered region in R, taking into consideration the difference in scale and orientation between the two images” and using initial GPS coordinates of the UAV “so a region of interest r can be cropped from the reference map R where r =R(l,wS+b) is the coordinate of the center pixel” (See at least Section 3.2, par. 1 & Section 3.3 par. 1).
Furthermore, Shue teaches using orientation data from the UAV to rotate new images so they are at an angle consistent with historical images, such that, if the historical images were all taken “with a tilt of 0° relative to a plane orthogonal to a gravitational vector, but a gust of wind occurring when UAV 102 captured a new image caused the new image to be taken with a tilt of 5° relative to the plane”, then the new image can be rotated by -5° to align with the historical images (See at least [0041] & [0058]). Shue further teaches determining parameters such as yaw, attitude, tilt, and pitch based on orientation data generated by the UAV when the historical images were captured (See [0041] & [0058]). Therefore, the combination of Nassar and Shue render obvious identifying a gravity aligned pixel corresponding with a gravity vector of the UAV in a current aerial image based on attitude of the UAV, matching the gravity aligned pixel to a corresponding geolocated pixel in a reference aerial image, and localizing the UAV based on the matching, which provides the benefit of “to sustain reasonably accurate translation, rotation and scale components between UAV images and reference map images” (See Section 3.2, par. 2 of Nassar) and “correct[ing] for yaw, attitude, and tilt differences” (See [0041] of Shue).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nassar and Harris’s method with the teachings of Shue. Doing so would be obvious “to sustain reasonably accurate translation, rotation and scale components between UAV images and reference map images” (See Section 3.2, par. 2 of Nassar) and “to correct for yaw, attitude, and tilt differences” (See [0041] of Shue).
Regarding claim 21, Nassar and Harris in combination teach all the limitations of claim 13 as discussed above.
Nassar additionally teaches:
wherein the asset map is annotated with geolocation labels, (See at least Fig. 6, which shows “The contours found in a segmented image, along with the area, and radius calculated” & Section 3.5, par. 1: “Applying the semantic segmentation step to S(i) and r results in two images containing only buildings and roads as seen in Figure 6. Next, the two images are each split into two layers: building and road objects. Morphological techniques and contouring are applied to each layer to find the boundaries of different objects and extract shape descriptor information to be used in the matching process such as location, area, roundedness, etc. In fact, knowledge of object types provides information on the surroundings of the UAV which could be used recurrently to improve segmentation results and can also be beneficial in some applications such surveillance, parcel delivery, etc. The Semantic Shape Matching SSM pipeline accepts a classified image with a box drawn around each contour…”)
Nassar and Harris in combination do not explicitly teach:
wherein the operations further comprise: identifying a gravity aligned pixel in the current aerial image based upon an attitude measurement output from a sensor disposed onboard the UAV, wherein the gravity aligned pixel corresponds to a portion of the current aerial image disposed immediately below the UAV along a gravity vector passing through the UAV; matching the gravity aligned pixel to a corresponding geolocated pixel in the reference aerial image of the asset map based upon the mapping of the correspondences; and localizing the UAV based upon the matching.
However, Nassar teaches a calibration component “for computing an affine geometric transformation between the UAV image and the reference map image in order to map the UAV onto a reference map location, i.e. map S(i) with its field of view extents to corresponding position and covered region in R, taking into consideration the difference in scale and orientation between the two images” and using initial GPS coordinates of the UAV “so a region of interest r can be cropped from the reference map R where r =R(l,wS+b) is the coordinate of the center pixel” (See at least Section 3.2, par. 1 & Section 3.3 par. 1).
Furthermore, Shue teaches using orientation data from the UAV to rotate new images so they are at an angle consistent with historical images, such that, if the historical images were all taken “with a tilt of 0° relative to a plane orthogonal to a gravitational vector, but a gust of wind occurring when UAV 102 captured a new image caused the new image to be taken with a tilt of 5° relative to the plane”, then the new image can be rotated by -5° to align with the historical images (See at least [0041] & [0058]). Shue further teaches determining parameters such as yaw, attitude, tilt, and pitch based on orientation data generated by the UAV when the historical images were captured (See [0041] & [0058]). Therefore, the combination of Nassar and Shue render obvious identifying a gravity aligned pixel corresponding with a gravity vector of the UAV in a current aerial image based on attitude of the UAV, matching the gravity aligned pixel to a corresponding geolocated pixel in a reference aerial image, and localizing the UAV based on the matching, which provides the benefit of “to sustain reasonably accurate translation, rotation and scale components between UAV images and reference map images” (See Section 3.2, par. 2 of Nassar) and “correct[ing] for yaw, attitude, and tilt differences” (See [0041] of Shue).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nassar and Harris’s method with the teachings of Shue. Doing so would be obvious “to sustain reasonably accurate translation, rotation and scale components between UAV images and reference map images” (See Section 3.2, par. 2 of Nassar) and “to correct for yaw, attitude, and tilt differences” (See [0041] of Shue).
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nassar in view of Harris and further in view of Price of US 20210383144 A1, published 12/09/2021, hereinafter “Price”, Kugelmass of US 20160046374 A1, filed 05/29/2015, hereinafter “Kugelmass”, and Wang of CN 116105719 A, published 05/12/2023, hereinafter “Wang”.
Regarding claim 11, Nassar and Harris in combination teach all the limitations of claim 1 as discussed above.
Nassar and Harris in combination do not explicitly teach:
further comprising: storing a library of asset maps including the asset map,
Price teaches:
further comprising: storing a library of asset maps including the asset map, (See at least [0018]: “…the labelled satellite photograph(s) may be pre-loaded to assistive device 102 for use in such processing. For example, assistive device 102 may be pre-configured with global and/or regional satellite data with geospatial labels at any suitable resolution. For example, assistive device 102 may be configured to download a regional satellite image database including a plurality of satellite photographs and pre-defined geospatial coordinates, so that photography of the nearby environment may be correlated against one or more of the plurality of satellite photographs…”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nassar and Harris’s method with Price’s technique of storing a library of asset maps. Doing so would be obvious so that the asset maps are readily available and “so that photography of the nearby environment may be correlated against one or more of the plurality of satellite photographs” (See [0018] of Price).
Nassar, Harris, and Price in combination do not explicitly teach:
wherein each of the asset maps corresponds to a different aerial image of the ground area captured from a different altitude or captured at a different time of day; and
Kugelmass teaches:
wherein each of the asset maps corresponds to a different aerial image of the ground area captured from a different altitude or captured at a different time of day; and (See at least [0101]: “…In this variation, Block S260 can retrieve one or more images stored from a previous mission, intersecting the ground area, and captured in a previous mission within a period of time from current that satisfies the time accuracy requirement, and Block S260 combines these stored images within images captured by the UAV during the current mission to generate the geospatial map. Block S260 can thus aggregate images captured over various missions or campaigns (i.e., in different periods of time) into the geospatial map of the selected ground area.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nassar, Harris, and Price’s method with Kugelmass’s geospatial map corresponding to images captured over different periods of time. Doing so would be obvious so that “a user subsequently interfacing with the geospatial map can identify timeliness and geographic locations of visual data (i.e., pixels) within the geographic map by selecting corresponding pixels or groups of pixels” (See [0102] of Kugelmass).
Nassar, Harris, Price, and Kugelmass in combination do not explicitly teach:
selecting the asset map from the library of asset maps based upon at least one of a current altitude of the UAV or a current time of day when acquiring the current aerial image.
Wang teaches:
selecting the asset map from the library of asset maps based upon at least one of a current altitude of the UAV or a current time of day (See at least [0065]: “The local semantic map in this embodiment can be constructed online in real time based on the vehicle's driving process. For the vehicle's current location, it does not need to use all the previously constructed local semantic map data, but only a part of the local semantic map data close to the current time. Therefore, the optimized vehicle pose of the previous frame can be used as a reference to obtain the local semantic map corresponding to the optimized vehicle pose of the previous frame, that is, the local semantic map around the current vehicle.”)
Although Wang does not explicitly teach a current time of day when acquiring an aerial image, Wang does teach localizing a vehicle by obtaining image data of a current frame and performing semantic segmentation on the image to determine the relative transformation relationship between the semantic segmentation result of the current frame and the local semantic map, of which only the “part of the local semantic map data close to the current time” is used, as discussed above (See at least [0044] & [0065-0066]). Therefore, one having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to perform the teachings of Wang while taking into account the current time of day when acquiring an aerial image, such as the aerial image taught by Price (See at least [0021] of Price), since “it does not need to use all the previously constructed local semantic map data” (See [0065] of Wang).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nassar, Harris, Price, and Kugelmass’s method with Wang’s technique of selecting the asset map from the library of asset maps based on a current time of day when acquiring an image. Doing so would be obvious since “it does not need to use all the previously constructed local semantic map data” (See [0065] of Wang).
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nassar in view of Harris and further in view of Price and Yan of CN 118427383 A, published 08/02/2024, hereinafter “Yan”.
Regarding claim 12, Nassar and Harris in combination teach all the limitations of claim 1 as discussed above.
Nassar and Harris in combination do not explicitly teach:
further comprising: storing a library of asset maps including the asset map, wherein each of the asset maps corresponds to a different aerial image of the ground area, and
Price teaches:
further comprising: storing a library of asset maps including the asset map, wherein each of the asset maps corresponds to a different aerial image of the ground area, and (See at least [0018]: “…the labelled satellite photograph(s) may be pre-loaded to assistive device 102 for use in such processing. For example, assistive device 102 may be pre-configured with global and/or regional satellite data with geospatial labels at any suitable resolution. For example, assistive device 102 may be configured to download a regional satellite image database including a plurality of satellite photographs and pre-defined geospatial coordinates, so that photography of the nearby environment may be correlated against one or more of the plurality of satellite photographs…”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nassar and Harris’s method with Price’s technique of storing a library of asset maps including the asset map, wherein each of the asset maps corresponds to a different aerial image of the ground area. Doing so would be obvious so that the asset maps are readily available and “so that photography of the nearby environment may be correlated against one or more of the plurality of satellite photographs” (See [0018] of Price).
Nassar, Harris, and Price in combination do not explicitly teach:
wherein the asset maps are indexed to reference vector embeddings; and
generating a current vector embedding based on the current aerial image; and
selecting the asset map from the library of asset maps by comparing the current vector embedding to the reference vector embeddings.
Yan teaches:
wherein the asset maps are indexed to reference vector embeddings; and (See at least [0027]: “Phase 2 (Inference): Convert the query image into an embedding vector, retrieve the closest vector from the vector database obtained in the previous phase, complete the image matching, and obtain the location data.”)
generating a current vector embedding based on the current aerial image; and (See at least [0008]: “Further, step S1 includes: S11: generating key semantic features of the image using a visual language model; S12: constructing a natural language scene description from the key semantic features obtained in step S11; S13: converting the language scene description obtained in step S12 into an embedding vector and saving it” & [0011]: “Furthermore, in step S13, a word embedding model is used to convert the semantic description of each reference image into a corresponding embedding vector and save it for querying.”)
selecting the asset map from the library of asset maps by comparing the current vector embedding to the reference vector embeddings. (See at least [0014]: “Further, step S2 includes: S21: For the input query image, use the same method as in step S1 to generate an embedding vector; S22: In the vector corresponding to the reference image, find the vector that is closest to the input embedding vector through cosine similarity, thereby finding the corresponding reference image and completing image matching; S23: Read the corresponding vehicle location information from the label of the reference image to complete the localization of the vehicle”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nassar, Harris, and Price’s method with Yan’s technique of the asset maps being indexed to reference vector embeddings, generating a current vector embedding based on the current aerial image, and selecting the asset map from the library of asset maps by comparing the current vector embedding to the reference vector embeddings. Doing so would be obvious for “improving the positioning accuracy” (See [0042] of Yan).
Allowable Subject Matter
Claims 8 and 20 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
The closest prior art of record is Nassar et al. (NPL Document titled “A Deep CNN-Based Framework For Enhanced Aerial Imagery Registration with Applications to UAV Geolocalization”), hereinafter “Nassar”, and Harris et al. (U.S. Pub. No. 2017/0090271 A1), hereinafter “Harris”.
Regarding claims 8 and 20, Nassar and Harris in combination teach all the limitations of parent claims 1 and 13, as discussed above. Harris additionally teaches determining whether it is safe for a UAV to land during descent of the UAV based on estimating a homography between paired feature points and using a generated depth disparity map to confirm whether objects are within the destination and/or whether there is a planar surface of a sufficient size available at the destination for the UAV to land (See at least [0022-0023], [0060] & [0074]). However, neither Nassar nor Harris teach decreasing an influence of the mapping of the correspondences from the homography estimating tool on the navigation decisions, relative to the detection of the first object by the object detection model, as an above ground level (AGL) altitude of the UAV decreases. Accordingly, none of the references, taken either alone, or in combination teach “decreasing an influence of the mapping of the correspondences from the homography estimating tool on the navigation decisions, relative to the detection of the first object by the object detection model, as an above ground level (AGL) altitude of the UAV decreases” - in combination with the remaining elements and features of the claimed invention. It is for these reasons that the Applicant' s invention defines over the prior art of record.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20230029573 A1 is directed to passing UAV visual data through a CNN to detect targets of interest and define reference points and pixel coordinates for the UAV visual data.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIKKI MARIE M MOLINA whose telephone number is (571)272-5180. The examiner can normally be reached M-F, 9am-6pm PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aniss Chad can be reached at 571-270-3832. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NIKKI MARIE M MOLINA/Examiner, Art Unit 3662
