Detailed Office Action
Status of Claims
This Office Action is in response to the Applicant’s amendments and remarks filed 09/08/2025. The applicant has amended claims 1, 19, and 20. The applicant has cancelled claims 16, 17 and 18. Claims 1-15 and 19-20 are presently pending and are presented for examination.
Response to Amendment
The amendment filed 09/08/2025 has been entered. Claims 1-15 and 19-20 remain pending in the application.
Reply to Applicant’s Remarks
Applicant’s remarks filed 09/08/2025 have been fully considered and are addressed as follows:
Claim Rejections Under 35 U.S.C. 112:
In applicant’s amendments to the claims filed 09/08/2024, the applicant has cancelled claims 16-18. Therefore the 35 U.S.C. 112(b) rejections previously set forth regarding those claims have been withdrawn.
Claim Rejections Under 35 U.S.C. 101:
Applicant’s amendments to the claims filed 09/08/2025 have overcome the 35 U.S.C. 101 rejections previously set forth. Therefore, the rejection has been withdrawn.
Claim Rejections Under 35 U.S.C. 103:
Applicant’s arguments, see Arguments/Remarks, filed 09/08/2025, with regard to the rejections of Claims 1, 19, and 20 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of newly found prior art reference(s).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 7, 9-11, 13-15, 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Holtz et al (US 20220234733 A1) in view of Kim et al (US 20140064624 A1) and Shoeb et al (US 20230312091 A1). Hereafter referred to as Holtz, Kim, and Shoeb respectively.
Regarding Claim 1, Holtz teaches a method comprising:
accessing an image of an environment surrounding an aerial vehicle, the image comprising latent pixel information (see at least Holtz [¶ 32, 142] the UAV 100 also includes various sensors for automated navigation and flight control 112, and one or more image capture devices 114 and 115 for capturing images of the surrounding physical environment while in flight. “Images,” in this context, include both still images and captured video....As images are received, the tracking system 140 may extract semantic information regarding certain objects captured in the images based on an analysis of the pixels in the images. Semantic information regarding a captured object can include information such as an object's category (i.e., class), location, shape, size, scale, pixel segmentation, orientation, inter-class appearance, activity, and pose)
applying a state recognition model to the image to determine whether the image represents a location of interest, the state recognition model configured to:
determine a navigational state of the aerial vehicle using latent pixel information of the image (see at least Holtz [¶ 33, 46, 75-76] the UAV 100 may autonomously (i.e., without direct human control) navigate the physical environment, for example, by processing images captured by any one or more image capture devices…The landing system 150, operating separately or in conjunction with the motion planner 130, may be configured to…identify a landing location (e.g., based on images received from image capture devices 114 and/or 115 and/or data from other sensors 112 (e.g., IMU, GPS, proximity sensors, etc.)) and generate control commands configured to cause the UAV to land at the selected location…the sensors onboard the UAV 100 include a downward facing stereoscopic camera configured to capture images of the ground while the UAV 100 is in flight through a physical environment…At step 704, the received sensor data is processed to determine (i.e., generate) data points that are indicative of height values at multiple points along a surface (e.g., the ground) in the physical environment) Holtz discloses a landing system configured to identify landing location through captured images, which is analogous to determining if the image represents a location of interest. Holtz further discloses determining a navigational state of the aerial vehicle using latent pixel information when the system possess data from the camera to determine height values and position of the UAV
determine an uncertainty of the navigational state using latent pixel information of the image (see at least Holtz [¶ 85, 145] Where stereo vision is applied, the geometric smart landing technique may be configured to account for uncertainty in estimated/measured height values. In stereo vision, as disparity gets larger, the uncertainty of the range (distance from the cameras) of that point also grows. The expected variance of a point scales with the range of that point to the fourth power. To account for this, the landing system 150 may adjust certain height values by an appropriate correction factor. For example, in some embodiments, the landing system 150 may divide the increment to the sum of squared differences in height for a given point by its range to the fourth power).
However, Holtz does not explicitly teach applying a state recognition model to the image to determine whether the image represents a location of interest, the state recognition model configured to:
generate a reconstructed image using the navigational state and the uncertainty, and
compare the image to the reconstructed image to determine whether the image represents the location of interest.
Kim, in the same field as the endeavor, teaches applying a state recognition model to the image to determine whether the image represents a location of interest (see at least Kim [Abstract] a system and method for estimating the geographical location at which image data was captured with a camera identifies matching feature points between the captured images, estimates a pose of the camera during the image capture from the feature points, performs geometric reconstruction of a scene in the images using the estimated pose of the camera to obtain a reconstructed scene, and compares the reconstructed scene to overhead images of known geographical origin to identify potential matches)
the state recognition model configured to:
generate a reconstructed image using the navigational state and the uncertainty (see at least Kim [Abstract, ¶ 21, 36, 57] performs geometric reconstruction of a scene in the images using the estimated pose of the camera to obtain a reconstructed scene...the movement can be the result of movement of an object, such as an unmanned autonomous vehicle (UAV), to which the camera is mounted. Regardless, understanding how the camera is moving relative to the scene enables determination of the geometry and position of objects (and voids) within the scene. The determination of such that geometry and position is a structure-from-motion problem. In other words, the geometry of the scene can be reconstructed from an estimated motion of the camera...geometric reconstruction of the scene is performed using the estimated pose and either the known length or known velocity to obtain a reconstructed scene...The OGM comprises a two-dimensional array of cells corresponding to a horizontal grid imposed on the area to be mapped. The grid has n.times.m cells, and each cell has size of s.times.s. Occupancy status with an associated certainty factor are assigned to every cell in the OGM (occupancy grid map) using "0" for empty and "1 " for occupied. Probabilistic representation can alternatively be used in which case the probability of a cell being occupied is represented with values between "0" to "1 ". The OGM representation is simple to construct, even in large-scale environments. Because the intrinsic geometry of a grid corresponds directly to the geometry of the environment, the location estimation of the reconstructed scene can be determined by its pose (position and orientation) in real world)
compare the image to the reconstructed image to determine whether the image represents the location of interest (see at least Kim [¶ 58] Once the reconstructed scene has been generated, it can be compared to overhead images to identify possible matches, as indicated in block 22 of Fig. 7. For example, the object geometries and the white spaces of the reconstructed scene can be compared to the geometries and white space of the OGM obtained from a satellite image. Possible matches can then be identified in the OGM. Such a situation is shown in Fig. 8 in which the highest probability match is identified in the lower left corner, which correlates with the location identified in the map of Fig. 1A at which the image of Fig. 1 B was captured).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a system for generating a reconstructed image using the navigational state and the uncertainty and comparing the image to the reconstructed image to determine whether the image represents the location of interest with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of successfully identifying a location at which an image is taken based on the image data, as discussed in Kim.
Further, Holtz does not explicitly teach responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle based on the uncertainty; and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle.
Shoeb, in the same field as the endeavor, teaches responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle based on the uncertainty (see at least Shoeb [¶ 199] the UAV may also determine an uncertainty measure based on the percentages of obstacle pixels of each of the multiple images. The uncertainty metric may be a statistical measure of uncertainty, such as the standard deviation of the determined percentages of obstacle pixels. Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure)
and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle (see at least Shoeb [¶ 199] Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure) Shoeb teaches wherein a navigational action that controls the vehicle (the delivery process, landing, etc) is aborted when the uncertainty metric is greater than a threshold, meaning that the navigational action the controls the vehicle (the delivery process, landing, etc) will be performed when the uncertainty metric is below the threshold.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle based on the uncertainty; and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the safety of the UAV by analyzing risk and uncertainty to know when it is safe to perform a control operation of the vehicle or not, such as landing, as discussed in Shoeb (see at least Shoeb [¶ 41] Another issue that might arise during delivery of the payload is that the area around the delivery point may be too full of obstacles to safely deliver the payload. Therefore, in some examples, the UAV could evaluate one or more delivery points to determine whether to deliver the payload or whether to abort the delivery).
Regarding Claim 2, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. Holtz further teaches wherein the latent pixel information comprises information representing the location of interest, the navigational state of the vehicle and the uncertainty of the navigational state (see at least Holtz [¶ 97, 145] the landing system 150 may extract semantic information regarding certain objects captured in the images based on an analysis of the pixels in the images. Semantic information regarding a captured object can include information such as an object's category (i.e., class), location, shape, size, scale, pixel segmentation, orientation, inter-class appearance, activity, and pose…Object detections in captured images create rays from a center position of a capturing camera to the object along which the object lies, with some uncertainty…The depth computation can look specifically at pixels that are labeled to be part of an object of interest (e.g., a subject 102). The combination of these rays and planes over time can be fused into an accurate prediction of the 3D position and velocity trajectory of the object over time).
Regarding Claim 3, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. Holtz further teaches wherein determining the image represents the location of interest with the state recognition model further comprises:
identifying additional latent variables representing the location of interest using the latent pixel information (see at least Holtz [¶ 97] the landing system 150 may extract semantic information regarding certain objects captured in the images based on an analysis of the pixels in the images. Semantic information regarding a captured object can include information such as an object's category (i.e., class), location, shape, size, scale, pixel segmentation, orientation, inter-class appearance, activity, and pose)
wherein generating the reconstructed image additionally uses the additional latent variables (see at least Holtz [¶ 145] Object detections in captured images create rays from a center position of a capturing camera to the object along which the object lies, with some uncertainty. The tracking system 140 can compute depth measurements for these detections, creating a plane parallel to a focal plane of a camera along which the object lies, with some uncertainty. These depth measurements can be computed by a stereo vision algorithm operating on pixels corresponding with the object between two or more camera images at different views. The depth computation can look specifically at pixels that are labeled to be part of an object of interest (e.g., a subject 102). The combination of these rays and planes over time can be fused into an accurate prediction of the 3D position and velocity trajectory of the object over time).
Regarding Claim 4, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. Holtz further teaches wherein comparing the image to the reconstructed image comprises calculating a distance metric quantifying differences between the image and the reconstructed image (see at least Holtz [¶ 135] data received from sensors onboard the UAV 100 can be processed to generate a 3D map of the surrounding physical environment while estimating the relative positions and/or orientations of the UAV 100 and/or other objects within the physical environment. This process is sometimes referred to as simultaneous localization and mapping (SLAM). In such embodiments, using computer vision processing, a system in accordance with the present teaching, can search for dense correspondence between images with overlapping FOV (e.g., images taken during sequential time steps and/or stereoscopic images taken at the same time step). The system can then use the dense correspondences to estimate a depth or distance to each pixel represented in each image).
Regarding Claim 5, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. Holtz further teaches wherein accessing the image of the environment comprises: capturing an image of the environment using an camera system of the aerial vehicle (see at least Holtz [¶ 75] Example process 700a begins at step 702 with receiving sensor data from sensors onboard the UAV 100, for example, as described in step 402 of example process 400. In an illustrative example, the sensors onboard the UAV 100 include a downward facing stereoscopic camera configured to capture images of the ground while the UAV 100 is in flight through a physical environment. The downward facing stereoscopic camera may, for example, be part of an array of stereoscopic navigation cameras that comprise the previously described image capture device 114).
Regarding Claim 7, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. Holtz further teaches wherein the location of interest comprises any one of: a runway, a landing pad, a dynamic object surrounding the aerial vehicle, and a static object surrounding the aerial vehicle (see at least Holtz [¶ 67] The step of evaluating a footprint may also include identifying an arrangement of cells that include semantic information indicative of a safe landing location such as a designated landing pad, an area of pavement, or grass).
Regarding Claim 9, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. Holtz further teaches wherein the aerial vehicle comprises any one of: an autonomously controlled aerial vehicle, a semi-autonomously controlled aerial vehicle, a remote-controlled aerial vehicle, a drone, a helicopter, a glider, a rotorcraft, a lighter than air vehicle, a powered lift vehicle, and an airplane (see at least Holtz [¶ 4, 176] FIG. 1 shows an example configuration of an autonomous vehicle in the form of an unmanned aerial vehicle (UAV) within which certain techniques described herein may be applied…A UAV 100, according to the present teachings, may be implemented as any type of UAV. A UAV, sometimes referred to as a drone, is generally defined as any aircraft capable of controlled flight without a human pilot onboard).
Regarding Claim 10, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. However, Holtz in view of Shoeb does not explicitly teach wherein the threshold protection level is implemented by a system designer of the protection model.
However, Shoeb teaches the use of a threshold protection level (see at least Shoeb [¶ 199]).
Therefore, the combination of Holtz, Kim, and Shoeb discloses the claimed invention except for wherein the threshold protection level is implemented by a system designer of the protection model. It would have been obvious to anyone of ordinary skill in the art before the effective filling date of the claimed invention to have included wherein the threshold protection level is implemented by a system designer of the protection model since it has been held to be within the general skill of a worker in the art to select such a method based on its suitability for the intended use as a matter of design choice.
Regarding Claim 11, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. However, Holtz does not explicitly teach wherein each protection level corresponds to a range of uncertainties for the navigational state.
Shoeb, in the same field as the endeavor, teaches wherein each protection level corresponds to a range of uncertainties for the navigational state (see at least Shoeb [¶ 199] The uncertainty metric may be a statistical measure of uncertainty, such as the standard deviation of the determined percentages of obstacle pixels) A standard deviation is a range.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for wherein each protection level corresponds to a range of uncertainties for the navigational state with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of increasing the safety of the operation of the UAV by analyzing a larger range of risk and uncertainty to know when it is safe to perform a control operation of the vehicle or not, such as landing, as discussed in Shoeb (see at least Shoeb [¶ 41] Another issue that might arise during delivery of the payload is that the area around the delivery point may be too full of obstacles to safely deliver the payload. Therefore, in some examples, the UAV could evaluate one or more delivery points to determine whether to deliver the payload or whether to abort the delivery).
Regarding Claim 13, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. However, Holtz does not explicitly teach wherein the protection level is associated with a system of the aerial vehicle, and the system performs the navigation action.
Shoeb, in the same field as the endeavor, teaches wherein the protection level is associated with a system of the aerial vehicle, and the system performs the navigation action (see at least Shoeb [¶ 199] the UAV may determine percentages of obstacle pixels for each of the multiple segmentation images and the UAV may determine an average of these determined percentages. Based on the average percentage being greater than a threshold average percentage, the UAV may abort the delivery process of the UAV. As another example, the UAV may also determine an uncertainty measure based on the percentages of obstacle pixels of each of the multiple images. The uncertainty metric may be a statistical measure of uncertainty, such as the standard deviation of the determined percentages of obstacle pixels. Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for wherein the protection level is associated with a system of the aerial vehicle, and the system performs the navigation action with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of implementing a factor of safety into the navigational system of a UAV, a practice that is well known in the art.
Regarding Claim 14, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. However, Holtz does not explicitly teach wherein the state recognition model is trained using a plurality of training images, the plurality of training images comprising real images, simulated images, or a combination of real and simulated images.
Shoeb, in the same field as the endeavor, teaches wherein the state recognition model is trained using a plurality of training images, the plurality of training images comprising real images, simulated images, or a combination of real and simulated images (see at least Shoeb [¶128-133, 137] A UAV may be configured to use one or more machine learning models to facilitate perception, localization, navigation, and/or other UAV operations…Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and/or predictions about (patterns in the) training data…Then, trained machine learning model 432 can receive input data 430 and one or more inference/prediction requests 440 (perhaps as part of input data 430) and responsively provide as an output one or more inferences and/or predictions 450…Inference(s) and/or prediction(s) 450 can include output images, output intermediate images, numerical values, and/or other output data produced by trained machine learning model(s) 432 operating on input data 430 (and training data 410). In some examples, trained machine learning model(s) 432 can use output inference(s) and/or prediction(s) 450 as input feedback 460. Trained machine learning model(s) 432 can also rely on past inferences as inputs for generating new inferences) Because the trained model outputs images, it must be true that it was trained on images as the input training set.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for wherein the state recognition model is trained using a plurality of training images, the plurality of training images comprising real images, simulated images, or a combination of real and simulated images with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the operation and location of interest identification capabilities of the UAV by training a machine learning algorithm on similar data so it may more easily identify desired patterns as discussed in Shoeb (see at least Shoeb [¶ 128] Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and/or predictions about (patterns in the) training data).
Regarding Claim 15, Holtz in view of Kim and Shoeb teaches all limitations of Claim 14 as set forth above. However, Holtz does not explicitly teach wherein each training image of the plurality comprises latent pixel information representing a similar location of interest and an acceptable navigational state and an acceptable uncertainty of the navigational state.
Shoeb, in the same field as the endeavor, teaches wherein each training image of the plurality comprises latent pixel information representing a similar location of interest and an acceptable navigational state and an acceptable uncertainty of the navigational state (see at least Shoeb [¶ 97, 137, 145-146] the landing system 150 may extract semantic information regarding certain objects captured in the images based on an analysis of the pixels in the images. Semantic information regarding a captured object can include information such as an object's category (i.e., class), location, shape, size, scale, pixel segmentation, orientation, inter-class appearance, activity, and pose…To determine which elements of image 600 are obstacles, UAV 500 may input captured image 600 into a trained machine learning model to obtain segmentation image 650, which may include semantic classifications. As mentioned, these semantic classifications may describe pixels or pixel areas within the image, e.g., as trees, roads, or sidewalks, among other semantic classifications…Object detections in captured images create rays from a center position of a capturing camera to the object along which the object lies, with some uncertainty. The tracking system 140 can compute depth measurements for these detections, creating a plane parallel to a focal plane of a camera along which the object lies, with some uncertainty…Trained machine learning model(s) 432 can also rely on past inferences as inputs for generating new inferences)
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a system for wherein each training image of the plurality comprises latent pixel information representing a similar location of interest and an acceptable navigational state and an acceptable uncertainty of the navigational state with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the location of interest identification capabilities of the UAV by including similar data and images taken by the drone in the past to more accurately identify desirable patterns in future missions as discussed in Shoeb (see at least Shoeb [¶ 128, 137] Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and/or predictions about (patterns in the) training data…Trained machine learning model(s) 432 can also rely on past inferences as inputs for generating new inferences).
Regarding Claim 19, Holtz teaches a method comprising:
at a computer system comprising a processor and a computer-readable medium (see at least Holtz [¶ 39, 178] executable by one or more processors…one or more computer-readable storage media):
accessing an image of an environment surrounding an aerial vehicle, the image comprising latent pixel information (see at least Holtz [¶ 32, 142] the UAV 100 also includes various sensors for automated navigation and flight control 112, and one or more image capture devices 114 and 115 for capturing images of the surrounding physical environment while in flight. “Images,” in this context, include both still images and captured video....As images are received, the tracking system 140 may extract semantic information regarding certain objects captured in the images based on an analysis of the pixels in the images. Semantic information regarding a captured object can include information such as an object's category (i.e., class), location, shape, size, scale, pixel segmentation, orientation, inter-class appearance, activity, and pose)
applying a state recognition model to the image to determine whether the image represents a location of interest, the state recognition model configured to: determine a navigational state of the aerial vehicle using latent pixel information of the image (see at least Holtz [¶ 33, 46, 75-76] the UAV 100 may autonomously (i.e., without direct human control) navigate the physical environment, for example, by processing images captured by any one or more image capture devices…The landing system 150, operating separately or in conjunction with the motion planner 130, may be configured to…identify a landing location (e.g., based on images received from image capture devices 114 and/or 115 and/or data from other sensors 112 (e.g., IMU, GPS, proximity sensors, etc.)) and generate control commands configured to cause the UAV to land at the selected location…the sensors onboard the UAV 100 include a downward facing stereoscopic camera configured to capture images of the ground while the UAV 100 is in flight through a physical environment…At step 704, the received sensor data is processed to determine (i.e., generate) data points that are indicative of height values at multiple points along a surface (e.g., the ground) in the physical environment) Holtz discloses a landing system configured to identify landing location through captured images, which is analogous to determining if the image represents a location of interest. Holtz further discloses determining a navigational state of the aerial vehicle using latent pixel information when the system possess data from the camera to determine height values and position of the UAV
determine an uncertainty of the navigational state using latent pixel information of the image (see at least Holtz [¶ 85, 145] Where stereo vision is applied, the geometric smart landing technique may be configured to account for uncertainty in estimated/measured height values. In stereo vision, as disparity gets larger, the uncertainty of the range (distance from the cameras) of that point also grows. The expected variance of a point scales with the range of that point to the fourth power. To account for this, the landing system 150 may adjust certain height values by an appropriate correction factor. For example, in some embodiments, the landing system 150 may divide the increment to the sum of squared differences in height for a given point by its range to the fourth power).
However, Holtz does not explicitly teach applying a state recognition model to the image to determine whether the image represents a location of interest, the state recognition model configured to:
generate a reconstructed image using the navigational state and the uncertainty, and
compare the image to the reconstructed image to determine whether the image represents the location of interest.
Kim, in the same field as the endeavor, teaches applying a state recognition model to the image to determine whether the image represents a location of interest (see at least Kim [Abstract] a system and method for estimating the geographical location at which image data was captured with a camera identifies matching feature points between the captured images, estimates a pose of the camera during the image capture from the feature points, performs geometric reconstruction of a scene in the images using the estimated pose of the camera to obtain a reconstructed scene, and compares the reconstructed scene to overhead images of known geographical origin to identify potential matches)
the state recognition model configured to:
generate a reconstructed image using the navigational state and the uncertainty (see at least Kim [Abstract, ¶ 21, 36, 57] performs geometric reconstruction of a scene in the images using the estimated pose of the camera to obtain a reconstructed scene...the movement can be the result of movement of an object, such as an unmanned autonomous vehicle (UAV), to which the camera is mounted. Regardless, understanding how the camera is moving relative to the scene enables determination of the geometry and position of objects (and voids) within the scene. The determination of such that geometry and position is a structure-from-motion problem. In other words, the geometry of the scene can be reconstructed from an estimated motion of the camera...geometric reconstruction of the scene is performed using the estimated pose and either the known length or known velocity to obtain a reconstructed scene...The OGM comprises a two-dimensional array of cells corresponding to a horizontal grid imposed on the area to be mapped. The grid has n.times.m cells, and each cell has size of s.times.s. Occupancy status with an associated certainty factor are assigned to every cell in the OGM (occupancy grid map) using "0" for empty and "1 " for occupied. Probabilistic representation can alternatively be used in which case the probability of a cell being occupied is represented with values between "0" to "1 ". The OGM representation is simple to construct, even in large-scale environments. Because the intrinsic geometry of a grid corresponds directly to the geometry of the environment, the location estimation of the reconstructed scene can be determined by its pose (position and orientation) in real world)
compare the image to the reconstructed image to determine whether the image represents the location of interest (see at least Kim [¶ 58] Once the reconstructed scene has been generated, it can be compared to overhead images to identify possible matches, as indicated in block 22 of Fig. 7. For example, the object geometries and the white spaces of the reconstructed scene can be compared to the geometries and white space of the OGM obtained from a satellite image. Possible matches can then be identified in the OGM. Such a situation is shown in Fig. 8 in which the highest probability match is identified in the lower left corner, which correlates with the location identified in the map of Fig. 1A at which the image of Fig. 1 B was captured).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a system for generating a reconstructed image using the navigational state and the uncertainty and comparing the image to the reconstructed image to determine whether the image represents the location of interest with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of successfully identifying a location at which an image is taken based on the image data, as discussed in Kim.
Further, Holtz does not explicitly teach responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle based on the uncertainty; and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle.
Shoeb, in the same field as the endeavor, teaches responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle based on the uncertainty (see at least Shoeb [¶ 199] the UAV may also determine an uncertainty measure based on the percentages of obstacle pixels of each of the multiple images. The uncertainty metric may be a statistical measure of uncertainty, such as the standard deviation of the determined percentages of obstacle pixels. Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure)
and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle (see at least Shoeb [¶ 199] Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure) Shoeb teaches wherein a navigational action that controls the vehicle (the delivery process, landing, etc) is aborted when the uncertainty metric is greater than a threshold, meaning that the navigational action the controls the vehicle (the delivery process, landing, etc) will be performed when the uncertainty metric is below the threshold.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle based on the uncertainty; and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the safety of the UAV by analyzing risk and uncertainty to know when it is safe to perform a control operation of the vehicle or not, such as landing, as discussed in Shoeb (see at least Shoeb [¶ 41] Another issue that might arise during delivery of the payload is that the area around the delivery point may be too full of obstacles to safely deliver the payload. Therefore, in some examples, the UAV could evaluate one or more delivery points to determine whether to deliver the payload or whether to abort the delivery).
Regarding Claim 20, Holtz teaches a non-transitory, computer-readable medium storing instructions (see at least Holtz [Claim 56] one or more non-transitory computer readable storage media having program instructions stored thereon that)
that, when executed by a processor, cause the processor to:
accessing an image of an environment surrounding an aerial vehicle, the image comprising latent pixel information (see at least Holtz [¶ 32, 142] the UAV 100 also includes various sensors for automated navigation and flight control 112, and one or more image capture devices 114 and 115 for capturing images of the surrounding physical environment while in flight. “Images,” in this context, include both still images and captured video....As images are received, the tracking system 140 may extract semantic information regarding certain objects captured in the images based on an analysis of the pixels in the images. Semantic information regarding a captured object can include information such as an object's category (i.e., class), location, shape, size, scale, pixel segmentation, orientation, inter-class appearance, activity, and pose)
applying a state recognition model to the image to determine whether the image represents a location of interest, the state recognition model configured to: determine a navigational state of the aerial vehicle using latent pixel information of the image (see at least Holtz [¶ 33, 46, 75-76] the UAV 100 may autonomously (i.e., without direct human control) navigate the physical environment, for example, by processing images captured by any one or more image capture devices…The landing system 150, operating separately or in conjunction with the motion planner 130, may be configured to…identify a landing location (e.g., based on images received from image capture devices 114 and/or 115 and/or data from other sensors 112 (e.g., IMU, GPS, proximity sensors, etc.)) and generate control commands configured to cause the UAV to land at the selected location…the sensors onboard the UAV 100 include a downward facing stereoscopic camera configured to capture images of the ground while the UAV 100 is in flight through a physical environment…At step 704, the received sensor data is processed to determine (i.e., generate) data points that are indicative of height values at multiple points along a surface (e.g., the ground) in the physical environment) Holtz discloses a landing system configured to identify landing location through captured images, which is analogous to determining if the image represents a location of interest. Holtz further discloses determining a navigational state of the aerial vehicle using latent pixel information when the system possess data from the camera to determine height values and position of the UAV
determine an uncertainty of the navigational state using latent pixel information of the image (see at least Holtz [¶ 85, 145] Where stereo vision is applied, the geometric smart landing technique may be configured to account for uncertainty in estimated/measured height values. In stereo vision, as disparity gets larger, the uncertainty of the range (distance from the cameras) of that point also grows. The expected variance of a point scales with the range of that point to the fourth power. To account for this, the landing system 150 may adjust certain height values by an appropriate correction factor. For example, in some embodiments, the landing system 150 may divide the increment to the sum of squared differences in height for a given point by its range to the fourth power).
However, Holtz does not explicitly teach instructions that, when executed by a processor, cause the processor to apply a state recognition model to the image to determine whether the image represents a location of interest, the state recognition model configured to:
generate a reconstructed image using the navigational state and the uncertainty, and
compare the image to the reconstructed image to determine whether the image represents the location of interest.
Kim, in the same field as the endeavor, teaches instructions that, when executed by a processor, cause the processor to apply a state recognition model to the image to determine whether the image represents a location of interest (see at least Kim [Abstract] a system and method for estimating the geographical location at which image data was captured with a camera identifies matching feature points between the captured images, estimates a pose of the camera during the image capture from the feature points, performs geometric reconstruction of a scene in the images using the estimated pose of the camera to obtain a reconstructed scene, and compares the reconstructed scene to overhead images of known geographical origin to identify potential matches)
the state recognition model configured to:
generate a reconstructed image using the navigational state and the uncertainty (see at least Kim [Abstract, ¶ 21, 36, 57] performs geometric reconstruction of a scene in the images using the estimated pose of the camera to obtain a reconstructed scene...the movement can be the result of movement of an object, such as an unmanned autonomous vehicle (UAV), to which the camera is mounted. Regardless, understanding how the camera is moving relative to the scene enables determination of the geometry and position of objects (and voids) within the scene. The determination of such that geometry and position is a structure-from-motion problem. In other words, the geometry of the scene can be reconstructed from an estimated motion of the camera...geometric reconstruction of the scene is performed using the estimated pose and either the known length or known velocity to obtain a reconstructed scene...The OGM comprises a two-dimensional array of cells corresponding to a horizontal grid imposed on the area to be mapped. The grid has n.times.m cells, and each cell has size of s.times.s. Occupancy status with an associated certainty factor are assigned to every cell in the OGM (occupancy grid map) using "0" for empty and "1 " for occupied. Probabilistic representation can alternatively be used in which case the probability of a cell being occupied is represented with values between "0" to "1 ". The OGM representation is simple to construct, even in large-scale environments. Because the intrinsic geometry of a grid corresponds directly to the geometry of the environment, the location estimation of the reconstructed scene can be determined by its pose (position and orientation) in real world)
compare the image to the reconstructed image to determine whether the image represents the location of interest (see at least Kim [¶ 58] Once the reconstructed scene has been generated, it can be compared to overhead images to identify possible matches, as indicated in block 22 of Fig. 7. For example, the object geometries and the white spaces of the reconstructed scene can be compared to the geometries and white space of the OGM obtained from a satellite image. Possible matches can then be identified in the OGM. Such a situation is shown in Fig. 8 in which the highest probability match is identified in the lower left corner, which correlates with the location identified in the map of Fig. 1A at which the image of Fig. 1 B was captured).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a system for generating a reconstructed image using the navigational state and the uncertainty and comparing the image to the reconstructed image to determine whether the image represents the location of interest with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of successfully identifying a location at which an image is taken based on the image data, as discussed in Kim.
Further, Holtz does not explicitly teach responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle based on the uncertainty; and perform, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle.
Shoeb, in the same field as the endeavor, teaches responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle based on the uncertainty (see at least Shoeb [¶ 199] the UAV may also determine an uncertainty measure based on the percentages of obstacle pixels of each of the multiple images. The uncertainty metric may be a statistical measure of uncertainty, such as the standard deviation of the determined percentages of obstacle pixels. Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure)
and perform, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state, wherein the performed navigation action controls the aerial vehicle (see at least Shoeb [¶ 199] Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure) Shoeb teaches wherein a navigational action that controls the vehicle (the delivery process, landing, etc) is aborted when the uncertainty metric is greater than a threshold, meaning that the navigational action the controls the vehicle (the delivery process, landing, etc) will be performed when the uncertainty metric is below the threshold.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle based on the uncertainty; and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the safety of the UAV by analyzing risk and uncertainty to know when it is safe to perform a control operation of the vehicle or not, such as landing, as discussed in Shoeb (see at least Shoeb [¶ 41] Another issue that might arise during delivery of the payload is that the area around the delivery point may be too full of obstacles to safely deliver the payload. Therefore, in some examples, the UAV could evaluate one or more delivery points to determine whether to deliver the payload or whether to abort the delivery).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Holtz et al (US 20220234733 A1) in view of Kim et al (US 20140064624 A1), Shoeb et al (US 20230312091 A1) and Wang et al (CN 112840374 A). Hereafter referred to as Holtz, Kim, Shoeb, and Wang respectively.
Regarding Claim 6, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. However, the combination does not explicitly teach wherein determining the protection level using the protection model comprises: calibrating an actual uncertainty based on the uncertainty and a dataset of navigational states and state uncertainties, the dataset previously calculated from a plurality of acceptable images.
Wang, in the same field as the endeavor, teaches wherein determining the protection level using the protection model comprises: calibrating an actual uncertainty based on the uncertainty and a dataset of navigational states and state uncertainties, the dataset previously calculated from a plurality of acceptable images (see at least Wang [English Translation pg.8 para.11] the central position of the target area and the central position of the target calibration area may be different, therefore, the processor 13 can control the second camera 12 to obtain the test image, then the test image and the image of the target area is compared; for example, determining the matching area matched with the test image in the first image; then calculating the deviation of the central position coordinate of the matching area and the central position coordinate of the target area; determining the deflection degree according to the deviation)
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a system for wherein determining the protection level using the protection model comprises: calibrating an actual uncertainty based on the uncertainty and a dataset of navigational states and state uncertainties, the dataset previously calculated from a plurality of acceptable images with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the accuracy of obtained images as discussed in Wang (see at least Wang [English Translation pg.8 para.11] improving the accuracy of shooting angle obtaining).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Holtz et al (US 20220234733 A1) in view of Kim et al (US 20140064624 A1), Shoeb et al (US 20230312091 A1) and Schertler (US 20150310277 A1). Hereafter referred to as Holtz, Kim, Shoeb, and Schertler respectively.
Regarding Claim 8, Holtz in view of Shoeb teaches all limitations of Claim 7 as set forth above. However, the combination does not explicitly teach wherein the runway comprises one or more of: an approach light system, a runway threshold, runway threshold markings, runway end identifier lights, a slope indicator, a touchdown zone, touchdown zone lights, runway markings, and runway lights.
Schertler, in the same field as the endeavor, teaches wherein the runway comprises one or more of: an approach light system, a runway threshold, runway threshold markings, runway end identifier lights, a slope indicator, a touchdown zone, touchdown zone lights, runway markings, and runway lights (see at least Schertler [¶ 1-2] The invention concerns a method and a device for runway localization on the basis of a feature analysis of at least one image of the runway surroundings taken by a landing aircraft….features and templates of the specific visual components of the runways and possibly their surroundings such as boundary features, boundary markings, center lines, thresholds, runway identifiers, runway beacons or target markings are entered as model knowledge in feature and template databases. Through feature matching or template matching of the features or templates contained in the camera image with the model knowledge entered into the database, a resulting general situation of the runway can be determined in the image).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a system for wherein the runway comprises one or more of: an approach light system, a runway threshold, runway threshold markings, runway end identifier lights, a slope indicator, a touchdown zone, touchdown zone lights, runway markings, and runway lights with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of including features on a runway that are conventional in the art.
Claim 12 are rejected under 35 U.S.C. 103 as being unpatentable over Holtz et al (US 20220234733 A1) in view of Kim et al (US 20140064624 A1), Shoeb et al (US 20230312091 A1) and Ahuja et al (US 20200226430 A1). Hereafter referred to as Holtz, Kim, Shoeb, and Ahuja respectively.
Regarding Claim 12, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. However, the combination does not explicitly teach wherein the determined uncertainty is an aleatoric uncertainty.
Ahuja, in the same field as the endeavor, teaches wherein the determined uncertainty is an aleatoric uncertainty (see at least Ahuja [¶ 25, 78] A vehicle may be…. A drone, an aircraft…Newly observed data identified as uncertain may be efficiently annotated and used to retrain the probabilistic deep neural network. By identifying the data for which the predictions are associated with high input uncertainty (aleatoric) and/or high model uncertainty (epistemic), the ADAS can utilize manual annotation. For the predictions associated with low uncertainty, ADAS can annotate the data automatically from the predictions).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for wherein the determined uncertainty is an aleatoric uncertainty with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the uncertainly calculations of the system by measuring a type of uncertainly that is conventional to use in the art.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSEPH A YANOSKA whose telephone number is (703)756-5891. The examiner can normally be reached M-F 9:00am to 5:00pm (Pacific Time).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Rachid Bendidi can be reached on (571) 272-4896. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSEPH ANDERSON YANOSKA/Examiner, Art Unit 3664
/RACHID BENDIDI/Supervisory Patent Examiner, Art Unit 3664
Prosecution Insights