Detailed Office Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a non-final Office Action on the merits. Claims 1-15 and 19-20 are currently pending and are addressed below.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/25/2026 has been entered.
Response to Arguments
Applicant’s amendments and/or arguments with respect to the rejection of claims 1, 19, and 20 under 35 USC 103 as set forth in the office action of 12/10/2025 have been considered but are respectfully not persuasive. Applicant argues the references of Holtz, Kim, and Shoeb do not teach or suggest the following claim language:
“responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle to navigate based on the uncertainty of the navigational state, wherein applying the protection model comprises:
comparing the uncertainty with uncertainties of labelled images to obtain a calibration factor, wherein the labelled images are associated with the location of interest; and
adjusting the uncertainty according to the calibration factor” The examiner respectfully disagrees.
The combination of Holtz, Kim, and Shoeb teach the above claim limitations:
Shoeb teaches responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle to navigate based on the uncertainty of the navigational state (see at least Shoeb [¶ 199, 252, 197, 78] the UAV may also determine an uncertainty measure based on the percentages of obstacle pixels of each of the multiple images. The uncertainty metric may be a statistical measure of uncertainty, such as the standard deviation of the determined percentages of obstacle pixels. Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure…where aborting the delivery process of the UAV is aborting the descent process of the UAV… the evaluation of determining whether to abort the delivery process may be based on multiple images of the delivery location, perhaps taken during descent of the UAV and perhaps periodically during the descent process or during the navigation of the UAV at the delivery point…The navigation module 214 may provide functionality that allows the UAV 200 to, e.g., move about its environment and reach a desired location. To do so, the navigation module 214 may control the altitude and/or direction of flight)
The disclosure in Shoeb teaches a drone aborting a decent procedure based on the uncertainty metric derived from imaging, and because the abort procedure means halting a decent and a new flight path must be navigated, the disclosure teaches what is analogous to applying a protection model to determine a protection level for the vehicle to navigate based on the uncertainty of the navigational state.
Kim teaches wherein applying the protection model comprises:
comparing the uncertainty with uncertainties of labelled images to obtain a calibration factor, wherein the labelled images are associated with the location of interest (see at least Kim [¶ 36, 38-39, 58-59, 11] the environment is represented with two OGMs. The first OGM is a local model of the environment, which represents three-dimensional virtual space reconstructed from the image data, and the second OGM is a global model of the environment, which represents virtual space of a satellite image that is segmented into different layers… Conditional probability of a match X given an ith orientation local OGM is calculated by counting the number of matching points then using mismatched points to penalize the function …Once the reconstructed scene has been generated, it can be compared to overhead images to identify possible matches, as indicated in block 22 of FIG. 7. For example, the object geometries and the white spaces of the reconstructed scene can be compared to the geometries and white space of the OGM obtained from a satellite imager…the white space in both the reconstructed scene and the overhead images can be categorized to further improve the estimation. For example, that white space can be categorized as a field, a forest, a body of water, a parking lot, etc. to reduce the number of possible matches)
The disclosure in Kim teaches comparing two images and their corresponding Occupancy Grid Maps (OGMs). The OGMs contain an certainty factor (which is analogous to an uncertainty factor) for each pixel, and when the OGMs and their certainties are compared, a penalization factor is calculated to adjust the confidence of matches between the images and their OGMs, which is analogous to a calibration factor. Further, the images are labelled via categorization of white space in the images into fields, forests, bodies of water, etc.
Kim further teaches adjusting the uncertainty according to the calibration factor (see at least Kim [¶ 38] Conditional probability of a match X given an ith orientation local OGM is calculated by counting the number of matching points then using mismatched points to penalize the function).
The disclosure in Kim teaches wherein a penalization factor is calculated to adjust the confidence of matches between the images and their OGMs, this is analogous to a calibration factor adjusting the uncertainty.
Please see detailed rejection below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 7, 9-11, 13-15, 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Holtz et al (US 20220234733 A1) in view of Kim et al (US 20140064624 A1) and Shoeb et al (US 20230312091 A1). Hereafter referred to as Holtz, Kim, and Shoeb respectively.
Regarding Claim 1, Holtz teaches a method comprising:
accessing an image of an environment surrounding an aerial vehicle, the image comprising latent pixel information (see at least Holtz [¶ 32, 142] the UAV 100 also includes various sensors for automated navigation and flight control 112, and one or more image capture devices 114 and 115 for capturing images of the surrounding physical environment while in flight. “Images,” in this context, include both still images and captured video....As images are received, the tracking system 140 may extract semantic information regarding certain objects captured in the images based on an analysis of the pixels in the images. Semantic information regarding a captured object can include information such as an object's category (i.e., class), location, shape, size, scale, pixel segmentation, orientation, inter-class appearance, activity, and pose)
applying a state recognition model to the image to determine whether the image represents a location of interest, the state recognition model configured to:
determine a navigational state of the aerial vehicle using latent pixel information of the image (see at least Holtz [¶ 33, 46, 75-76] the UAV 100 may autonomously (i.e., without direct human control) navigate the physical environment, for example, by processing images captured by any one or more image capture devices…The landing system 150, operating separately or in conjunction with the motion planner 130, may be configured to…identify a landing location (e.g., based on images received from image capture devices 114 and/or 115 and/or data from other sensors 112 (e.g., IMU, GPS, proximity sensors, etc.)) and generate control commands configured to cause the UAV to land at the selected location…the sensors onboard the UAV 100 include a downward facing stereoscopic camera configured to capture images of the ground while the UAV 100 is in flight through a physical environment…At step 704, the received sensor data is processed to determine (i.e., generate) data points that are indicative of height values at multiple points along a surface (e.g., the ground) in the physical environment) Holtz discloses a landing system configured to identify landing location through captured images, which is analogous to determining if the image represents a location of interest. Holtz further discloses determining a navigational state of the aerial vehicle using latent pixel information when the system possess data from the camera to determine height values and position of the UAV
determine an uncertainty of the navigational state using latent pixel information of the image (see at least Holtz [¶ 85, 145] Where stereo vision is applied, the geometric smart landing technique may be configured to account for uncertainty in estimated/measured height values. In stereo vision, as disparity gets larger, the uncertainty of the range (distance from the cameras) of that point also grows. The expected variance of a point scales with the range of that point to the fourth power. To account for this, the landing system 150 may adjust certain height values by an appropriate correction factor. For example, in some embodiments, the landing system 150 may divide the increment to the sum of squared differences in height for a given point by its range to the fourth power).
However, Holtz does not explicitly teach responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle based on the uncertainty; and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle.
Shoeb, in the same field as the endeavor, teaches responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle to navigate based on the uncertainty of the navigational state (see at least Shoeb [¶ 199, 252, 197, 78] the UAV may also determine an uncertainty measure based on the percentages of obstacle pixels of each of the multiple images. The uncertainty metric may be a statistical measure of uncertainty, such as the standard deviation of the determined percentages of obstacle pixels. Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure…where aborting the delivery process of the UAV is aborting the descent process of the UAV… the evaluation of determining whether to abort the delivery process may be based on multiple images of the delivery location, perhaps taken during descent of the UAV and perhaps periodically during the descent process or during the navigation of the UAV at the delivery point…The navigation module 214 may provide functionality that allows the UAV 200 to, e.g., move about its environment and reach a desired location. To do so, the navigation module 214 may control the altitude and/or direction of flight) The disclosure in Shoeb teaches a drone aborting a decent procedure based on the uncertainty metric derived from imaging, and because the abort procedure means halting a decent and a new flight path must be navigated, the disclosure teaches what is analogous to applying a protection model to determine a protection level for the vehicle to navigate based on the uncertainty of the navigational state
and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle (see at least Shoeb [¶ 199] Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure) Shoeb teaches wherein a navigational action that controls the vehicle (the delivery process, landing, etc) is aborted when the uncertainty metric is greater than a threshold, meaning that the navigational action the controls the vehicle (the delivery process, landing, etc) will be performed when the uncertainty metric is below the threshold.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle to navigate based on the uncertainty of the navigational state; and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the safety of the UAV by analyzing risk and uncertainty to know when it is safe to perform a control operation of the vehicle or not, such as landing, as discussed in Shoeb (see at least Shoeb [¶ 41] Another issue that might arise during delivery of the payload is that the area around the delivery point may be too full of obstacles to safely deliver the payload. Therefore, in some examples, the UAV could evaluate one or more delivery points to determine whether to deliver the payload or whether to abort the delivery).
Further, Holtz does not explicitly teach applying a state recognition model to the image to determine whether the image represents a location of interest, the state recognition model configured to:
generate a reconstructed image using the navigational state and the uncertainty, and
compare the image to the reconstructed image to determine whether the image represents the location of interest; and
wherein applying the protection model comprises:
comparing the uncertainty with uncertainties of labelled images to obtain a calibration factor, wherein the labelled images are associated with the location of interest; and
adjusting the uncertainty according to the calibration factor.
Kim, in the same field as the endeavor, teaches applying a state recognition model to the image to determine whether the image represents a location of interest (see at least Kim [Abstract] a system and method for estimating the geographical location at which image data was captured with a camera identifies matching feature points between the captured images, estimates a pose of the camera during the image capture from the feature points, performs geometric reconstruction of a scene in the images using the estimated pose of the camera to obtain a reconstructed scene, and compares the reconstructed scene to overhead images of known geographical origin to identify potential matches)
the state recognition model configured to:
generate a reconstructed image using the navigational state and the uncertainty (see at least Kim [Abstract, ¶ 21, 36, 57] performs geometric reconstruction of a scene in the images using the estimated pose of the camera to obtain a reconstructed scene...the movement can be the result of movement of an object, such as an unmanned autonomous vehicle (UAV), to which the camera is mounted. Regardless, understanding how the camera is moving relative to the scene enables determination of the geometry and position of objects (and voids) within the scene. The determination of such that geometry and position is a structure-from-motion problem. In other words, the geometry of the scene can be reconstructed from an estimated motion of the camera...geometric reconstruction of the scene is performed using the estimated pose and either the known length or known velocity to obtain a reconstructed scene...The OGM comprises a two-dimensional array of cells corresponding to a horizontal grid imposed on the area to be mapped. The grid has n.times.m cells, and each cell has size of s.times.s. Occupancy status with an associated certainty factor are assigned to every cell in the OGM (occupancy grid map) using "0" for empty and "1 " for occupied. Probabilistic representation can alternatively be used in which case the probability of a cell being occupied is represented with values between "0" to "1 ". The OGM representation is simple to construct, even in large-scale environments. Because the intrinsic geometry of a grid corresponds directly to the geometry of the environment, the location estimation of the reconstructed scene can be determined by its pose (position and orientation) in real world)
compare the image to the reconstructed image to determine whether the image represents the location of interest (see at least Kim [¶ 58] Once the reconstructed scene has been generated, it can be compared to overhead images to identify possible matches, as indicated in block 22 of Fig. 7. For example, the object geometries and the white spaces of the reconstructed scene can be compared to the geometries and white space of the OGM obtained from a satellite image. Possible matches can then be identified in the OGM. Such a situation is shown in Fig. 8 in which the highest probability match is identified in the lower left corner, which correlates with the location identified in the map of Fig. 1A at which the image of Fig. 1 B was captured)
wherein applying the protection model comprises:
comparing the uncertainty with uncertainties of labelled images to obtain a calibration factor, wherein the labelled images are associated with the location of interest (see at least Kim [¶ 36, 38-39, 58-59, 11] the environment is represented with two OGMs. The first OGM is a local model of the environment, which represents three-dimensional virtual space reconstructed from the image data, and the second OGM is a global model of the environment, which represents virtual space of a satellite image that is segmented into different layers… Conditional probability of a match X given an ith orientation local OGM is calculated by counting the number of matching points then using mismatched points to penalize the function …Once the reconstructed scene has been generated, it can be compared to overhead images to identify possible matches, as indicated in block 22 of FIG. 7. For example, the object geometries and the white spaces of the reconstructed scene can be compared to the geometries and white space of the OGM obtained from a satellite imager…the white space in both the reconstructed scene and the overhead images can be categorized to further improve the estimation. For example, that white space can be categorized as a field, a forest, a body of water, a parking lot, etc. to reduce the number of possible matches) The disclosure in Kim teaches comparing two images and their corresponding Occupancy Grid Maps (OGMs). The OGMs contain an certainty factor (which is analogous to an uncertainty factor) for each pixel, and when the OGMs and their certainties are compared, a penalization factor is calculated to adjust the confidence of matches between the images and their OGMs, which is analogous to a calibration factor. Further the images are labelled via categorization of white space in the images into fields, forests, bodies of water, etc.
adjusting the uncertainty according to the calibration factor (see at least Kim [¶ 38] Conditional probability of a match X given an ith orientation local OGM is calculated by counting the number of matching points then using mismatched points to penalize the function) The disclosure in Kim teaches wherein a penalization factor is calculated to adjust the confidence of matches between the images and their OGMs, this is analogous to a calibration factor adjusting the uncertainty.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a system for generating a reconstructed image using the navigational state and the uncertainty and comparing the image to the reconstructed image to determine whether the image represents the location of interest and wherein applying the protection model comprises: comparing the uncertainty with uncertainties of labelled images to obtain a calibration factor, wherein the labelled images are associated with the location of interest; and adjusting the uncertainty according to the calibration factor with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of successfully identifying a location at which an image is taken based on the image data, as discussed in Kim.
Regarding Claim 2, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. Holtz further teaches wherein the latent pixel information comprises information representing the location of interest, the navigational state of the vehicle and the uncertainty of the navigational state (see at least Holtz [¶ 97, 145] the landing system 150 may extract semantic information regarding certain objects captured in the images based on an analysis of the pixels in the images. Semantic information regarding a captured object can include information such as an object's category (i.e., class), location, shape, size, scale, pixel segmentation, orientation, inter-class appearance, activity, and pose…Object detections in captured images create rays from a center position of a capturing camera to the object along which the object lies, with some uncertainty…The depth computation can look specifically at pixels that are labeled to be part of an object of interest (e.g., a subject 102). The combination of these rays and planes over time can be fused into an accurate prediction of the 3D position and velocity trajectory of the object over time).
Regarding Claim 3, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. Holtz further teaches wherein determining the image represents the location of interest with the state recognition model further comprises:
identifying additional latent variables representing the location of interest using the latent pixel information (see at least Holtz [¶ 97] the landing system 150 may extract semantic information regarding certain objects captured in the images based on an analysis of the pixels in the images. Semantic information regarding a captured object can include information such as an object's category (i.e., class), location, shape, size, scale, pixel segmentation, orientation, inter-class appearance, activity, and pose)
wherein generating the reconstructed image additionally uses the additional latent variables (see at least Holtz [¶ 145] Object detections in captured images create rays from a center position of a capturing camera to the object along which the object lies, with some uncertainty. The tracking system 140 can compute depth measurements for these detections, creating a plane parallel to a focal plane of a camera along which the object lies, with some uncertainty. These depth measurements can be computed by a stereo vision algorithm operating on pixels corresponding with the object between two or more camera images at different views. The depth computation can look specifically at pixels that are labeled to be part of an object of interest (e.g., a subject 102). The combination of these rays and planes over time can be fused into an accurate prediction of the 3D position and velocity trajectory of the object over time).
Regarding Claim 4, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. Holtz further teaches wherein comparing the image to the reconstructed image comprises calculating a distance metric quantifying differences between the image and the reconstructed image (see at least Holtz [¶ 135] data received from sensors onboard the UAV 100 can be processed to generate a 3D map of the surrounding physical environment while estimating the relative positions and/or orientations of the UAV 100 and/or other objects within the physical environment. This process is sometimes referred to as simultaneous localization and mapping (SLAM). In such embodiments, using computer vision processing, a system in accordance with the present teaching, can search for dense correspondence between images with overlapping FOV (e.g., images taken during sequential time steps and/or stereoscopic images taken at the same time step). The system can then use the dense correspondences to estimate a depth or distance to each pixel represented in each image).
Regarding Claim 5, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. Holtz further teaches wherein accessing the image of the environment comprises: capturing an image of the environment using an camera system of the aerial vehicle (see at least Holtz [¶ 75] Example process 700a begins at step 702 with receiving sensor data from sensors onboard the UAV 100, for example, as described in step 402 of example process 400. In an illustrative example, the sensors onboard the UAV 100 include a downward facing stereoscopic camera configured to capture images of the ground while the UAV 100 is in flight through a physical environment. The downward facing stereoscopic camera may, for example, be part of an array of stereoscopic navigation cameras that comprise the previously described image capture device 114).
Regarding Claim 7, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. Holtz further teaches wherein the location of interest comprises any one of: a runway, a landing pad, a dynamic object surrounding the aerial vehicle, and a static object surrounding the aerial vehicle (see at least Holtz [¶ 67] The step of evaluating a footprint may also include identifying an arrangement of cells that include semantic information indicative of a safe landing location such as a designated landing pad, an area of pavement, or grass).
Regarding Claim 9, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. Holtz further teaches wherein the aerial vehicle comprises any one of: an autonomously controlled aerial vehicle, a semi-autonomously controlled aerial vehicle, a remote-controlled aerial vehicle, a drone, a helicopter, a glider, a rotorcraft, a lighter than air vehicle, a powered lift vehicle, and an airplane (see at least Holtz [¶ 4, 176] FIG. 1 shows an example configuration of an autonomous vehicle in the form of an unmanned aerial vehicle (UAV) within which certain techniques described herein may be applied…A UAV 100, according to the present teachings, may be implemented as any type of UAV. A UAV, sometimes referred to as a drone, is generally defined as any aircraft capable of controlled flight without a human pilot onboard).
Regarding Claim 10, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. However, Holtz in view of Shoeb does not explicitly teach wherein the protection level threshold is implemented by a system designer of the protection model.
However, Shoeb teaches the use of a threshold protection level (see at least Shoeb [¶ 199]).
Therefore, the combination of Holtz, Kim, and Shoeb discloses the claimed invention except for wherein the threshold protection level is implemented by a system designer of the protection model. It would have been obvious to anyone of ordinary skill in the art before the effective filling date of the claimed invention to have included wherein the threshold protection level is implemented by a system designer of the protection model since it has been held to be within the general skill of a worker in the art to select such a method based on its suitability for the intended use as a matter of design choice.
Regarding Claim 11, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. However, Holtz does not explicitly teach wherein each protection level corresponds to a range of uncertainties for the navigational state.
Shoeb, in the same field as the endeavor, teaches wherein each protection level corresponds to a range of uncertainties for the navigational state (see at least Shoeb [¶ 199] The uncertainty metric may be a statistical measure of uncertainty, such as the standard deviation of the determined percentages of obstacle pixels) A standard deviation is a range.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for wherein each protection level corresponds to a range of uncertainties for the navigational state with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of increasing the safety of the operation of the UAV by analyzing a larger range of risk and uncertainty to know when it is safe to perform a control operation of the vehicle or not, such as landing, as discussed in Shoeb (see at least Shoeb [¶ 41] Another issue that might arise during delivery of the payload is that the area around the delivery point may be too full of obstacles to safely deliver the payload. Therefore, in some examples, the UAV could evaluate one or more delivery points to determine whether to deliver the payload or whether to abort the delivery).
Regarding Claim 13, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. However, Holtz does not explicitly teach wherein the protection level is associated with a system of the aerial vehicle, and the system performs the navigation action.
Shoeb, in the same field as the endeavor, teaches wherein the protection level is associated with a system of the aerial vehicle, and the system performs the navigation action (see at least Shoeb [¶ 199] the UAV may determine percentages of obstacle pixels for each of the multiple segmentation images and the UAV may determine an average of these determined percentages. Based on the average percentage being greater than a threshold average percentage, the UAV may abort the delivery process of the UAV. As another example, the UAV may also determine an uncertainty measure based on the percentages of obstacle pixels of each of the multiple images. The uncertainty metric may be a statistical measure of uncertainty, such as the standard deviation of the determined percentages of obstacle pixels. Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for wherein the protection level is associated with a system of the aerial vehicle, and the system performs the navigation action with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of implementing a factor of safety into the navigational system of a UAV, a practice that is well known in the art.
Regarding Claim 14, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. However, Holtz does not explicitly teach wherein the state recognition model is trained using a plurality of training images, the plurality of training images comprising real images, simulated images, or a combination of real and simulated images.
Shoeb, in the same field as the endeavor, teaches wherein the state recognition model is trained using a plurality of training images, the plurality of training images comprising real images, simulated images, or a combination of real and simulated images (see at least Shoeb [¶128-133, 137] A UAV may be configured to use one or more machine learning models to facilitate perception, localization, navigation, and/or other UAV operations…Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and/or predictions about (patterns in the) training data…Then, trained machine learning model 432 can receive input data 430 and one or more inference/prediction requests 440 (perhaps as part of input data 430) and responsively provide as an output one or more inferences and/or predictions 450…Inference(s) and/or prediction(s) 450 can include output images, output intermediate images, numerical values, and/or other output data produced by trained machine learning model(s) 432 operating on input data 430 (and training data 410). In some examples, trained machine learning model(s) 432 can use output inference(s) and/or prediction(s) 450 as input feedback 460. Trained machine learning model(s) 432 can also rely on past inferences as inputs for generating new inferences) Because the trained model outputs images, it must be true that it was trained on images as the input training set.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for wherein the state recognition model is trained using a plurality of training images, the plurality of training images comprising real images, simulated images, or a combination of real and simulated images with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the operation and location of interest identification capabilities of the UAV by training a machine learning algorithm on similar data so it may more easily identify desired patterns as discussed in Shoeb (see at least Shoeb [¶ 128] Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and/or predictions about (patterns in the) training data).
Regarding Claim 15, Holtz in view of Kim and Shoeb teaches all limitations of Claim 14 as set forth above. However, Holtz does not explicitly teach wherein each training image of the plurality of training images comprises latent pixel information representing a similar location of interest and an acceptable navigational state and an acceptable uncertainty of the navigational state.
Shoeb, in the same field as the endeavor, teaches wherein each training image of the plurality of training images comprises latent pixel information representing a similar location of interest and an acceptable navigational state and an acceptable uncertainty of the navigational state (see at least Shoeb [¶ 97, 137, 145-146] the landing system 150 may extract semantic information regarding certain objects captured in the images based on an analysis of the pixels in the images. Semantic information regarding a captured object can include information such as an object's category (i.e., class), location, shape, size, scale, pixel segmentation, orientation, inter-class appearance, activity, and pose…To determine which elements of image 600 are obstacles, UAV 500 may input captured image 600 into a trained machine learning model to obtain segmentation image 650, which may include semantic classifications. As mentioned, these semantic classifications may describe pixels or pixel areas within the image, e.g., as trees, roads, or sidewalks, among other semantic classifications…Object detections in captured images create rays from a center position of a capturing camera to the object along which the object lies, with some uncertainty. The tracking system 140 can compute depth measurements for these detections, creating a plane parallel to a focal plane of a camera along which the object lies, with some uncertainty…Trained machine learning model(s) 432 can also rely on past inferences as inputs for generating new inferences)
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a system for wherein each training image of the plurality of training images comprises latent pixel information representing a similar location of interest and an acceptable navigational state and an acceptable uncertainty of the navigational state with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the location of interest identification capabilities of the UAV by including similar data and images taken by the drone in the past to more accurately identify desirable patterns in future missions as discussed in Shoeb (see at least Shoeb [¶ 128, 137] Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and/or predictions about (patterns in the) training data…Trained machine learning model(s) 432 can also rely on past inferences as inputs for generating new inferences).
Regarding Claim 19, Holtz teaches a method comprising:
at a computer system comprising a processor and a computer-readable medium (see at least Holtz [¶ 39, 178] executable by one or more processors…one or more computer-readable storage media):
accessing an image of an environment surrounding an aerial vehicle, the image comprising latent pixel information (see at least Holtz [¶ 32, 142] the UAV 100 also includes various sensors for automated navigation and flight control 112, and one or more image capture devices 114 and 115 for capturing images of the surrounding physical environment while in flight. “Images,” in this context, include both still images and captured video....As images are received, the tracking system 140 may extract semantic information regarding certain objects captured in the images based on an analysis of the pixels in the images. Semantic information regarding a captured object can include information such as an object's category (i.e., class), location, shape, size, scale, pixel segmentation, orientation, inter-class appearance, activity, and pose)
applying a state recognition model to the image to determine whether the image represents a location of interest, the state recognition model configured to: determine a navigational state of the aerial vehicle using latent pixel information of the image (see at least Holtz [¶ 33, 46, 75-76] the UAV 100 may autonomously (i.e., without direct human control) navigate the physical environment, for example, by processing images captured by any one or more image capture devices…The landing system 150, operating separately or in conjunction with the motion planner 130, may be configured to…identify a landing location (e.g., based on images received from image capture devices 114 and/or 115 and/or data from other sensors 112 (e.g., IMU, GPS, proximity sensors, etc.)) and generate control commands configured to cause the UAV to land at the selected location…the sensors onboard the UAV 100 include a downward facing stereoscopic camera configured to capture images of the ground while the UAV 100 is in flight through a physical environment…At step 704, the received sensor data is processed to determine (i.e., generate) data points that are indicative of height values at multiple points along a surface (e.g., the ground) in the physical environment) Holtz discloses a landing system configured to identify landing location through captured images, which is analogous to determining if the image represents a location of interest. Holtz further discloses determining a navigational state of the aerial vehicle using latent pixel information when the system possess data from the camera to determine height values and position of the UAV
determine an uncertainty of the navigational state using latent pixel information of the image (see at least Holtz [¶ 85, 145] Where stereo vision is applied, the geometric smart landing technique may be configured to account for uncertainty in estimated/measured height values. In stereo vision, as disparity gets larger, the uncertainty of the range (distance from the cameras) of that point also grows. The expected variance of a point scales with the range of that point to the fourth power. To account for this, the landing system 150 may adjust certain height values by an appropriate correction factor. For example, in some embodiments, the landing system 150 may divide the increment to the sum of squared differences in height for a given point by its range to the fourth power).
Further, Holtz does not explicitly teach responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle to navigate based on the uncertainty of the navigational state; and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle.
Shoeb, in the same field as the endeavor, teaches responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle to navigate based on the uncertainty of the navigational state (see at least Shoeb [¶ 199, 252, 197, 78] the UAV may also determine an uncertainty measure based on the percentages of obstacle pixels of each of the multiple images. The uncertainty metric may be a statistical measure of uncertainty, such as the standard deviation of the determined percentages of obstacle pixels. Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure…where aborting the delivery process of the UAV is aborting the descent process of the UAV… the evaluation of determining whether to abort the delivery process may be based on multiple images of the delivery location, perhaps taken during descent of the UAV and perhaps periodically during the descent process or during the navigation of the UAV at the delivery point…The navigation module 214 may provide functionality that allows the UAV 200 to, e.g., move about its environment and reach a desired location. To do so, the navigation module 214 may control the altitude and/or direction of flight) The disclosure in Shoeb teaches a drone aborting a decent procedure based on the uncertainty metric derived from imaging, and because the abort procedure means halting a decent and a new flight path must be navigated, the disclosure teaches what is analogous to applying a protection model to determine a protection level for the vehicle to navigate based on the uncertainty of the navigational state
and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle (see at least Shoeb [¶ 199] Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure) Shoeb teaches wherein a navigational action that controls the vehicle (the delivery process, landing, etc) is aborted when the uncertainty metric is greater than a threshold, meaning that the navigational action the controls the vehicle (the delivery process, landing, etc) will be performed when the uncertainty metric is below the threshold.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle to navigate based on the uncertainty of the navigational state; and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the safety of the UAV by analyzing risk and uncertainty to know when it is safe to perform a control operation of the vehicle or not, such as landing, as discussed in Shoeb (see at least Shoeb [¶ 41] Another issue that might arise during delivery of the payload is that the area around the delivery point may be too full of obstacles to safely deliver the payload. Therefore, in some examples, the UAV could evaluate one or more delivery points to determine whether to deliver the payload or whether to abort the delivery).
However, Holtz does not explicitly teach applying a state recognition model to the image to determine whether the image represents a location of interest, the state recognition model configured to:
generate a reconstructed image using the navigational state and the uncertainty, and
compare the image to the reconstructed image to determine whether the image represents the location of interest
wherein applying the protection model comprises:
comparing the uncertainty with uncertainties of labelled images to obtain a calibration factor, wherein the labelled images are associated with the location of interest; and
adjusting the uncertainty according to the calibration factor.
Kim, in the same field as the endeavor, teaches applying a state recognition model to the image to determine whether the image represents a location of interest (see at least Kim [Abstract] a system and method for estimating the geographical location at which image data was captured with a camera identifies matching feature points between the captured images, estimates a pose of the camera during the image capture from the feature points, performs geometric reconstruction of a scene in the images using the estimated pose of the camera to obtain a reconstructed scene, and compares the reconstructed scene to overhead images of known geographical origin to identify potential matches)
the state recognition model configured to:
generate a reconstructed image using the navigational state and the uncertainty (see at least Kim [Abstract, ¶ 21, 36, 57] performs geometric reconstruction of a scene in the images using the estimated pose of the camera to obtain a reconstructed scene...the movement can be the result of movement of an object, such as an unmanned autonomous vehicle (UAV), to which the camera is mounted. Regardless, understanding how the camera is moving relative to the scene enables determination of the geometry and position of objects (and voids) within the scene. The determination of such that geometry and position is a structure-from-motion problem. In other words, the geometry of the scene can be reconstructed from an estimated motion of the camera...geometric reconstruction of the scene is performed using the estimated pose and either the known length or known velocity to obtain a reconstructed scene...The OGM comprises a two-dimensional array of cells corresponding to a horizontal grid imposed on the area to be mapped. The grid has n.times.m cells, and each cell has size of s.times.s. Occupancy status with an associated certainty factor are assigned to every cell in the OGM (occupancy grid map) using "0" for empty and "1 " for occupied. Probabilistic representation can alternatively be used in which case the probability of a cell being occupied is represented with values between "0" to "1 ". The OGM representation is simple to construct, even in large-scale environments. Because the intrinsic geometry of a grid corresponds directly to the geometry of the environment, the location estimation of the reconstructed scene can be determined by its pose (position and orientation) in real world)
compare the image to the reconstructed image to determine whether the image represents the location of interest (see at least Kim [¶ 58] Once the reconstructed scene has been generated, it can be compared to overhead images to identify possible matches, as indicated in block 22 of Fig. 7. For example, the object geometries and the white spaces of the reconstructed scene can be compared to the geometries and white space of the OGM obtained from a satellite image. Possible matches can then be identified in the OGM. Such a situation is shown in Fig. 8 in which the highest probability match is identified in the lower left corner, which correlates with the location identified in the map of Fig. 1A at which the image of Fig. 1 B was captured).
wherein applying the protection model comprises:
comparing the uncertainty with uncertainties of labelled images to obtain a calibration factor, wherein the labelled images are associated with the location of interest (see at least Kim [¶ 36, 38-39, 58-59, 11] the environment is represented with two OGMs. The first OGM is a local model of the environment, which represents three-dimensional virtual space reconstructed from the image data, and the second OGM is a global model of the environment, which represents virtual space of a satellite image that is segmented into different layers… Conditional probability of a match X given an ith orientation local OGM is calculated by counting the number of matching points then using mismatched points to penalize the function …Once the reconstructed scene has been generated, it can be compared to overhead images to identify possible matches, as indicated in block 22 of FIG. 7. For example, the object geometries and the white spaces of the reconstructed scene can be compared to the geometries and white space of the OGM obtained from a satellite imager…the white space in both the reconstructed scene and the overhead images can be categorized to further improve the estimation. For example, that white space can be categorized as a field, a forest, a body of water, a parking lot, etc. to reduce the number of possible matches) The disclosure in Kim teaches comparing two images and their corresponding Occupancy Grid Maps (OGMs). The OGMs contain an certainty factor (which is analogous to an uncertainty factor) for each pixel, and when the OGMs and their certainties are compared, a penalization factor is calculated to adjust the confidence of matches between the images and their OGMs, which is analogous to a calibration factor. Further the images are labelled via categorization of white space in the images into fields, forests, bodies of water, etc.
adjusting the uncertainty according to the calibration factor (see at least Kim [¶ 38] Conditional probability of a match X given an ith orientation local OGM is calculated by counting the number of matching points then using mismatched points to penalize the function) The disclosure in Kim teaches wherein a penalization factor is calculated to adjust the confidence of matches between the images and their OGMs, this is analogous to a calibration factor adjusting the uncertainty.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a system for generating a reconstructed image using the navigational state and the uncertainty and comparing the image to the reconstructed image to determine whether the image represents the location of interest and wherein applying the protection model comprises: comparing the uncertainty with uncertainties of labelled images to obtain a calibration factor, wherein the labelled images are associated with the location of interest; and adjusting the uncertainty according to the calibration factor with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of successfully identifying a location at which an image is taken based on the image data, as discussed in Kim.
Regarding Claim 20, Holtz teaches a non-transitory, computer-readable medium storing instructions (see at least Holtz [Claim 56] one or more non-transitory computer readable storage media having program instructions stored thereon that)
that, when executed by a processor, cause the processor to:
accessing an image of an environment surrounding an aerial vehicle, the image comprising latent pixel information (see at least Holtz [¶ 32, 142] the UAV 100 also includes various sensors for automated navigation and flight control 112, and one or more image capture devices 114 and 115 for capturing images of the surrounding physical environment while in flight. “Images,” in this context, include both still images and captured video....As images are received, the tracking system 140 may extract semantic information regarding certain objects captured in the images based on an analysis of the pixels in the images. Semantic information regarding a captured object can include information such as an object's category (i.e., class), location, shape, size, scale, pixel segmentation, orientation, inter-class appearance, activity, and pose)
applying a state recognition model to the image to determine whether the image represents a location of interest, the state recognition model configured to: determine a navigational state of the aerial vehicle using latent pixel information of the image (see at least Holtz [¶ 33, 46, 75-76] the UAV 100 may autonomously (i.e., without direct human control) navigate the physical environment, for example, by processing images captured by any one or more image capture devices…The landing system 150, operating separately or in conjunction with the motion planner 130, may be configured to…identify a landing location (e.g., based on images received from image capture devices 114 and/or 115 and/or data from other sensors 112 (e.g., IMU, GPS, proximity sensors, etc.)) and generate control commands configured to cause the UAV to land at the selected location…the sensors onboard the UAV 100 include a downward facing stereoscopic camera configured to capture images of the ground while the UAV 100 is in flight through a physical environment…At step 704, the received sensor data is processed to determine (i.e., generate) data points that are indicative of height values at multiple points along a surface (e.g., the ground) in the physical environment) Holtz discloses a landing system configured to identify landing location through captured images, which is analogous to determining if the image represents a location of interest. Holtz further discloses determining a navigational state of the aerial vehicle using latent pixel information when the system possess data from the camera to determine height values and position of the UAV
determine an uncertainty of the navigational state using latent pixel information of the image (see at least Holtz [¶ 85, 145] Where stereo vision is applied, the geometric smart landing technique may be configured to account for uncertainty in estimated/measured height values. In stereo vision, as disparity gets larger, the uncertainty of the range (distance from the cameras) of that point also grows. The expected variance of a point scales with the range of that point to the fourth power. To account for this, the landing system 150 may adjust certain height values by an appropriate correction factor. For example, in some embodiments, the landing system 150 may divide the increment to the sum of squared differences in height for a given point by its range to the fourth power).
Further, Holtz does not explicitly teach responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle to navigate based on the uncertainty of the navigational state; and perform, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle.
Shoeb, in the same field as the endeavor, teaches responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle to navigate based on the uncertainty of the navigational state (see at least Shoeb [¶ 199, 252, 197, 78] the UAV may also determine an uncertainty measure based on the percentages of obstacle pixels of each of the multiple images. The uncertainty metric may be a statistical measure of uncertainty, such as the standard deviation of the determined percentages of obstacle pixels. Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure…where aborting the delivery process of the UAV is aborting the descent process of the UAV… the evaluation of determining whether to abort the delivery process may be based on multiple images of the delivery location, perhaps taken during descent of the UAV and perhaps periodically during the descent process or during the navigation of the UAV at the delivery point…The navigation module 214 may provide functionality that allows the UAV 200 to, e.g., move about its environment and reach a desired location. To do so, the navigation module 214 may control the altitude and/or direction of flight) The disclosure in Shoeb teaches a drone aborting a decent procedure based on the uncertainty metric derived from imaging, and because the abort procedure means halting a decent and a new flight path must be navigated, the disclosure teaches what is analogous to applying a protection model to determine a protection level for the vehicle to navigate based on the uncertainty of the navigational state
and perform, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state, wherein the performed navigation action controls the aerial vehicle (see at least Shoeb [¶ 199] Aborting the delivery process may be based on the uncertainty metric being greater than a threshold uncertainty measure) Shoeb teaches wherein a navigational action that controls the vehicle (the delivery process, landing, etc) is aborted when the uncertainty metric is greater than a threshold, meaning that the navigational action the controls the vehicle (the delivery process, landing, etc) will be performed when the uncertainty metric is below the threshold.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for responsive to determining the image represents the location of interest, applying a protection model to the image to determine a protection level for the aerial vehicle to navigate based on the uncertainty of the navigational state; and performing, with the aerial vehicle when the protection level is below a protection level threshold, a navigation action based on the determined navigational state: wherein the performed navigation action controls the aerial vehicle with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the safety of the UAV by analyzing risk and uncertainty to know when it is safe to perform a control operation of the vehicle or not, such as landing, as discussed in Shoeb (see at least Shoeb [¶ 41] Another issue that might arise during delivery of the payload is that the area around the delivery point may be too full of obstacles to safely deliver the payload. Therefore, in some examples, the UAV could evaluate one or more delivery points to determine whether to deliver the payload or whether to abort the delivery).
However, Holtz does not explicitly teach instructions that, when executed by a processor, cause the processor to apply a state recognition model to the image to determine whether the image represents a location of interest, the state recognition model configured to:
generate a reconstructed image using the navigational state and the uncertainty, and
compare the image to the reconstructed image to determine whether the image represents the location of interest
wherein applying the protection model comprises:
comparing the uncertainty with uncertainties of labelled images to obtain a calibration factor, wherein the labelled images are associated with the location of interest; and
adjusting the uncertainty according to the calibration factor.
Kim, in the same field as the endeavor, teaches instructions that, when executed by a processor, cause the processor to apply a state recognition model to the image to determine whether the image represents a location of interest (see at least Kim [Abstract] a system and method for estimating the geographical location at which image data was captured with a camera identifies matching feature points between the captured images, estimates a pose of the camera during the image capture from the feature points, performs geometric reconstruction of a scene in the images using the estimated pose of the camera to obtain a reconstructed scene, and compares the reconstructed scene to overhead images of known geographical origin to identify potential matches)
the state recognition model configured to:
generate a reconstructed image using the navigational state and the uncertainty (see at least Kim [Abstract, ¶ 21, 36, 57] performs geometric reconstruction of a scene in the images using the estimated pose of the camera to obtain a reconstructed scene...the movement can be the result of movement of an object, such as an unmanned autonomous vehicle (UAV), to which the camera is mounted. Regardless, understanding how the camera is moving relative to the scene enables determination of the geometry and position of objects (and voids) within the scene. The determination of such that geometry and position is a structure-from-motion problem. In other words, the geometry of the scene can be reconstructed from an estimated motion of the camera...geometric reconstruction of the scene is performed using the estimated pose and either the known length or known velocity to obtain a reconstructed scene...The OGM comprises a two-dimensional array of cells corresponding to a horizontal grid imposed on the area to be mapped. The grid has n.times.m cells, and each cell has size of s.times.s. Occupancy status with an associated certainty factor are assigned to every cell in the OGM (occupancy grid map) using "0" for empty and "1 " for occupied. Probabilistic representation can alternatively be used in which case the probability of a cell being occupied is represented with values between "0" to "1 ". The OGM representation is simple to construct, even in large-scale environments. Because the intrinsic geometry of a grid corresponds directly to the geometry of the environment, the location estimation of the reconstructed scene can be determined by its pose (position and orientation) in real world)
compare the image to the reconstructed image to determine whether the image represents the location of interest (see at least Kim [¶ 58] Once the reconstructed scene has been generated, it can be compared to overhead images to identify possible matches, as indicated in block 22 of Fig. 7. For example, the object geometries and the white spaces of the reconstructed scene can be compared to the geometries and white space of the OGM obtained from a satellite image. Possible matches can then be identified in the OGM. Such a situation is shown in Fig. 8 in which the highest probability match is identified in the lower left corner, which correlates with the location identified in the map of Fig. 1A at which the image of Fig. 1 B was captured)
wherein applying the protection model comprises:
comparing the uncertainty with uncertainties of labelled images to obtain a calibration factor, wherein the labelled images are associated with the location of interest (see at least Kim [¶ 36, 38-39, 58-59, 11] the environment is represented with two OGMs. The first OGM is a local model of the environment, which represents three-dimensional virtual space reconstructed from the image data, and the second OGM is a global model of the environment, which represents virtual space of a satellite image that is segmented into different layers… Conditional probability of a match X given an ith orientation local OGM is calculated by counting the number of matching points then using mismatched points to penalize the function …Once the reconstructed scene has been generated, it can be compared to overhead images to identify possible matches, as indicated in block 22 of FIG. 7. For example, the object geometries and the white spaces of the reconstructed scene can be compared to the geometries and white space of the OGM obtained from a satellite imager…the white space in both the reconstructed scene and the overhead images can be categorized to further improve the estimation. For example, that white space can be categorized as a field, a forest, a body of water, a parking lot, etc. to reduce the number of possible matches) The disclosure in Kim teaches comparing two images and their corresponding Occupancy Grid Maps (OGMs). The OGMs contain an certainty factor (which is analogous to an uncertainty factor) for each pixel, and when the OGMs and their certainties are compared, a penalization factor is calculated to adjust the confidence of matches between the images and their OGMs, which is analogous to a calibration factor. Further the images are labelled via categorization of white space in the images into fields, forests, bodies of water, etc.
adjusting the uncertainty according to the calibration factor (see at least Kim [¶ 38] Conditional probability of a match X given an ith orientation local OGM is calculated by counting the number of matching points then using mismatched points to penalize the function) The disclosure in Kim teaches wherein a penalization factor is calculated to adjust the confidence of matches between the images and their OGMs, this is analogous to a calibration factor adjusting the uncertainty.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a system for generating a reconstructed image using the navigational state and the uncertainty and comparing the image to the reconstructed image to determine whether the image represents the location of interest and wherein applying the protection model comprises: comparing the uncertainty with uncertainties of labelled images to obtain a calibration factor, wherein the labelled images are associated with the location of interest; and adjusting the uncertainty according to the calibration factor with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of successfully identifying a location at which an image is taken based on the image data, as discussed in Kim.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Holtz et al (US 20220234733 A1) in view of Kim et al (US 20140064624 A1), Shoeb et al (US 20230312091 A1) and Wang et al (CN 112840374 A). Hereafter referred to as Holtz, Kim, Shoeb, and Wang respectively.
Regarding Claim 6, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. However, the combination does not explicitly teach wherein determining the protection level using the protection model comprises: calibrating an actual uncertainty based on the uncertainty and a dataset of navigational states and state uncertainties, the dataset previously calculated from a plurality of acceptable images.
Wang, in the same field as the endeavor, teaches wherein determining the protection level using the protection model comprises: calibrating an actual uncertainty based on the uncertainty and a dataset of navigational states and state uncertainties, the dataset previously calculated from a plurality of acceptable images (see at least Wang [English Translation pg.8 para.11] the central position of the target area and the central position of the target calibration area may be different, therefore, the processor 13 can control the second camera 12 to obtain the test image, then the test image and the image of the target area is compared; for example, determining the matching area matched with the test image in the first image; then calculating the deviation of the central position coordinate of the matching area and the central position coordinate of the target area; determining the deflection degree according to the deviation)
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a system for wherein determining the protection level using the protection model comprises: calibrating an actual uncertainty based on the uncertainty and a dataset of navigational states and state uncertainties, the dataset previously calculated from a plurality of acceptable images with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the accuracy of obtained images as discussed in Wang (see at least Wang [English Translation pg.8 para.11] improving the accuracy of shooting angle obtaining).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Holtz et al (US 20220234733 A1) in view of Kim et al (US 20140064624 A1), Shoeb et al (US 20230312091 A1) and Schertler (US 20150310277 A1). Hereafter referred to as Holtz, Kim, Shoeb, and Schertler respectively.
Regarding Claim 8, Holtz in view of Shoeb teaches all limitations of Claim 7 as set forth above. However, the combination does not explicitly teach wherein the runway comprises one or more of: an approach light system, a runway threshold, runway threshold markings, runway end identifier lights, a slope indicator, a touchdown zone, touchdown zone lights, runway markings, and runway lights.
Schertler, in the same field as the endeavor, teaches wherein the runway comprises one or more of: an approach light system, a runway threshold, runway threshold markings, runway end identifier lights, a slope indicator, a touchdown zone, touchdown zone lights, runway markings, and runway lights (see at least Schertler [¶ 1-2] The invention concerns a method and a device for runway localization on the basis of a feature analysis of at least one image of the runway surroundings taken by a landing aircraft….features and templates of the specific visual components of the runways and possibly their surroundings such as boundary features, boundary markings, center lines, thresholds, runway identifiers, runway beacons or target markings are entered as model knowledge in feature and template databases. Through feature matching or template matching of the features or templates contained in the camera image with the model knowledge entered into the database, a resulting general situation of the runway can be determined in the image).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a system for wherein the runway comprises one or more of: an approach light system, a runway threshold, runway threshold markings, runway end identifier lights, a slope indicator, a touchdown zone, touchdown zone lights, runway markings, and runway lights with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of including features on a runway that are conventional in the art.
Claim 12 are rejected under 35 U.S.C. 103 as being unpatentable over Holtz et al (US 20220234733 A1) in view of Kim et al (US 20140064624 A1), Shoeb et al (US 20230312091 A1) and Ahuja et al (US 20200226430 A1). Hereafter referred to as Holtz, Kim, Shoeb, and Ahuja respectively.
Regarding Claim 12, Holtz in view of Kim and Shoeb teaches all limitations of Claim 1 as set forth above. However, the combination does not explicitly teach wherein the determined uncertainty is an aleatoric uncertainty.
Ahuja, in the same field as the endeavor, teaches wherein the determined uncertainty is an aleatoric uncertainty (see at least Ahuja [¶ 25, 78] A vehicle may be…. A drone, an aircraft…Newly observed data identified as uncertain may be efficiently annotated and used to retrain the probabilistic deep neural network. By identifying the data for which the predictions are associated with high input uncertainty (aleatoric) and/or high model uncertainty (epistemic), the ADAS can utilize manual annotation. For the predictions associated with low uncertainty, ADAS can annotate the data automatically from the predictions).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Holtz to contain a method for wherein the determined uncertainty is an aleatoric uncertainty with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the uncertainly calculations of the system by measuring a type of uncertainly that is conventional to use in the art.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSEPH A YANOSKA whose telephone number is (703)756-5891. The examiner can normally be reached M-F 9:00am to 5:00pm (Pacific Time).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Rachid Bendidi can be reached on (571) 272-4896. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSEPH ANDERSON YANOSKA/Examiner, Art Unit 3664
/RACHID BENDIDI/Supervisory Patent Examiner, Art Unit 3664
Prosecution Insights