Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This Office Action is sent in response to Applicant’s Communication received on July 21, 2025 and April 3, 2026 for application 19/274,833. This Office hereby acknowledges receipt of the following and placed of record in file: Oath/Declaration, Specification, Drawings, Abstract and Claims.
3. Claims 21-40 are presented for examination.
Information Disclosure Statement
4. The information disclosure statement (IDS) submitted on October 6, 2025 and July 21, 2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Remarks
5. Examiner notes that this application also discloses only subject matter disclosed in first prior application no 15/827,945, filed on November 30, 2017, second prior application no 17/712,613 filed on April 4, 2022, and third prior application no 18/400,113 filed on December 29,2023 and names the inventor or at least one joint inventor named in the prior applications. Accordingly, this application may constitute a continuation or division. Said first prior application was granted a patent, U.S. Patent No. 11,295,458 B2, said second prior application was granted a patent, U.S. Patent No. 11,861,892 B2 and said third prior application was granted a patent, U.S. Patent No. 12,367,670 B2. Examiner revised claims 1-31 of U.S. Patent No. 11,295,458 B2, claims 1-20 of U.S. Patent No. 11,861,892 B2 and claims 1-20 of U.S. Patent No. 12,367,670 B2 but couldn’t find any grounds of rejection of Double Patenting type.
Claim Rejections - 35 USC § 103
6. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
8. Claim 21-40 are rejected under 35 U.S.C. 103 as being unpatentable over Kokkeby et al.(US 2009/0157233 A1)(hereinafter Kokkeby) in view of Zeng(US 2010/0191391 A1)(hereinafter Zeng) in further view of Fragoso et al.(US 2017/0193830 A1)(hereinafter Fragoso).
Regarding claim 21, Kokkeby discloses a computer-implemented method for tracking an object by an unmanned aerial vehicle (UAV) [See Kokkeby: Figs. 1-8 and par. 0005-0009, 0020 regarding system and methods for autonomous tracking and surveillance are configured to enable an unmanned air vehicle (UAV) continuously to observe stationary and track moving target. The targets may be ground-based, airborne and/or seaborne. The targets may be fixed structures, such as buildings, and may even be subsurface.], the method comprising:
receiving, from one or more image capture devices onboard the UAV including a stereoscopic image capture device, image data of a physical environment [See Kokkeby: at least Figs. 1, 2, 7, par. 0026-0028, 0058 regarding a UAV (not shown) includes at least one video camera, which may be a digital camera. For simplicity the singular form of camera will be used throughout, although those of ordinary skill in the art will appreciate that the UAV may include more than one camera. The UAV further includes a plurality of sensors. A video signal generated by the camera and signals generated by the sensors are transmitted to hardware components that use the signals to visualize and track the target. The ATR module 22 includes instructions to analyze the video signal 26.].
Kokkeby does not explicitly disclose detecting an object instance in the image data and generating an instance segmentation for the object instance.
However, detecting an instance of an object in the image data and generating instance segmentation for the object was well known in the art at the time of the invention was filed as evident from the teaching of Zeng [See Zeng: at least Figs. 3-8 and par. 0007, 0028, 0035, 0036-0044, 0049, 0055, 0073 regarding monitoring a first tracked object and a second tracked object and determining dissimilarity measure between them to determine whether they are a single tracked object. That is, the method is configured to distinguish one or more instances of a tracked object. The method also is configured to classify each identified object. The motion analysis and salient region detection module 112 identifies image areas for further analysis that exhibit a threshold confidence for containing objects or parts of objects. Image areas corresponding to previously identified objects are determined using a priori information including object information of previous time-step such as the object track and position. The salient image areas can be identified using extraction algorithms including a scale-invariant feature transform (SIFT), a speeded up robust feature (SURF) algorithm, and/or a maximally stable extreme region (MSER) algorithm. A histogram of oriented gradients (HOG) and eigen-image coefficients can be used in one embodiment. Other regions of interest in the modified image may be identified using edge, corner, and salient blob detection methods…].
Therefore it would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Kokkeby with Zeng teachings by including “detecting an object instance in the image data and generating an instance segmentation for the object instance” for providing the benefit of a multiobject tracking in an image processing system for vehicles [See Zeng: abstract, par. 0007, 0035, 0049, 0055, 0073].
Further on, when combined, Zeng also teaches while visual contact with the object instance is maintained, updating a track state of the object instance including a last observed position and a last observed trajectory of the object instance [See Zeng: at least Figs. 1-8, par. 49-73 regarding The classification module 116 analyzes each image area and assigns a label (L) to each identified object and associated image areas based upon a pre-trained classifier. The classification module 116 takes the extracted features and prior identified objects (a priori) and associated data including position, image area, and classification label are updated for the current time-step in a prediction process module 117 as input and feeds to a classifier to determine whether an image area encloses an object… A data association module 118 analyzes the current object list including P, I, and L with the prior objects (a priori) and associated data including position, image area features, and classification label are updated for the current time-step in the prediction process module 117 using an object dynamic model. The data association module 118 matches the current identified objects with prior identified objects and outputs a list of matched object-measurement pairs. The prior identified objects are updated to the current time-step using the prediction process module 117 as described herein below. By comparing an object (P, I, and L) in a second tracking cycle analyzing a second input image to another object (P, I, and L) in a first tracking cycle analyzing a first input image, a determination can be made whether the two objects likely refer to the same tracked object in the proximity of the vehicle.. he data association module 118 links the current measurements with the predicted objects, or determines the source of a measurement (i.e., position, image patch, and label) is from a specific object. The object tracker module 119 generates updated positions for the objects for storage in the object track database 113.].
Kokkeby and Zeng do not explicitly disclose detecting a loss of visual contact with the object instance at a location of an occluding object in the physical environment.
However, Fragoso teaches detecting a loss of visual contact with the object instance at a location of an occluding object in the physical environment [See Fragoso: at least Fig. 5 and par. 90-99 regarding the trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Therefore it would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Kokkeby and Zeng with Fragoso teachings by including “detecting a loss of visual contact with the object instance at a location of an occluding object in the physical environment” because this combination has the benefit of providing a target tracking method for controlling an unmanned aerial vehicle (UAV) to avoid obstacle collision [See Fragoso: at least Fig. 5 and par. 90].
Further on, when combined, Fragoso teaches generating an occupancy map of the physical environment from stereo-derived depth information and segmentation information[See Fragoso: Figs. 2 and 5 and par. 90-99, 114 regarding At step 504, the range data is combined into an egospace representation that includes one or more pixels in egospace. Egospace defines/consists of a coordinate system. For each of the pixels of the egospace representation, a value may be assigned that uniquely encodes a distance to one or more of the visible object from a defined focal point. The defined focal point may be located at an arbitrary origin… At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…];
determining, based on the occupancy map, the instance segmentation, and one or more prior poses and fields of view of the image capture devices, that the object instance remains within an occluded volume associated with the occluding object by excluding candidate regions from which the object instance would have been visible in previously captured image data[See Fragoso: Figs. 2 and 5 and par. 40, 90-99, 103, 110, 114 regarding At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…]; and
determining one or more feasible escape paths for the object instance that originate from the last observed trajectory and are constrained to remain within the occluded volume and avoid occupied regions represented in the occupancy map [See Fragoso: at least Figs. 2-5 and par. 41, 81-85, 89-99 regarding For a sampling based method, the correct choice of a distribution in egospace immediately satisfies completeness and follows an identical procedure as in world coordinates—albeit with the more efficient collision-checking scheme by comparison. For a simulated obstacle data set, embodiments exhibit an egospace implementation of the Lazy PRM method [Bohlin 2000] generated using this approach. FIGS. 3A and 3B illustrate a lazy PRM motion planner implemented entirely in depth-image coordinates (FIG. 3A) for an MAV with negligible dynamics in a simulated forest environment and a world space representation (FIG. 3B). In the depth image, pixels with cooler colors/tones (e.g., trees 302 in the foreground) are closer than pixels with warmer colors/tones (e.g., trees 304 at the sensor horizon). Points (in embodiments of the invention) are sampled, selected and connected entirely in the egospace coordinates, for which the projected pixel coordinates of the motion plan are shown (points 306). White segments 306W of path 306 are obscured by obstacles (e.g., trees 304), but pass safely behind the obstacles 304 by a predetermined safety margin. Black segments 306B path are not obscured by the obstacles 304. An equivalent point cloud and world-space representation is shown in FIG. 3B. in which the path/trajectory 306 avoids the obstacles 304…At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…].
Zeng and Fragoso teach maintaining, during the loss of visual contact, a probabilistic estimate of the track state by propagating particles from the last observed position along the one or more feasible escape paths subject to constraints imposed by the occupancy map and the exclusion of the candidate regions[See Zeng: at least par. 69-71 regarding each object track includes a mixture of Gaussian distribution. This distribution can be randomly sampled to obtain a set of particles representing a probability distribution of the object. The collision threat assessment module 120 determines the likelihood for collision based upon a percentage of a number of threats per total particles associated with an object. The likelihood for collision is outputted to the collision counter-measures module 150. See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114 regarding At step 504, the range data is combined into an egospace representation that includes one or more pixels in egospace. Egospace defines/consists of a coordinate system. For each of the pixels of the egospace representation, a value may be assigned that uniquely encodes a distance to one or more of the visible object from a defined focal point. The defined focal point may be located at an arbitrary origin… At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…]; and
reacquiring the object instance by restricting evaluation of subsequent detections to an emergence region corresponding to a projection of the probabilistic estimate[See Zeng: at least Figs. 1-8 and par. 49, 52-73 regarding For example, in FIG. 8, the boxes A and B are identified as vehicles while the unlabelled box is identified as road-side object. The prediction process module 117 utilizes an object's historical information (i.e., position, image patch, and label of previous cycle) and predicts the current values. The data association module 118 links the current measurements with the predicted objects, or determines the source of a measurement (i.e., position, image patch, and label) is from a specific object. The object tracker module 119 generates updated positions for the objects for storage in the object track database 113…See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114 regarding Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Regarding claim 22, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 21, and are analyzed as previously discussed with respect to that claim. Further on, Fragoso teaches or suggests wherein the instance segmentation comprises an amodal segmentation that identifies an occluded portion of the object instance [See Fragoso: at least par. 112-115 regarding The Gaussian models are updated efficiently with the current depth map observation. This approach unites and extends the efficiency of sparse depth model updates in the SLAM literature with dense representation of multiview stereo. The use of Gaussian mixtures enables modeling partially occluded pixels due to robot's egomotion or independently moving objects in the scene… Use of Gaussian Mixture Models (GMM) is a common technique to perform background/foreground segmentation for detecting moving objects in surveillance video..(Here, GMMs are used to perform segmentation and identify an occluded portion of the object instance)].
Regarding claim 23, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 21, and are analyzed as previously discussed with respect to that claim. Further on, Fragoso teaches or suggests wherein excluding candidate regions comprises performing a line-of-sight visibility determination using the occupancy map and camera pose information[See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114, 122 regarding At step 504, the range data is combined into an egospace representation that includes one or more pixels in egospace. Egospace defines/consists of a coordinate system. For each of the pixels of the egospace representation, a value may be assigned that uniquely encodes a distance to one or more of the visible object from a defined focal point. The defined focal point may be located at an arbitrary origin… At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Regarding claim 24, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 21, and are analyzed as previously discussed with respect to that claim. Further on, Fragoso teaches or suggests wherein the occupancy map is generated by carving free space and occupied space using stereo-derived depth information and segmentation outputs[See Fragoso: Figs. 2 and 5 and par. 90-99, 114 regarding At step 504, the range data is combined into an egospace representation that includes one or more pixels in egospace. Egospace defines/consists of a coordinate system. For each of the pixels of the egospace representation, a value may be assigned that uniquely encodes a distance to one or more of the visible object from a defined focal point. The defined focal point may be located at an arbitrary origin… At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Regarding claim 25, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 21, and are analyzed as previously discussed with respect to that claim. Further on, Zeng and Fragoso teach or suggest wherein propagating the particles includes assigning particle weights based on consistency with the occupancy map, motion constraints, and the exclusion of the candidate regions [See Zeng: at least par. 69-71 regarding each object track includes a mixture of Gaussian distribution. This distribution can be randomly sampled to obtain a set of particles representing a probability distribution of the object. The collision threat assessment module 120 determines the likelihood for collision based upon a percentage of a number of threats per total particles associated with an object. The likelihood for collision is outputted to the collision counter-measures module 150… See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114 regarding At step 504, the range data is combined into an egospace representation that includes one or more pixels in egospace. Egospace defines/consists of a coordinate system. For each of the pixels of the egospace representation, a value may be assigned that uniquely encodes a distance to one or more of the visible object from a defined focal point. The defined focal point may be located at an arbitrary origin… At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Regarding claim 26, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 21, and are analyzed as previously discussed with respect to that claim. Further on, Zeng and Fragoso teach wherein the emergence region comprises an image-space projection of the probabilistic estimate onto a field of view of at least one image capture device[See Zeng: at least Figs. 1-8 and par. 32, 3649, 52-73 regarding The fusion module 110 integrates input from various sensing devices including cameras 104 and range sensors 102 and generates a fused track of an object in proximity to the vehicle 10… For example, in FIG. 8, the boxes A and B are identified as vehicles while the unlabelled box is identified as road-side object. The prediction process module 117 utilizes an object's historical information (i.e., position, image patch, and label of previous cycle) and predicts the current values. The data association module 118 links the current measurements with the predicted objects, or determines the source of a measurement (i.e., position, image patch, and label) is from a specific object. The object tracker module 119 generates updated positions for the objects for storage in the object track database 113…See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114 regarding Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Regarding claim 27, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 21, and are analyzed as previously discussed with respect to that claim. Further on, Zeng teaches further comprising, after reacquiring the object instance, verifying an identity of the object instance using an appearance representation of the object instance [See Zeng: at least Fig. 3 and par. 28, 44, 48, 52-55 regarding data association and clustering module to match current identified objects with prior identified objects and outputs a list of matched object-measurement pairs. Pixel classifications of pixels within segmented image areas or patches are analyzed. (It is understood that the grouping or clustering of pixels is based on how they fit an appearance of pixels already detected and pertaining to a class or by their similarity)].
Regarding claim 28, Kokkeby discloses an unmanned aerial vehicle[See Kokkeby: Fig. 8 regarding UAV 74] comprising:
one or more image capture devices[See Kokkeby: at least Figs. 1, 7, par. 0026-0028, 0058 regarding a UAV (not shown) includes at least one video camera, which may be a digital camera. For simplicity the singular form of camera will be used throughout, although those of ordinary skill in the art will appreciate that the UAV may include more than one camera.];
one or more processors[See Kokkeby: Fig. 1 and par. 0027 regarding a component may be, but is not limited to being, a process running on a processor, a processor, a hardware component, an object, an executable, a thread of execution, a program, and/or a computing system.]; and
a memory storing instructions that, when executed by the one or more processors[See Kokkeby: Fig. 1 and par. 0027 regarding these components can execute from various computer readable media having various data structures stored thereon. Computer executable components (or code) can be stored, for example, on computer readable media including, but not limited to, an ASIC (application specific integrated circuit), CD (compact disc), DVD (digital video disk), ROM (read only memory), floppy disk, hard disk, EEPROM (electrically erasable programmable read only memory) and memory stick in accordance with the claimed subject matter], cause the one or more processors to:
detect and track an object using image data captured by the one or more image capture devices[See Kokkeby: at least Fig. 1 and par. 28-30 regarding With reference to FIG. 1, one embodiment of the present system 20 includes an automatic target recognition (ATR) module 22 and a multi-sensor integration (MSI) module 24. The ATR module 22 receives a video signal 26 from the UAV (not shown). The ATR module 22 includes instructions to analyze the video signal 26 and generates an output 28 that it sends to the MSI module 24. In addition to the ATR output 28, the MSI module 24 also receives a UAV state signal 30 and a target state signal 32. The signals 30, 32 are generated by the sensors described above, and may also be generated by other sources observing the UAV and/or the target, such as ground-based observers, radar, satellites, etc…].
Kokkeby does not explicitly disclose detect and track an object instance using image data captured by the one or more image capture devices.
However, detecting and tracking an instance of an object in the image data was well known in the art at the time of the invention was filed as evident from the teaching of Zeng [See Zeng: at least Figs. 3-8 and par. 0007, 0028, 0035, 0036-0044, 0049, 0055, 0073 regarding monitoring a first tracked object and a second tracked object and determining dissimilarity measure between them to determine whether they are a single tracked object. That is, the method is configured to distinguish one or more instances of a tracked object. The method also is configured to classify each identified object. The motion analysis and salient region detection module 112 identifies image areas for further analysis that exhibit a threshold confidence for containing objects or parts of objects. Image areas corresponding to previously identified objects are determined using a priori information including object information of previous time-step such as the object track and position. The salient image areas can be identified using extraction algorithms including a scale-invariant feature transform (SIFT), a speeded up robust feature (SURF) algorithm, and/or a maximally stable extreme region (MSER) algorithm. A histogram of oriented gradients (HOG) and eigen-image coefficients can be used in one embodiment. Other regions of interest in the modified image may be identified using edge, corner, and salient blob detection methods…].
Therefore it would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Kokkeby with Zeng teachings by including “detect and track an object instance using image data captured by the one or more image capture devices” for providing the benefit of a multiobject tracking in an image processing system for vehicles [See Zeng: abstract, par. 0007, 0035, 0049, 0055, 0073].
Kokkeby and Zeng do not explicitly disclose detect a loss of visual contact with the object instance.
However, Fragoso teaches detect a loss of visual contact with the object instance [See Fragoso: at least Fig. 5 and par. 90-99 regarding the trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Therefore it would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Kokkeby and Zeng with Fragoso teachings by including “detect a loss of visual contact with the object instance” because this combination has the benefit of providing a target tracking method for controlling an unmanned aerial vehicle (UAV) to avoid obstacle collision [See Fragoso: at least Fig. 5 and par. 90].
Further on, when combined, Fragoso also teaches determine a constrained search region for the object instance based on (i) a last observed position of the object instance and (ii) one or more constraints derived from prior observations of the object instance and an environment of the UAV[See Fragoso: at least Figs. 2-5 and par. 41, 81-85, 89-99 regarding For a sampling based method, the correct choice of a distribution in egospace immediately satisfies completeness and follows an identical procedure as in world coordinates—albeit with the more efficient collision-checking scheme by comparison. For a simulated obstacle data set, embodiments exhibit an egospace implementation of the Lazy PRM method [Bohlin 2000] generated using this approach. FIGS. 3A and 3B illustrate a lazy PRM motion planner implemented entirely in depth-image coordinates (FIG. 3A) for an MAV with negligible dynamics in a simulated forest environment and a world space representation (FIG. 3B). In the depth image, pixels with cooler colors/tones (e.g., trees 302 in the foreground) are closer than pixels with warmer colors/tones (e.g., trees 304 at the sensor horizon). Points (in embodiments of the invention) are sampled, selected and connected entirely in the egospace coordinates, for which the projected pixel coordinates of the motion plan are shown (points 306). White segments 306W of path 306 are obscured by obstacles (e.g., trees 304), but pass safely behind the obstacles 304 by a predetermined safety margin. Black segments 306B path are not obscured by the obstacles 304. An equivalent point cloud and world-space representation is shown in FIG. 3B. in which the path/trajectory 306 avoids the obstacles 304…At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…].
Zeng and Fragoso teach maintain, during the loss of visual contact, a probabilistic estimate of a location of the object instance within the constrained search region[See Zeng: at least par. 69-71 regarding each object track includes a mixture of Gaussian distribution. This distribution can be randomly sampled to obtain a set of particles representing a probability distribution of the object. The collision threat assessment module 120 determines the likelihood for collision based upon a percentage of a number of threats per total particles associated with an object. The likelihood for collision is outputted to the collision counter-measures module 150. See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114 regarding At step 504, the range data is combined into an egospace representation that includes one or more pixels in egospace. Egospace defines/consists of a coordinate system. For each of the pixels of the egospace representation, a value may be assigned that uniquely encodes a distance to one or more of the visible object from a defined focal point. The defined focal point may be located at an arbitrary origin… At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…]; and
reacquire the object instance based on the probabilistic estimate[See Zeng: at least Figs. 1-8 and par. 49, 52-73 regarding For example, in FIG. 8, the boxes A and B are identified as vehicles while the unlabelled box is identified as road-side object. The prediction process module 117 utilizes an object's historical information (i.e., position, image patch, and label of previous cycle) and predicts the current values. The data association module 118 links the current measurements with the predicted objects, or determines the source of a measurement (i.e., position, image patch, and label) is from a specific object. The object tracker module 119 generates updated positions for the objects for storage in the object track database 113…See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114 regarding Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Regarding claim 29, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 28, and are analyzed as previously discussed with respect to that claim. Further on, Fragoso teaches wherein the constrained search region is further determined based on a visibility analysis using depth information and prior sensor viewpoints[See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114, 122 regarding At step 504, the range data is combined into an egospace representation that includes one or more pixels in egospace. Egospace defines/consists of a coordinate system. For each of the pixels of the egospace representation, a value may be assigned that uniquely encodes a distance to one or more of the visible object from a defined focal point. The defined focal point may be located at an arbitrary origin… At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Regarding claim 30, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 28, and are analyzed as previously discussed with respect to that claim. Further on, Zeng and Fragoso teach or suggest wherein maintaining the probabilistic estimate comprises propagating particles along candidate motion paths [See Zeng: at least par. 69-71 regarding each object track includes a mixture of Gaussian distribution. This distribution can be randomly sampled to obtain a set of particles representing a probability distribution of the object. The collision threat assessment module 120 determines the likelihood for collision based upon a percentage of a number of threats per total particles associated with an object. The likelihood for collision is outputted to the collision counter-measures module 150. See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114 regarding At step 504, the range data is combined into an egospace representation that includes one or more pixels in egospace. Egospace defines/consists of a coordinate system. For each of the pixels of the egospace representation, a value may be assigned that uniquely encodes a distance to one or more of the visible object from a defined focal point. The defined focal point may be located at an arbitrary origin… At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Regarding claim 31, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 28, and are analyzed as previously discussed with respect to that claim. Further on, Fragoso teaches wherein the one or more constraints include obstacle constraints derived from a spatial representation of the environment [See Fragoso: at least Fig. 5 and par. 10-12, 86, 89-99 regarding Vision-based obstacle detection has emerged as a popular way of satisfying constraints on sensing, primarily because cameras have low power requirements and are light enough to carry on even the smallest MAVs. Binocular stereo vision has been successfully used for forward-looking obstacle detection, and can be used to populate an occupancy grid world model that is useful for mapping and slow navigation missions… In view of the above, what is needed is the ability to extend the compactness and efficiency advantages of disparity-space obstacle representations to an “egospace” data structure that can accommodate a general range sensor configuration. Further, what is needed is basic obstacle avoidance behavior in egospace coordinates, that may be used with unmanned vehicles (e.g., configuration flat vehicles) as part of a streamlined pipeline for motion planning in unknown, cluttered environments that are referred to herein as “egoplanning”.].
Regarding claim 32, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 28, and are analyzed as previously discussed with respect to that claim. Further on, when combined, Kokkeby and Fragoso teach wherein the instructions further cause the one or more processors to direct at least one image capture device toward a predicted location of the object instance [See Kokkeby: at least Fig. 7 and par. 57-62 regarding In step S126, future UAV and target positions are predicted after comparing track file information with previous predictions... In step S128, future UAV states are predicted and either UAV plan is created (if one does not exist) or an existing UAV plan is modified. In one embodiment, planner module 42 creates and modifies the UAV plan. The plan is based on inputs received from one or more modules, for example, target module 36, ownship module 38, weave corridor module 48, loiter module 50, legs module 46, region search module 52, command module 54 and camera module 56 that are described above. The plan itself may be a model to predict future UAV states. In step S130, the plan is verified and validated. In one embodiment, the plan is validated by the command module. Thereafter, camera control commands and navigation commands are generated in step S132… See Zeng: at least Fig. 5 and par. 89-99 regarding At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path… At step 514, inputs that control the CFV model to follow the trackable path are identified/generated and used to control the UAV…].
Regarding claim 33, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 28, and are analyzed as previously discussed with respect to that claim. Further on, when combined, Zeng and Fragoso teach or suggests wherein the instructions further cause the one or more processors to maintain multiple object tracks and selectively apply the constrained search region to one or more tracked object instances[See Zeng: at least par. 69-71 regarding each object track includes a mixture of Gaussian distribution. This distribution can be randomly sampled to obtain a set of particles representing a probability distribution of the object. The collision threat assessment module 120 determines the likelihood for collision based upon a percentage of a number of threats per total particles associated with an object. The likelihood for collision is outputted to the collision counter-measures module 150. See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114 regarding At step 504, the range data is combined into an egospace representation that includes one or more pixels in egospace. Egospace defines/consists of a coordinate system. For each of the pixels of the egospace representation, a value may be assigned that uniquely encodes a distance to one or more of the visible object from a defined focal point. The defined focal point may be located at an arbitrary origin… At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Regarding claim 34, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 28, and are analyzed as previously discussed with respect to that claim. Further on, when combined, Kokkeby and Fragoso teach wherein the instructions further cause the one or more processors to generate control commands for autonomous navigation based at least in part on the probabilistic estimate[See Kokkeby: at least Fig. 7 and par. 57-62 regarding In step S126, future UAV and target positions are predicted after comparing track file information with previous predictions... In step S128, future UAV states are predicted and either UAV plan is created (if one does not exist) or an existing UAV plan is modified. In one embodiment, planner module 42 creates and modifies the UAV plan. The plan is based on inputs received from one or more modules, for example, target module 36, ownship module 38, weave corridor module 48, loiter module 50, legs module 46, region search module 52, command module 54 and camera module 56 that are described above. The plan itself may be a model to predict future UAV states. In step S130, the plan is verified and validated. In one embodiment, the plan is validated by the command module. Thereafter, camera control commands and navigation commands are generated in step S132… See Zeng: at least Fig. 5 and par. 89-99 regarding At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path… At step 514, inputs that control the CFV model to follow the trackable path are identified/generated and used to control the UAV…].
Regarding claim 35, Kokkeby discloses a non-transitory computer-readable medium storing instructions that, when executed by one or more processors of an unmanned aerial vehicle [See Kokkeby: Fig. 1 and par. 0027 regarding these components can execute from various computer readable media having various data structures stored thereon. Computer executable components (or code) can be stored, for example, on computer readable media including, but not limited to, an ASIC (application specific integrated circuit), CD (compact disc), DVD (digital video disk), ROM (read only memory), floppy disk, hard disk, EEPROM (electrically erasable programmable read only memory) and memory stick in accordance with the claimed subject matter], cause the unmanned aerial vehicle to:
track an object using image using image data captured by one or more image capture devices [See Kokkeby: at least Fig. 1 and par. 28-30 regarding With reference to FIG. 1, one embodiment of the present system 20 includes an automatic target recognition (ATR) module 22 and a multi-sensor integration (MSI) module 24. The ATR module 22 receives a video signal 26 from the UAV (not shown). The ATR module 22 includes instructions to analyze the video signal 26 and generates an output 28 that it sends to the MSI module 24. In addition to the ATR output 28, the MSI module 24 also receives a UAV state signal 30 and a target state signal 32. The signals 30, 32 are generated by the sensors described above, and may also be generated by other sources observing the UAV and/or the target, such as ground-based observers, radar, satellites, etc…].
Kokkeby does not explicitly disclose track an object instance using image data captured by one or more image capture devices.
However, tracking an instance of an object in the image data was well known in the art at the time of the invention was filed as evident from the teaching of Zeng [See Zeng: at least Figs. 3-8 and par. 0007, 0028, 0035, 0036-0044, 0049, 0055, 0073 regarding monitoring a first tracked object and a second tracked object and determining dissimilarity measure between them to determine whether they are a single tracked object. That is, the method is configured to distinguish one or more instances of a tracked object. The method also is configured to classify each identified object. The motion analysis and salient region detection module 112 identifies image areas for further analysis that exhibit a threshold confidence for containing objects or parts of objects. Image areas corresponding to previously identified objects are determined using a priori information including object information of previous time-step such as the object track and position. The salient image areas can be identified using extraction algorithms including a scale-invariant feature transform (SIFT), a speeded up robust feature (SURF) algorithm, and/or a maximally stable extreme region (MSER) algorithm. A histogram of oriented gradients (HOG) and eigen-image coefficients can be used in one embodiment. Other regions of interest in the modified image may be identified using edge, corner, and salient blob detection methods…].
Therefore it would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Kokkeby with Zeng teachings by including “track an object instance using image data captured by one or more image capture devices” for providing the benefit of a multiobject tracking in an image processing system for vehicles [See Zeng: abstract, par. 0007, 0035, 0049, 0055, 0073].
Kokkeby and Zeng do not explicitly disclose detect a loss of visual contact with the object instance.
However, Fragoso teaches detect a loss of visual contact with the object instance [See Fragoso: at least Fig. 5 and par. 90-99 regarding the trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Therefore it would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Kokkeby and Zeng with Fragoso teachings by including “detect a loss of visual contact with the object instance” because this combination has the benefit of providing a target tracking method for controlling an unmanned aerial vehicle (UAV) to avoid obstacle collision [See Fragoso: at least Fig. 5 and par. 90].
Further on, when combined, Fragoso also teaches determine a set of candidate locations for the object instance based on a last observed position of the object instance and one or more constraints associated with motion of the object instance and an environment of the unmanned aerial vehicle [See Fragoso: at least Figs. 2-5 and par. 41, 81-85, 89-99 regarding For a sampling based method, the correct choice of a distribution in egospace immediately satisfies completeness and follows an identical procedure as in world coordinates—albeit with the more efficient collision-checking scheme by comparison. For a simulated obstacle data set, embodiments exhibit an egospace implementation of the Lazy PRM method [Bohlin 2000] generated using this approach. FIGS. 3A and 3B illustrate a lazy PRM motion planner implemented entirely in depth-image coordinates (FIG. 3A) for an MAV with negligible dynamics in a simulated forest environment and a world space representation (FIG. 3B). In the depth image, pixels with cooler colors/tones (e.g., trees 302 in the foreground) are closer than pixels with warmer colors/tones (e.g., trees 304 at the sensor horizon). Points (in embodiments of the invention) are sampled, selected and connected entirely in the egospace coordinates, for which the projected pixel coordinates of the motion plan are shown (points 306). White segments 306W of path 306 are obscured by obstacles (e.g., trees 304), but pass safely behind the obstacles 304 by a predetermined safety margin. Black segments 306B path are not obscured by the obstacles 304. An equivalent point cloud and world-space representation is shown in FIG. 3B. in which the path/trajectory 306 avoids the obstacles 304…At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…].
Zeng and Fragoso teach propagate a probability distribution over the set of candidate locations during a period in which the object instance is not observable[See Zeng: at least par. 69-71 regarding each object track includes a mixture of Gaussian distribution. This distribution can be randomly sampled to obtain a set of particles representing a probability distribution of the object. The collision threat assessment module 120 determines the likelihood for collision based upon a percentage of a number of threats per total particles associated with an object. The likelihood for collision is outputted to the collision counter-measures module 150. See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114 regarding At step 504, the range data is combined into an egospace representation that includes one or more pixels in egospace. Egospace defines/consists of a coordinate system. For each of the pixels of the egospace representation, a value may be assigned that uniquely encodes a distance to one or more of the visible object from a defined focal point. The defined focal point may be located at an arbitrary origin… At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…]; and
re-establish tracking of the object instance based on the probability distribution[See Zeng: at least Figs. 1-8 and par. 49, 52-73 regarding For example, in FIG. 8, the boxes A and B are identified as vehicles while the unlabelled box is identified as road-side object. The prediction process module 117 utilizes an object's historical information (i.e., position, image patch, and label of previous cycle) and predicts the current values. The data association module 118 links the current measurements with the predicted objects, or determines the source of a measurement (i.e., position, image patch, and label) is from a specific object. The object tracker module 119 generates updated positions for the objects for storage in the object track database 113…See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114 regarding Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Regarding claim 36, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 35, and are analyzed as previously discussed with respect to that claim. Further on, Fragoso teaches or suggests wherein determining the set of candidate locations further comprises excluding locations inconsistent with visibility from prior viewpoints[See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114, 122 regarding At step 504, the range data is combined into an egospace representation that includes one or more pixels in egospace. Egospace defines/consists of a coordinate system. For each of the pixels of the egospace representation, a value may be assigned that uniquely encodes a distance to one or more of the visible object from a defined focal point. The defined focal point may be located at an arbitrary origin… At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Regarding claim 37, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 35, and are analyzed as previously discussed with respect to that claim. Further on, Fragoso teaches or suggests wherein the one or more constraints include constraints derived from depth information representing the environment [See Fragoso: at least Figs. 2-5 and par. 41, 81-85, 89-99 regarding For a sampling based method, the correct choice of a distribution in egospace immediately satisfies completeness and follows an identical procedure as in world coordinates—albeit with the more efficient collision-checking scheme by comparison. For a simulated obstacle data set, embodiments exhibit an egospace implementation of the Lazy PRM method [Bohlin 2000] generated using this approach. FIGS. 3A and 3B illustrate a lazy PRM motion planner implemented entirely in depth-image coordinates (FIG. 3A) for an MAV with negligible dynamics in a simulated forest environment and a world space representation (FIG. 3B). In the depth image, pixels with cooler colors/tones (e.g., trees 302 in the foreground) are closer than pixels with warmer colors/tones (e.g., trees 304 at the sensor horizon). Points (in embodiments of the invention) are sampled, selected and connected entirely in the egospace coordinates, for which the projected pixel coordinates of the motion plan are shown (points 306). White segments 306W of path 306 are obscured by obstacles (e.g., trees 304), but pass safely behind the obstacles 304 by a predetermined safety margin. Black segments 306B path are not obscured by the obstacles 304. An equivalent point cloud and world-space representation is shown in FIG. 3B. in which the path/trajectory 306 avoids the obstacles 304…At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…].
Regarding claim 38, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 35, and are analyzed as previously discussed with respect to that claim. Further on, Zeng and Fragoso teach wherein propagating the probability distribution comprises performing particle filtering [See Zeng: at least par. 69-71 regarding each object track includes a mixture of Gaussian distribution. This distribution can be randomly sampled to obtain a set of particles representing a probability distribution of the object. The collision threat assessment module 120 determines the likelihood for collision based upon a percentage of a number of threats per total particles associated with an object. The likelihood for collision is outputted to the collision counter-measures module 150. See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114 regarding At step 504, the range data is combined into an egospace representation that includes one or more pixels in egospace. Egospace defines/consists of a coordinate system. For each of the pixels of the egospace representation, a value may be assigned that uniquely encodes a distance to one or more of the visible object from a defined focal point. The defined focal point may be located at an arbitrary origin… At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path…Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Regarding claim 39, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 35, and are analyzed as previously discussed with respect to that claim. Further on, Zeng and Fragoso teach wherein re-establishing tracking comprises restricting detection to a region corresponding to the probability distribution[See Zeng: at least Figs. 1-8 and par. 49, 52-73 regarding For example, in FIG. 8, the boxes A and B are identified as vehicles while the unlabelled box is identified as road-side object. The prediction process module 117 utilizes an object's historical information (i.e., position, image patch, and label of previous cycle) and predicts the current values. The data association module 118 links the current measurements with the predicted objects, or determines the source of a measurement (i.e., position, image patch, and label) is from a specific object. The object tracker module 119 generates updated positions for the objects for storage in the object track database 113…See Fragoso: Figs. 2-5 and par. 41, 89-99, 110, 114 regarding Further, the trackable path may be generated by generating one or more waypoints (in egospace), and searching for a sequence of waypoints that define the trackable path, wherein the searching includes comparing the depth of the visible objects to egospace coordinates of the waypoints to determine whether the waypoints and segments connecting the waypoints are valid. Such searching may be performed dynamically on the fly (e.g., in real-time) on board the UAV. The trackable path may also be generated based on an acceptable time to contact. In particular, an acceptable time to contact is determined. A check is then made to determine if the assigned destination in egospace is collision free within the acceptable time to contact. The checking converts the acceptable time to contact to an acceptable depth value, and compares the acceptable depth value against a depth value corresponding to one of the visible objects that occlude the assigned destination. The check is repeated for each of the pixels of/in the egospace representation. One of the pixels in the egospace representation that is collision free is selected as a target. A velocity vector of the UAV is then configured such that the velocity vector of the UAV aligns with the target…].
Regarding claim 40, Kokkeby, Zeng and Fragoso teach all of the limitations of claim 35, and are analyzed as previously discussed with respect to that claim. Further on, Kokkeby and Fragoso teach wherein the instructions further cause the unmanned aerial vehicle to direct at least one image capture device toward the region corresponding to the probability distribution [See Kokkeby: at least Fig. 7 and par. 57-62 regarding In step S126, future UAV and target positions are predicted after comparing track file information with previous predictions... In step S128, future UAV states are predicted and either UAV plan is created (if one does not exist) or an existing UAV plan is modified. In one embodiment, planner module 42 creates and modifies the UAV plan. The plan is based on inputs received from one or more modules, for example, target module 36, ownship module 38, weave corridor module 48, loiter module 50, legs module 46, region search module 52, command module 54 and camera module 56 that are described above. The plan itself may be a model to predict future UAV states. In step S130, the plan is verified and validated. In one embodiment, the plan is validated by the command module. Thereafter, camera control commands and navigation commands are generated in step S132… See Zeng: at least Fig. 5 and par. 89-99 regarding At step 512, a trackable path is generated from the UAV to the assigned destination through egospace that avoids collision with the visible objects based on the expanded apparent sizes of each of the visible objects. The trackable path may be generated by generating one or more predefine maneuvers simultaneously in egospace and world space, followed by the selecting and linking the predefined maneuvers to generate the trackable path… At step 514, inputs that control the CFV model to follow the trackable path are identified/generated and used to control the UAV…].
Conclusion
9. Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANA J PICON-FELICIANO whose telephone number is (571)272-5252. The examiner can normally be reached Monday-Friday 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Christopher Kelley can be reached at 571 272 7331. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ana Picon-Feliciano/Examiner, Art Unit 2482
/KAITLIN A RETALLICK/Primary Examiner, Art Unit 2482
Prosecution Insights