Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/26/2026 has been entered.
Status of Claims
Claims 1-227 and 233 are canceled.
Claims 228-232 and 234-254 are pending and have been examined.
Claims 228-232 and 234-254 are either amended directly or via a claim they depend from.
Claims 228-232 and 234-254 are rejected.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/12/2026 was filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
Regarding the Claim Rejections under 35 § USC 102/103: Applicant’s arguments and corresponding amendments, see pages 9-14 filed on 01/26/2026, have been fully considered are respectfully deemed by the Examiner as not being persuasive towards the claims being in condition for allowance. The specific arguments are addressed as follows.
First, regarding Applicant’s arguments that Donahoe does not teach, (Applicant Argument/Amendments, Page 11, Last Line) “to base a virtual gimbal functionality,” the Examiner is in agreement. However, with regards to the argument that, “Donahoe would have to been seen as leading away from the present invention because the document teaches to make use of mechanical adjustments,” the Examiner respectfully disagrees, and has applied an obviousness-type rejection over in Donahoe in view of Pohl in the following, Claim Rejections - 35 USC § 103, section. In order to convince the Examiner that the burden of “teaching away” has been met, explicit evidence must be presented. From the Examiner’s understanding after reviewing the reference, Donahoe’s (MPEP VI. 2141.02) “disclosure does not criticize, discredit, or otherwise discourage the solution claimed,” the solution of which is the case being a virtual gimbal implementation.
Second, regarding Applicant’s argument that, (Applicant Argument/Amendments, Page 13, First Paragraph) “if one saw the claimed features of motion data, measurement data and GPS-data disclosed by Donahoe … it is seemingly not obvious to do so,” the Examiner respectfully disagrees and has cited in greater detail the portion of the Donahoe reference which applies to the newly amended claim limitations in the following, Claim Rejections - 35 USC § 102 section, for Applicant’s consideration.
Claim Objections
Claim 1 is objected to because of the following informalities:
Claim 1, Lines 26-27 contain grammatical issues. One recommended solution to overcome the objection may be, “a virtual gimbal functionality is provided, which enables capability to virtually gimbal the view, by the touch input, wherein the capability to virtually gimbal the view is decoupled from the movement of the UAV.”
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 239-244 and 246-254 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Donahoe et al. (US 2019/0250601 A1)
Claim 239 Discloses: (Currently Amended)
“A computer implemented method for conditioning sensor raw data generated by a multipurpose sensor system of a UAV flying in a physical environment,”
Donahoe teaches, (Paragraph [0032], Lines 2-8) “the UAV 100 depicted in FIG. 1 … may be utilized for a live video feed presented via a GUI according to the introduced technique.”
Donahoe additionally teaches, (Paragraph [0230], Lines 8-14) “The term “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the computing system and that cause the computing system to perform any one or more of the methodologies of the presently disclosed embodiments.”
“the method including: generating sensor raw data by the multipurpose sensor system in the form of: image data from a camera system of the UAV,”
Donahoe teaches, (Paragraph [0030], Lines 1-6) “In the example depicted in FIG. 1, the image capture devices 114 and/or 115 are depicted capturing an object 102 in the physical environment that happens to be a person. In some cases, the image capture devices may be configured to capture images for display to users (e.g., as an aerial video platform).”
Donahoe additionally teaches (Paragraph [0168] Lines 8-10) “each of the image capture devices 114 shown mounted to a UAV 100 in FIG. 1 may include cameras.”
“motion data from an inertial measurement unit (IMU) of the UAV,”
Donahoe teaches, (Paragraph [0218]) “UAV system 4100 may include one or more IMU 4128. An IMU 4128 may measure and report the UAV's velocity, acceleration, orientation, and gravitational forces using a combination of gyroscopes and accelerometers (e.g., accelerometer 4126).”
“measurement data, in particular 3D point data, from a directional distance measuring module of the UAV, in particular wherein the directional distance measuring module measures distances and directions to object surfaces based on the light detection and ranging (lidar) principle,”
Donahoe teaches, (Paragraph [0161], Lines 1-11) “Computer vision may also be applied using sensing technologies other than cameras, such as light detection and ranging (LIDAR) technology. For example, a UAV 100 equipped with LIDAR may emit one or more laser beams in a scan up to 360 degrees around the UAV 100. Light received by the UAV 100 as the laser beams reflect off physical objects in the surrounding physical world may be analyzed to construct a real time 3D computer model of the surrounding physical world. Depth sensing through the use of LIDAR may in some embodiments augment depth sensing through pixel correspondence as described earlier.”
“and global position data from a GNSS receiver module of the UAV, providing a flight control support functionality using support data for supporting the flight control,”
Donahoe teaches, (Paragraph [0219]) “UAV system 4100 may include a global positioning system (GPS) receiver 4120. FIG. 41 shows a GPS receiver 4120 coupled to the peripherals interface 4110. Alternately, the GPS receiver 4120 may be coupled to an input controller 4140 in the I/O subsystem 4160. The GPS receiver 4120 may receive signals from GPS satellites in orbit around the earth, calculate a distance to each of the GPS satellites (through the use of GPS software), and thereby pinpoint a current global position of UAV 100.”
Donahoe additionally teaches, (Paragraph [0220]) “the software components stored in memory 4116 may include an operating system, a communication module (or set of instructions), a flight control module (or set of instructions), a localization module (or set of instructions), a computer vision module (or set of instructions), a graphics module (or set of instructions), and other applications (or sets of instructions).”
“and a sensor data recording functionality for recording sensor data, which enables a generation of a representation of the physical environment of the UAV,
Donahoe teaches, (Paragraph [0141], Lines 1-2) “the image capture device 115 records video continuously from takeoff to landing.”
Donahoe additionally teaches, (Paragraph [0136], Lines 10-14) “In some embodiments, the view presented via the GUI can include indications of obstacles in the physical environment in the form of a 3D occupancy map … The 3D occupancy map may be continually generated and updated based on data received from one or more sensors onboard the UAV 100 as the UAV 100 flies through the physical environment.”
“autonomously supporting, by the flight control support functionality, the control of the flight of the UAV, and recording, by the sensor data recording functionality, sensor data, which enables the generation of a representation of the physical environment of the UAV,”
Donahoe teaches, (Paragraph [0147], Lines 1-5) “A navigation system 120 of a UAV 100 may employ any number of systems and techniques for localization. FIG. 32 shows an illustration of an example localization system 3200 that may be utilized to guide autonomous navigation of a vehicle such as a UAV 100.”
“wherein receiving the sensor raw data, by a sensor raw data conditioning unit of the UAV, and conditioning, by the sensor raw data conditioning unit, the sensor raw data to generate the support data and the sensor data, wherein
For the following mapping, the Examiner wishes to clarify their interpretation of support data.
Support data is being interpreted as data which contributes to controlling the flight of the UAV by, for example, avoiding obstacles or providing flight stability. Conditioning support data is being broadly interpreted as creating, gathering, applying etc. support data to be used by the UAV system. The Examiner’s interpretation is derived from at least, (Applicant’s Specification, Paragraph [0455], Lines 1-2) “A step of providing a flight control support functionality using support data for supporting the flight control,” and further that, (Applicant’s, Specification, Paragraph [0458]) “Supporting the flight of the UAV can relate to, for example, autonomously avoid collisions/obstacle stabilize the flight of the UAV, etc.”
Regarding conditioning image data to generate support data …
Donahoe teaches, (Paragraph [0117], Lines 12-14) “The motion planning 130 will utilize perception inputs from the various sensors to generate a safe trajectory that avoids any obstacles.”
Regarding mapping conditioning motion data to generate support data …
Donahoe teaches, (Paragraph [0102], Lines 5-8) “the manner in which the UAV 100 avoids the obstacle will depend on a number of factors such as the relative position and/or motions between the UAV 100 and obstacle,” and further that, (Paragraph [0160], Lines 11-14) “A navigation system 120 of a UAV 100 can be configured to navigate the physical environment by planning a 3D trajectory 3420 through the 3D occupancy map 3402 that avoids the voxels.”
Regarding conditioning measurement data to generate support data …
Donahoe teaches, (Paragraph [0161], Lines 9-11) “Depth sensing through the use of LIDAR may in some embodiments augment depth sensing through pixel correspondence as described earlier,” and further that, (Paragraph [0160], Lines 8-21) “Each of the voxels in the 3D occupancy map 3402 corresponds to a space in the physical environment that is at least partially occupied by a physical object. A navigation system 120 of a UAV 100 can be configured to navigate the physical environment by planning a 3D trajectory 3420 through the 3D occupancy map 3402 that avoids the voxels. In some embodiments, this 3D trajectory 3420 plan using the 3D occupancy map 1402 can be optimized by applying an image space motion planning process. In such an embodiment, the planned 3D trajectory 3420 of the UAV 100 is projected into an image space of captured images for analysis relative to certain identified high cost regions (e.g., regions having invalid depth estimates).”
Regarding conditioning global position data to generate support data …
Donahoe teaches, (Paragraph [0049], Lines 9-14) “For example, the motion planner 130 may generate a planned trajectory that maneuvers the UAV 100 to a particular GPS coordinate while following a tracked object, capturing images of the tracked object, maintaining line of sight with the tracked object, and avoiding collisions with other objects.”
For the following mapping, the Examiner additionally wishes to clarify their interpretation of sensor data.
Conditioning sensor data is being broadly interpolated as creating, gathering, applying etc. support data to be used by the UAV system to generate a display of a view to the environment of the UAV system. The Examiner’s interpretation is derived from at least, (Applicant’s Specification, Paragraph [0456], Lines 3-6) “The representation can be generated based on the recorded sensor data, for example, in a post-processing step. Furthermore, the recorded sensor data can enable, for example, the generation and display of a view to the environment of the UAV.”
Regarding conditioning image data to generate sensor data …
Donahoe teaches, (Paragraph [0215], Lines 41-44) “a UAV 100 may include some cameras dedicated for image capture of a subject and other cameras dedicated for image capture for visual navigation (e.g., through visual inertial odometry).”
Donahoe additionally teaches, (Paragraph [0153], Lines 7-14) “Computer vision may be used to estimate position and/or orientation using a number of different methods. For example, in some embodiments, raw image data received from one or more image capture devices (onboard or remote from the UAV 100) may be received and processed to correct for certain variables (e.g., differences in camera orientation and/or intrinsic parameters (e.g., lens variations)),” and that, (Paragraph [0198], Lines 1-4) “the tracking system 140 may be configured to process images (e.g., the raw pixel data) received from one or more image capture devices 114/115 onboard a UAV 100,” as well as, (Paragraph [0215], Lines 1-4) “UAV system 4100 may also include one or more image capture devices 4134. Image capture devices 4134 may be the same as the image capture devices 114/115 of UAV 100 described with respect to FIG. 1,” and further that, (Paragraph [0030], Lines 4-5) “the image capture devices may be configured to capture images for display to users.”
Regarding conditioning measurement data to generate sensor data …
Donahoe teaches, (Paragraph [0076, Lines 19-22) “In some cases, the view displayed in the GUI will directly correspond with a view from an image capture device 115 capturing images (including video) for recording and later display,” and that, (Paragraph [0161], Lines 9-11) “Depth sensing through the use of LIDAR may in some embodiments augment depth sensing through pixel correspondence as described earlier,” as well as, (Paragraph [0160], Lines 6-8) “FIG. 34 shows an example view of a 3D occupancy map 3402 of a physical environment including multiple cubical voxels.”
Regarding conditioning motion data to generate sensor data …
Donahoe teaches, (Paragraph [0111]) “A velocity slider can be implemented for the zoom control element 1910 to control the range or distance or zoom on the subject. Sliding element 1910 up moves the UAV 100 toward the subject or makes the subject larger in the recorded image or video (e.g., through optical or digital zoom). Sliding element 1910 down moves the UAV 100 away from the subject or makes the subject smaller in the recorded image or video (e.g., through optical or digital zoom).”
Regarding conditioning global position data to generate sensor data …
Donahoe teaches, (Paragraph [0030], Lines 4-5) “the image capture devices may be configured to capture images for display to users,” and that, (Paragraph [0171], Lines 1-11) “While a tracking system 140 can be configured to rely exclusively on visual data from image capture devices onboard a UAV 100, data from other sensors (e.g., sensors on the object, on the UAV 100, or in the environment) can be incorporated into this framework when available. Additional sensors may include GPS, IMU, barometer, magnetometer, and cameras or other devices such as a mobile device 104. For example, a GPS signal from a mobile device 104 held by a person can provide rough position measurements of the person that are fused with the visual information from image capture devices onboard the UAV 100.”
Claim 240 Discloses: (Previously Presented)
“The method according to claim 239, the flight control support functionality including a visual inertial system (VIS), the visual inertial system using support data in the form of conditioned image data to derive motion data related to the movement/motion of the UAV based on tracking predetermined features in the image data.”
Donahoe teaches, (Paragraph [0152], Lines 1-2) “An inertial measurement unit (IMU) may be used to estimate position and/or orientation of a device.”
Donahoe additionally teaches, (Paragraph [0224]) “A computer vision module, which may be a component of a graphics module, provides analysis and recognition of graphics data. For example, while UAV 100 is in flight, the computer vision module along with a graphics module (if separate), GPU 4112, and image capture devices(s) 4134 and/or proximity sensors 4130 may recognize and track the captured image of an object located on the ground. The computer vision module may further communicate with a localization/navigation module and flight control module to update a position and/or orientation of the UAV 100 and to provide course corrections to fly along a planned trajectory through a physical environment.”
Donahoe teaches, (Paragraph [0215], Lines 41-44) “a UAV 100 may include some cameras dedicated for image capture of a subject and other cameras dedicated for image capture for visual navigation (e.g., through visual inertial odometry).”
Claim 241 Discloses: (Previously Presented)
“The method according to claim 240, wherein said conditioning includes conditioning the image data based on a criterion relating to using the image data by the VIS.”
Donahoe teaches, (Paragraph [0044], Lines 6-17) “certain built-in objectives, such as obstacle avoidance and vehicle dynamic limits, can be combined with other input objectives (e.g., a tracking objective) as part of a trajectory generation process. In some embodiments, the trajectory generation process can include gradient-based optimization, gradient-free optimization, sampling, end-to-end learning, or any combination thereof. The output of this trajectory generation process can be a planned trajectory over some time horizon (e.g., 10 seconds) that is configured to be interpreted and utilized by a flight controller 160 to generate control commands that cause the UAV 100 to maneuver according to the planned trajectory.” Under broadest reasonable interpretation, the gradient-based optimization can be considered a form of conditioning imaging data for VIS, as it modifies the image data to make it better suited for the analysis performed by VIS.
Claim 242 Discloses: (Previously Presented)
“The method according to claim 239, wherein the sensor data enables a generation and display of a view of the physical environment of the UAV to a user,”
Donahoe teaches, (Paragraph [0076], Lines 1-10) “FIGS. 6-31 show a series of screen captures illustrating various features of an example GUI that can be implemented to facilitate user control of the previously described UAV 100. The GUI can be displayed as a graphical output via an interactive display device 402 (e.g., a touch-sensitive display) of a computing device such as mobile device 104 (e.g., a user's tablet or smartphone). In each of the screen captures, the GUI includes a view of the surrounding physical environment based, at least in part, on sensor data captured by sensors onboard the UAV 100.”
“and the generation and display is based on sensor data in the form of conditioned image data.”
Donahoe teaches, (Paragraph [0174], Lines 10-16) “An array of image capture devices 114 about a perimeter of the UAV 100 can provide low-latency information about objects up to 360 degrees around the UAV 100 and can be used to compute depth using stereo vision algorithms. Conversely, the other image capture device 115 can provide more detailed images (e.g., high resolution, color, etc.) in a limited FOV.”
Donahoe additionally teaches, (Paragraph [0175], Lines 3-7) “the high-resolution color information from an image capture device 115 can be fused with depth information from the image capture devices 114 to create a 3D representation of a tracked object.”
Donahoe additionally teaches, (Paragraph [0147]) “In some embodiments, the object detection system processes received images and outputs a dense per-pixel segmentation, where each pixel is associated with a value corresponding to either an object class label (e.g., human, building, car, animal, etc.) and/or a likelihood of belonging to that object class.”
Claim 243 Discloses: (Previously Presented)
“The method according to claim 240, wherein said conditioning includes conditioning the image data based on a criterion relating to generating sensor data based on the image data”
Donahoe teaches, (Paragraph [0175], Lines 3-7) “the high-resolution color information from an image capture device 115 can be fused with depth information from the image capture devices 114 to create a 3D representation of a tracked object.”
Donahoe additionally teaches, (Paragraph [0147]) “In some embodiments, the object detection system processes received images and outputs a dense per-pixel segmentation, where each pixel is associated with a value corresponding to either an object class label (e.g., human, building, car, animal, etc.) and/or a likelihood of belonging to that object class.”
“to enable a generation and display of a view of the physical environment of the UAV to a user.”
Donahoe teaches, (Paragraph [0076], Lines 1-10) “FIGS. 6-31 show a series of screen captures illustrating various features of an example GUI that can be implemented to facilitate user control of the previously described UAV 100. The GUI can be displayed as a graphical output via an interactive display device 402 (e.g., a touch-sensitive display) of a computing device such as mobile device 104 (e.g., a user's tablet or smartphone). In each of the screen captures, the GUI includes a view of the surrounding physical environment based, at least in part, on sensor data captured by sensors onboard the UAV 100.”
Claim 244 Discloses: (Previously Presented)
“The method according to claim 239, the flight control support functionality including a collision avoidance functionality, the collision avoidance functionality using support data in the form of conditioned measurement data to detect obstacles in the physical environment and avoid the obstacles.”
Donahoe teaches, (Paragraph [0194], Lines 10-15) “Through a motion planning process, a navigation system of the UAV 100 may generate control commands configured to cause the UAV 100 to maneuver, for example, to avoid a collision, maintain separation with a tracked object in motion, and/or satisfy any other navigation objectives.”
Claim 246 Discloses: (Previously Presented)
“The method according to claim 245, wherein said conditioning includes conditioning the measurement data based on a criterion relating to generating sensor data based on the measurement data”
Donahoe teaches, (Paragraph [0174], Lines 10-16) “An array of image capture devices 114 about a perimeter of the UAV 100 can provide low-latency information about objects up to 360 degrees around the UAV 100 and can be used to compute depth using stereo vision algorithms. Conversely, the other image capture device 115 can provide more detailed images (e.g., high resolution, color, etc.) in a limited FOV.”
Donahoe additionally teaches, (Paragraph [0175], Lines 3-7) “the high-resolution color information from an image capture device 115 can be fused with depth information from the image capture devices 114 to create a 3D representation of a tracked object.”
Donahoe additionally teaches, (Paragraph [0147]) “In some embodiments, the object detection system processes received images and outputs a dense per-pixel segmentation, where each pixel is associated with a value corresponding to either an object class label (e.g., human, building, car, animal, etc.) and/or a likelihood of belonging to that object class.”
“to enable the generation and display of a view of the physical environment of the UAV to a user.”
Donahoe teaches, (Paragraph [0076], Lines 1-10) “FIGS. 6-31 show a series of screen captures illustrating various features of an example GUI that can be implemented to facilitate user control of the previously described UAV 100. The GUI can be displayed as a graphical output via an interactive display device 402 (e.g., a touch-sensitive display) of a computing device such as mobile device 104 (e.g., a user's tablet or smartphone). In each of the screen captures, the GUI includes a view of the surrounding physical environment based, at least in part, on sensor data captured by sensors onboard the UAV 100.”
Claim 247 Discloses:
“The method according to claim 239, wherein the sensor data enables a display of the representation of the physical environment of the UAV to a user, and the display is based on sensor data in the form of conditioned measurement data including 3D point data.”
Donahoe teaches, (Paragraph [0076], Lines 1-10) “FIGS. 6-31 show a series of screen captures illustrating various features of an example GUI that can be implemented to facilitate user control of the previously described UAV 100. The GUI can be displayed as a graphical output via an interactive display device 402 (e.g., a touch-sensitive display) of a computing device such as mobile device 104 (e.g., a user's tablet or smartphone). In each of the screen captures, the GUI includes a view of the surrounding physical environment based, at least in part, on sensor data captured by sensors onboard the UAV 100.”
Donahoe teaches, (Paragraph [0161], Lines 9-14) “Depth sensing through the use of LIDAR may in some embodiments augment depth sensing through pixel correspondence as described earlier. Further, images captured by cameras (e.g., as described earlier) may be combined with the laser constructed 3D models to form textured 3D models.”
Claim 248 Discloses: (Previously Presented)
“The method according to claim 247, wherein said conditioning includes conditioning the measurement data based on a criterion relating to generating sensor data based on the measurement data to enable a display of the representation of the physical environment of the UAV to a user.”
Donahoe teaches, (Paragraph [0174], Lines 10-16) “An array of image capture devices 114 about a perimeter of the UAV 100 can provide low-latency information about objects up to 360 degrees around the UAV 100 and can be used to compute depth using stereo vision algorithms. Conversely, the other image capture device 115 can provide more detailed images (e.g., high resolution, color, etc.) in a limited FOV.”
Donahoe additionally teaches, (Paragraph [0175], Lines 3-7) “the high-resolution color information from an image capture device 115 can be fused with depth information from the image capture devices 114 to create a 3D representation of a tracked object.”
Donahoe additionally teaches, (Paragraph [0147]) “In some embodiments, the object detection system processes received images and outputs a dense per-pixel segmentation, where each pixel is associated with a value corresponding to either an object class label (e.g., human, building, car, animal, etc.) and/or a likelihood of belonging to that object class.”
Donahoe additionally teaches, (Paragraph [0076], Lines 1-10) “FIGS. 6-31 show a series of screen captures illustrating various features of an example GUI that can be implemented to facilitate user control of the previously described UAV 100. The GUI can be displayed as a graphical output via an interactive display device 402 (e.g., a touch-sensitive display) of a computing device such as mobile device 104 (e.g., a user's tablet or smartphone). In each of the screen captures, the GUI includes a view of the surrounding physical environment based, at least in part, on sensor data captured by sensors onboard the UAV 100.”
Claim 249 Discloses: (Previously Presented)
“The method according to claim 239, the flight control support functionality using support data in the form of conditioned global position data for controlling the flight of the UAV in the physical environment.”
Donahoe teaches, (Paragraph [0219]) “UAV system 4100 may include a global positioning system (GPS) receiver 4120. FIG. 41 shows a GPS receiver 4120 coupled to the peripherals interface 4110. Alternately, the GPS receiver 4120 may be coupled to an input controller 4140 in the I/O subsystem 4160. The GPS receiver 4120 may receive signals from GPS satellites in orbit around the earth, calculate a distance to each of the GPS satellites (through the use of GPS software), and thereby pinpoint a current global position of UAV 100.”
Claim 250 Discloses: (Previously Presented)
“The method according to claim 239, wherein the sensor data enables a display of the representation of the physical environment of the UAV to a user, and the display is based on sensor data in the form of conditioned global position data by using the conditioned global position data to assign a global position to the representation.”
Donahoe teaches, (Paragraph [0076], Lines 1-10) “FIGS. 6-31 show a series of screen captures illustrating various features of an example GUI that can be implemented to facilitate user control of the previously described UAV 100. The GUI can be displayed as a graphical output via an interactive display device 402 (e.g., a touch-sensitive display) of a computing device such as mobile device 104 (e.g., a user's tablet or smartphone). In each of the screen captures, the GUI includes a view of the surrounding physical environment based, at least in part, on sensor data captured by sensors onboard the UAV 100.”
Donahoe additionally teaches, (Paragraph [0152], Lines 18-28) “an embodiment utilizing localization using an IMU may include localization data from other sources (e.g., the GPS, Wi-Fi, and cellular systems described above) to continually update the last known position and/or orientation of the object. Further, a nonlinear estimation algorithm (one embodiment being an “extended Kalman filter”) may be applied to a series of measured positions and/or orientations to produce a real-time optimized prediction of the current position and/or orientation based on assumed uncertainties in the observed data.”
Claim 251 Discloses: (Previously Presented)
“The method according to claim 250, wherein said conditioning includes conditioning the global position data based on a criterion relating to generating sensor data based on the global position data to enable a display of the representation of the physical environment of the UAV to a user with the representation having assigned thereto a global position.”
Donahoe teaches, (Paragraph [0152], Lines 18-28) “an embodiment utilizing localization using an IMU may include localization data from other sources (e.g., the GPS, Wi-Fi, and cellular systems described above) to continually update the last known position and/or orientation of the object. Further, a nonlinear estimation algorithm (one embodiment being an “extended Kalman filter”) may be applied to a series of measured positions and/or orientations to produce a real-time optimized prediction of the current position and/or orientation based on assumed uncertainties in the observed data.”
Donahoe additionally teaches, (Paragraph [0076], Lines 1-10) “FIGS. 6-31 show a series of screen captures illustrating various features of an example GUI that can be implemented to facilitate user control of the previously described UAV 100. The GUI can be displayed as a graphical output via an interactive display device 402 (e.g., a touch-sensitive display) of a computing device such as mobile device 104 (e.g., a user's tablet or smartphone). In each of the screen captures, the GUI includes a view of the surrounding physical environment based, at least in part, on sensor data captured by sensors onboard the UAV 100.”
Claim 252 Discloses: (Previously Presented)
“The method according to claim 239, the support data and the sensor data each including a combination of at least two of image data, motion data, measurement data, and global position data.”
Donahoe teaches, (Paragraph [0039], Lines 1-8) “the motion planner 130, operating separately or in conjunction with the tracking system 140, is configured to generate a planned trajectory through a three-dimensional (3D) space of a physical environment based, for example, on images received from image capture devices 114 and/or 115, data from other sensors 112 (e.g., IMU, GPS, proximity sensors, etc.), and/or one or more control inputs 170.”
Claim 253 Discloses: (Previously Presented)
“The method according to claim 239, the sensor raw data being generated at a predefined maximum rate and at a predefined maximum resolution, wherein conditioning the sensor raw data includes providing the sensor raw data at a predefined resolution and/or at a predefined rate based on a criterion relating to generating support data and/or sensor data from the sensor raw data.”
Donahoe teaches, (Paragraph [0196], Lines 5-10) “a tracking system 140 may be configured to take accurate measurements of the current position and motion of an object and use differentiated velocities and/or accelerations to predict a trajectory a short time (e.g., seconds) into the future and continually update such prediction as new measurements are taken.”
Donahoe additionally teaches, (Paragraph [0197], Lines 2-8) “the tracking system 140 may also be configured to perform a frame-to-frame tracking process, for example, to detect motion of a particular set or region of pixels in images at subsequent time frame (e.g., video frames). Such a process may involve applying a mean-shift algorithm, a correlation filter, and/or a deep network.”
Donahoe additionally teaches, (Paragraph [0032], Lines 7-12) “Accordingly, in some embodiments, the image capture device 115 may be configured to capture relatively high resolution (e.g., 3840×2160 or higher) color images, while the image capture devices 114 may be configured to capture relatively low resolution (e.g., 320×240 or lower) grayscale images.”
Claim 254 Discloses: (Previously Presented)
“A computer program product comprising machine readable program code stored in a non-transitory computer-readable medium, which when executed by processing units related to”
Donahoe teaches, (Paragraph [0230], Lines 8-14) “The term “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the computing system and that cause the computing system to perform any one or more of the methodologies of the presently disclosed embodiments.”
“a mobile control device having a touch sensitive display and/or a UAV enables conditioning of sensor raw data according to the method of claim 239.”
Donahoe teaches, (Paragraph [0136], Lines 10-14) “In some embodiments, the view presented via the GUI can include indications of obstacles in the physical environment in the form of a 3D occupancy map … The 3D occupancy map may be continually generated and updated based on data received from one or more sensors onboard the UAV 100 as the UAV 100 flies through the physical environment.”
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 228-229 and 234-237 are rejected under 35 U.S.C. 103 as being unpatentable over Donahoe in view of Pohl et al. (US 2019/0324448 A1)
Claim 228 Discloses: (Currently Amended)
“Computer implemented method for providing a live- view of a UAV's physical environment,”
Donahoe teaches, (Paragraph [0032], Lines 2-8) “the UAV 100 depicted in FIG. 1 … may be utilized for a live video feed presented via a GUI according to the introduced technique.”
Donahoe additionally teaches, (Paragraph [0230], Lines 8-14) “The term “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the computing system and that cause the computing system to perform any one or more of the methodologies of the presently disclosed embodiments.”
“the method including: continuously generating a view of the physical environment of the UAV based on image data from a camera system of the UAV,”
Donahoe teaches, (Paragraph [0136], Lines 10-14) “In some embodiments, the view presented via the GUI can include indications of obstacles in the physical environment in the form of a 3D occupancy map … The 3D occupancy map may be continually generated and updated based on data received from one or more sensors onboard the UAV 100 as the UAV 100 flies through the physical environment.”
“the camera system including a plurality of cameras arranged peripherally at the UAV, the cameras each having a field of view with a fixed orientation in relation to the UAV and directed away from the UAV,”
PNG
media_image1.png
466
382
media_image1.png
Greyscale
Donahoe teaches, (Paragraph [0030], Lines 1-6) “In the example depicted in FIG. 1, the image capture devices 114 and/or 115 are depicted capturing an object 102 in the physical environment that happens to be a person. In some cases, the image capture devices may be configured to capture images for display to users (e.g., as an aerial video platform),” and that, (Paragraph [0168] Lines 8-10) “each of the image capture devices 114 shown mounted to a UAV 100 in FIG. 1 may include cameras.”
“and the camera system providing available image data of the plurality of cameras for generating an all-round view to the physical environment,”
Donahoe teaches, (Paragraph [0031], Lines 10-15) “the example configuration of UAV 100 depicted in FIG. 1 includes an array of multiple stereoscopic image capture devices 114 placed around a perimeter of the UAV 100 so as to provide stereoscopic image capture up to a full 360 degrees around the UAV 100.”
“continuously displaying the view of the physical environment in a live-view by a touch sensitive display,”
Donahoe teaches, (Paragraph [0030], Lines 1-6) “In the example depicted in FIG. 1, the image capture devices 114 … may be configured to capture images for display to users (e.g., as an aerial video platform),” and that, (Paragraph [0035], Lines 10-12) “the mobile device 104 may include a touch screen display and an associated GUI for receiving user inputs and presenting information.”
“receiving and identifying a touch input, indicative of a desired viewing direction in which the view of the physical environment is to be generated, and based thereon generating and displaying in the live-view a view of the physical environment in the desired viewing direction, wherein selecting the image data from the available image data based on the desired viewing direction, and generating and displaying in the live-view the view of the physical environment in the desired viewing direction based on the selected image data,”
Donahoe teaches, (Paragraph [0062], Lines 4-6) “the view of the physical environment may include a live video feed from an image capture device 114/115 onboard the UAV.”
Donahoe additionally teaches, (Paragraph [0043], Lines 10-14) “the tracking system 140 may generate control commands configured to adjust an orientation of an image capture device 115 so as to keep the tracked object centered in the field of view (FOV) of the image capture device 115 while the UAV 100 is in motion.”
Donahoe additionally teaches, (Paragraph [0079]) “As shown in FIG. 6, a user can input a pan/tilt command by dragging a finger 610 across a displayed view 402a-b of the physical environment.”
Donahoe additionally teaches, (Paragraphs [0137-0138]) “In some embodiments, the GUI can include views of the physical environment from perspectives other than that of the image capture device 115. For example, FIG. 31 shows a screen 3100 of the example GUI including a plan view or overhead map 3110 of the physical environment. In the example depicted in FIG. 31, the plan view 3110 is included as a separate view overlaying (at least partially) the main view 3106 (e.g., a live stream from the image capture device 115). A user may switch between views, for example, by touching the plan view 3110. The plan view 3110 may be generated based on sensors onboard the UAV 100 as it flies through the physical environment as well as data from other sources such as other sensing devices in the vicinity (e.g., other UAVs and/or other mobile devices) or other data sources such as a database including maps and other environmental data. Although FIG. 31 shows a plan view perspective, other views can similarly be constructed from other perspectives. For example, a view from a user's perspective can be generated based on a continually updated 3D model of the surrounding physical environment based on data from sensors onboard the UAV 100. This might allow a user on the ground to effectively see behind objects, for example, where the view is presented as an augmentation via an AR or VR device.”
“wherein: by: selecting the image data from the available image data based on the desired viewing direction, and generating and displaying in the live-view the view of the physical environment in the desired viewing direction based on the selected image data, a virtual gimbal functionality is provided, which enables to virtually gimbal the view, by the touch input, wherein virtually gimbal the view is decoupled from the movement of the UAV.”
Donahoe does not teach the preceding limitations. However, Donahoe does teach the following.
Donahoe teaches, (Paragraph [0034], Lines 10-24) “The UAV 100 may be configured to automatically adjust an orientation of the image capture device 115 to track image capture of an object (e.g., human subject 102) as both the UAV 100 and object are in motion through the physical environment. In some embodiments, this adjustable mechanism may include a mechanical gimbal mechanism that rotates an attached image capture device about one or more axes. In some embodiments, the gimbal mechanism may be configured as a hybrid mechanical-digital gimbal system coupling the image capture device 115 to the body of the UAV 100. In a hybrid mechanical-digital gimbal system, orientation of the image capture device 115 about one or more axes may be adjusted by mechanical means, while orientation about other axes may be adjusted by digital means.” Therefore, Donahoe teaches using a hybrid digital-mechanical gimbal system in the context of object tracking. However, the gimbal is not a virtual gimbal.
Donahoe additionally teaches, (Paragraph [0079]) “FIG. 6 shows a sequence of screens 600a and 600b that illustrate a panning/tilting feature that can be implemented using the described GUI. As shown in FIG. 6, a user can input a pan/tilt command by dragging a finger 610 across a displayed view 402a-b of the physical environment. In some embodiments, an interactive display device 402 may detect the user interaction and, depending on a selected control mode, a GUI module 404 may interpret the detected user interaction as a dragging gesture that is indicative of a pan and/or tilt command … In some embodiments, this input by the user may cause a gimbaled camera such as image capture device 115 to rotate while the UAV 100 remains stationary such that the view presented in the GUI pans and/or tilts.” Therefore, a gimbaled camera may operate independently of the motion of the UAV. Once again, the gimbal is not explicitly virtual.
However, it would have been obvious to modify the Donahoe reference to arrive at the virtual gimbal system as claimed, in light of, for example, Pohl.
Despite being directed towards a head mounted device, Pohl is relevant to the Applicant’s disclosure due to its description of utilizing a virtual gimbal to control the view from a UAV.
Pohl teaches, (Paragraph [0056-0057]) “Having a plurality of cameras able to capture images or video of the vicinity of the UAV and processors to generate a spherical image based on the captured images, allows the controlling device to seamlessly display all directions of the vicinity of the UAV without the need to control the UAV to move. For example, the field of view displayed within a head mounted device may change based on rotation of the head mounted device. Accordingly, roll, pitch, and yaw movements of the head mounted device of the unmanned aerial vehicle controlling device will translate into displaying a different field of view of the visual sphere. This does not required rolling, pitching, and yawing the UAV itself. This feature is referred to as a virtual gimbal.”
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to combine a virtual gimbal system to implement the touchscreen UAV viewport selection system of Donahoe in light of the Pohl reference, in order to yield predictable results.
Combining the references would yield the benefits of implementing a well-known methodology of using a virtual gimbal to alter the viewpoint from a UAV in a decoupled fashion from the movement of the UAV itself. As Pohl describes, (Paragraph [00570]) “roll, pitch, and yaw movements … of the unmanned aerial vehicle controlling device will translate into displaying a different field of view of the visual sphere. This does not required rolling, pitching, and yawing the UAV itself. This feature is referred to as a virtual gimbal.”
Claim 229 Discloses: (Previously Presented)
“The method according to claim 228, including receiving the touch input by the touch sensitive display, the touch sensitive display comprising a plurality of touch zones spread to the live-view, wherein the desired viewing direction is determined based on identifying the touch zone, where the touch input is received, and a touch zone having assigned thereto, predetermined image data selection information based on which the image data is selected from the available image data.”
Donahoe teaches, (Paragraph [0079]) “FIG. 6 shows a sequence of screens 600a and 600b that illustrate a panning/tilting feature that can be implemented using the described GUI. As shown in FIG. 6, a user can input a pan/tilt command by dragging a finger 610 across a displayed view 402a-b of the physical environment. In some embodiments, an interactive display device 402 may detect the user interaction and, depending on a selected control mode, a GUI module 404 may interpret the detected user interaction as a dragging gesture that is indicative of a pan and/or tilt command. This interpreted interaction may then be translated into a behavioral objective that is fed into a motion planner 130 such that as the user drags the finger 610 across the screen, the displayed view pans and/or tilts based on the detected dragging motion. Note that the manner in which the UAV 100 responds to produce this pan and/or tilt effect will depend on the implementation and the capabilities of the UAV 100. For example, in some embodiments, this input by the user may cause the UAV 100 to rotate in place about a current position such that the view presented in the GUI pans and/or tilts. In some embodiments, this input by the user may cause a gimbaled camera such as image capture device 115 to rotate while the UAV 100 remains stationary such that the view presented in the GUI pans and/or tilts. In some embodiments, this input by the user may cause some combination of motion by the UAV 100 and rotation of a gimbaled image capture device 115.”
Claim 234 Discloses: (Previously Presented)
“A computer program product comprising machine readable program code stored in a non-transitory machine-readable medium, which when executed by processing units related to a mobile control device having a touch sensitive display and/or a UAV enables providing a live-view of a UAV's physical environment according to the method of claim 228.”
Donahoe teaches, (Paragraph [0032], Lines 2-8) “the UAV 100 depicted in FIG. 1 … may be utilized for a live video feed presented via a GUI according to the introduced technique.”
Donahoe additionally teaches, (Paragraph [0230], Lines 8-14) “The term “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the computing system and that cause the computing system to perform any one or more of the methodologies of the presently disclosed embodiments.”
Claim 235 Discloses: (Previously Presented)
“The system for controlling the flight of a UAV in a physical environment, the system including: a UAV having a camera system including a plurality of cameras arranged peripherally at the UAV,”
Donahoe teaches, (Paragraph [0032], Lines 2-8) “the UAV 100 depicted in FIG. 1 … may be utilized for a live video feed presented via a GUI according to the introduced technique.”
“the cameras having a field of view with a fixed orientation in relation to the UAV and directed away from the UAV,”
PNG
media_image1.png
466
382
media_image1.png
Greyscale
Donahoe teaches, (Paragraph [0030], Lines 1-6) “In the example depicted in FIG. 1, the image capture devices 114 and/or 115 are depicted capturing an object 102 in the physical environment that happens to be a person. In some cases, the image capture devices may be configured to capture images for display to users (e.g., as an aerial video platform).”
Donahoe additionally teaches (Paragraph [0168] Lines 8-10) “each of the image capture devices 114 shown mounted to a UAV 100 in FIG. 1 may include cameras.”
“and the camera system providing available image data of the plurality of cameras for generating an all-round view to the physical environment,
Donahoe teaches, (Paragraph [0031], Lines 10-15) “the example configuration of UAV 100 depicted in FIG. 1 includes an array of multiple stereoscopic image capture devices 114 placed around a perimeter of the UAV 100 so as to provide stereoscopic image capture up to a full 360 degrees around the UAV 100.”
“and a computer program product according to claim 234.”
Donahoe teaches, (Paragraph [0230], Lines 8-14) “The term “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the computing system and that cause the computing system to perform any one or more of the methodologies of the presently disclosed embodiments.”
Claim 236 Discloses: (Previously Presented)
“The system according to claim 235, the UAV having a directional distance measuring module recording directional distance information by measuring the physical environment.”
Donahoe teaches, (Paragraph [0161], Lines 1-11) “Computer vision may also be applied using sensing technologies other than cameras, such as light detection and ranging (LIDAR) technology. For example, a UAV 100 equipped with LIDAR may emit one or more laser beams in a scan up to 360 degrees around the UAV 100. Light received by the UAV 100 as the laser beams reflect off physical objects in the surrounding physical world may be analyzed to construct a real time 3D computer model of the surrounding physical world. Depth sensing through the use of LIDAR may in some embodiments augment depth sensing through pixel correspondence as described earlier.”
Claim 237 Discloses: (Previously Presented)
“The system according to claim 235, further including a mobile control device having a touch sensitive display.”
Donahoe teaches, (Paragraph [0076], Lines 1-7) “FIGS. 6-31 show a series of screen captures illustrating various features of an example GUI that can be implemented to facilitate user control of the previously described UAV 100. The GUI can be displayed as a graphical output via an interactive display device 402 (e.g., a touch-sensitive display) of a computing device such as mobile device 104 (e.g., a user's tablet or smartphone).”
Claims 230-232 are rejected under 35 U.S.C. 103 as being unpatentable over Donahoe in view of Pohl, further in view of et al. (US 2022/0270277 A1, hereinafter Nielsen)
Claim 230 Discloses: (Previously Presented)
“The method according to claim 228, including stitching the selected image data using an image stitching algorithm and based thereon generating and displaying in the live- view the view of the physical environment in the desired viewing direction.”
Donahoe does not explicitly teach utilizing image stitching. However, Donahoe does teach the following.
Donahoe teaches, (Paragraph [0159], Lines 7-12) “using computer vision processing, a system in accordance with the present teaching, can search for dense correspondence between images with overlapping FOV (e.g., images taken during sequential time steps and/or stereoscopic images taken at the same time step).”
Donahoe additionally teaches, (Paragraph [0045], Lines 7-10) “The perception inputs 306 may include images received from one or more image capture devices 114/115, results of processing such images (e.g., disparity images, depth values, semantic data, etc.)”
Pohl does not explicitly teach image stitching.
Nielsen does explicitly teach image stitching.
Nielsen teaches, (Paragraph [0011]) “FIG. 6 is a block diagram illustrating an example UAV control system of a UAV such as a drone.”
Nielsen additionally teaches, (Abstract) “A method and system for creating a point cloud are disclosed. A first image is captured by a first camera sensor and a second image is captured by a second camera sensor. The first and the second image have an area of overlap … Based on the area of overlap, the first and the second image are stitched to create a composite stitched image. In one aspect, depth information from the area of overlap is extracted based on the predetermined location and a point cloud is created from otherwise to be discarded image data in the area of overlap.”
Nielsen additionally teaches, (Paragraph [0065], Lines 17-21) “The output components 852 may include visual components (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)).”
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed inventio to combine the UAV system capable of capturing and processing images from multiple cameras and displaying the end result to a user on a display, and the virtual gimbal of Pohl, with the explicit image stitching technique described in Nielsen, in order to yield predictable results.
The rationale for combining with Nielsen would be to achieve the wider views and/or increased resolution pertaining to stitched images with depth information. As Nielsen describes, (Paragraph [0034], Lines 48-51) “The greater the overlapping regions, the fuller and more encompassing is the depth information picture of the surrounding environment.”
Claim 231 Discloses: (Previously Presented)
“The method according to claim 230, including correlating directional distance information, recorded by a directional distance measuring module of the UAV by measuring the physical environment, with the selected image data such that selected image data with depth information is generated,
Donahoe teaches, (Paragraph [0045], Lines 7-10) “The perception inputs 306 may include images received from one or more image capture devices 114/115, results of processing such images (e.g., disparity images, depth values, semantic data, etc.)”
“wherein the image stitching algorithm is stitching the selected image data based on the depth information.”
Donahoe does not explicitly teach utilizing image stitching. However, Donahoe does teach the following.
Donahoe teaches, (Paragraph [0159], Lines 7-12) “using computer vision processing, a system in accordance with the present teaching, can search for dense correspondence between images with overlapping FOV (e.g., images taken during sequential time steps and/or stereoscopic images taken at the same time step).”
Pohl does not explicitly teach image stitching.
Nielsen does explicitly teach image stitching.
Nielsen teaches, (Paragraph [0011]) “FIG. 6 is a block diagram illustrating an example UAV control system of a UAV such as a drone.”
Nielsen additionally teaches, (Abstract) “A method and system for creating a point cloud are disclosed. A first image is captured by a first camera sensor and a second image is captured by a second camera sensor. The first and the second image have an area of overlap … Based on the area of overlap, the first and the second image are stitched to create a composite stitched image. In one aspect, depth information from the area of overlap is extracted based on the predetermined location and a point cloud is created from otherwise to be discarded image data in the area of overlap.”
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed inventio to combine the UAV system capable of capturing and processing images from multiple cameras and displaying the end result to a user on a display, and the virtual gimbal of Pohl, with the explicit image stitching technique described in Nielsen, in order to yield predictable results.
The rationale for combining with Nielsen would be to achieve the wider views and/or increased resolution pertaining to stitched images with depth information. As Nielsen describes, (Paragraph [0034], Lines 48-51) “The greater the overlapping regions, the fuller and more encompassing is the depth information picture of the surrounding environment.”
Claim 232 Discloses: (Previously Presented)
“The method according to claim 228, including correcting a parallax-offset between cameras of the camera system based on the depth information and, based thereon, generating and displaying in the live-view the view of the physical environment in the desired viewing direction.”
Donahoe does not explicitly teach correcting a parallax-offset between cameras. However, Donahoe does teach the following.
Donahoe teaches, Paragraph [0168], Lines 8-11) “each of the image capture devices 114 shown mounted to a UAV 100 in FIG. 1 may include cameras at slightly offset positions (to achieve stereoscopic capture).”
Pohl does not explicitly teach correcting a parallax offset between cameras.
Nielsen does explicitly teach correcting a parallax offset between cameras.
Nielsen teaches, (Paragraph [0029]) “The depth sensing component 240 is configured to obtain the depth information from image data of the overlap region found by the searching component 220. In one embodiment, the depth sensing component 240 may obtain depth information in parallel, before or after the image stitching component 230 has stitched the images obtained by the image obtaining component 210. Once the searching component 200 has identified the areas of overlap between the obtained two or more images, the overlap regions are used for both stitching and obtaining depth information. In one embodiment, the depth information may be obtained by performing a calculation of the parallax between the two images. The parallax means a distance existing between the two camera sensors, and the distance, i.e., the depth, from an object point to the cameras. Therefore, the imaging position of the same object point is different in the two cameras, which is known as parallax. By triangulation, the distance from an object point to the cameras is determined, according to one embodiment. Therefore, the depth information may be calculated according to the parallax after the parallax is calculated. In one embodiment, the parallax between the corresponding pixels in the overlap region of the two images directly is calculated and the depth of the overlap region according to the parallax is calculated.”
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to combine the UAV system comprising offset cameras of Donahoe, and the virtual gimbal of Pohl, with the explicit parallax offset correction taught by Nielsen, in order to yield predictable results.
The rationale for combining the references would be to mitigate image error associated with the offset of cameras during image stitching. As Nielsen describes, (Paragraph [0023], Lines 7-14) “When stitching images to create one panoramic image issues to be solved include the presence of parallax, lens distortion, scene motion, exposure differences, etc. For panoramic stitching, a reasonable amount of overlap may be, for example, at least 15-30% to overcome or compensate for lens distortion and aliasing, and other errors that can degrade a high-quality image.”
Claim 238 is rejected under 35 U.S.C. 103 as being unpatentable over Donahoe in view of Pohl, further in view of Wan et al. (US 2018/0186472 A1, hereinafter Wan).
Claim 238 Discloses: (Previously Presented)
“The system according to claim 235, the camera system including a plurality of cameras arranged peripherally at the UAV, with each camera having a field of view with a fixed orientation in relation to the UAV and directed away from the UAV, one front camera facing forward … and at least one side camera facing sideways,”
PNG
media_image1.png
466
382
media_image1.png
Greyscale
Donahoe teaches, (Paragraph [0030], Lines 1-6) “In the example depicted in FIG. 1, the image capture devices 114 and/or 115 are depicted capturing an object 102 in the physical environment that happens to be a person. In some cases, the image capture devices may be configured to capture images for display to users (e.g., as an aerial video platform).”
“wherein the cameras are arranged such that each field of view overlaps to a predefined degree at least one adjacent field of view,”
Donahoe teaches, (Paragraph [0215], Lines 32-38) “the cameras of an image capture device 4134 may be arranged such that at least two cameras are provided with overlapping FOV at multiple angles around the UAV 100, thereby allowing for stereoscopic (i.e., 3D) image/video capture and depth recovery (e.g., through computer vision algorithms) at multiple angles around UAV 100.”
“and the camera system provides an all-round view to the physical environment, and the camera system provides available image data of the plurality of cameras for generating an all-round view to the physical environment.”
Donahoe teaches, (Paragraph [0031], Lines 10-15) “the example configuration of UAV 100 depicted in FIG. 1 includes an array of multiple stereoscopic image capture devices 114 placed around a perimeter of the UAV 100 so as to provide stereoscopic image capture up to a full 360 degrees around the UAV 100.”
Donahoe and Pohl do not teach the following limitation. However, Wan does teach the following limitation.
“one top camera facing up, one bottom camera facing down,”
Wan teaches, (Paragraph [0021], Lines 1-9) “In both exemplary UAVs, the 360-degree camera system comprises a top lens 212 coupled to a top camera 213 and mounted to a top portion of the UAV body 210 and a bottom lens 214 coupled to a bottom camera 215 and mounted to a bottom portion of the UAV body 210. In one embodiment, the top lens 212 and bottom lens 214 have angles of view α and β, respectively, and the collective angle of view for these lenses is equal to or greater than 360 degrees.”
Wan additionally teaches, (Paragraph [0022], Lines 1-4) “Once images are captured with the top lens 112 and the bottom lens 114, they are then stitched together to form a composite image showing the entire 360-degree spherical space surrounding the UAV 200.”
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to combine the UAV system capable of generating an generating an all-round view, and the virtual gimbal of Pohl, with the top and bottom cameras of Wan, in order to yield predictable results.
The rationale for combining the references would be to acquire top and bottom viewing angle of the UAV which could be applied to an all-round view. As Wan describes, (Paragraph [0021], Lines 5-9) “In one embodiment, the top lens 212 and bottom lens 214 have angles of view α and β, respectively, and the collective angle of view for these lenses is equal to or greater than 360 degrees.”
Claim 245 is rejected under 35 U.S.C. 103 as being unpatentable over Donahoe in view of Nielsen.
Claim 245 Discloses: (Previously Presented)
“The method according to claim 239, wherein the sensor data enables the generation and display of a view of the physical environment of the UAV to a user, and the generation and display is based on sensor data in the form of conditioned measurement data, wherein the conditioned measurement data is used for supporting a stitching of conditioned image data.”
Donahoe teaches, (Paragraph [0174], Lines 10-16) “An array of image capture devices 114 about a perimeter of the UAV 100 can provide low-latency information about objects up to 360 degrees around the UAV 100 and can be used to compute depth using stereo vision algorithms. Conversely, the other image capture device 115 can provide more detailed images (e.g., high resolution, color, etc.) in a limited FOV.”
Donahoe additionally teaches, (Paragraph [0175], Lines 3-7) “the high-resolution color information from an image capture device 115 can be fused with depth information from the image capture devices 114 to create a 3D representation of a tracked object.”
Donahoe additionally teaches, (Paragraph [0147]) “In some embodiments, the object detection system processes received images and outputs a dense per-pixel segmentation, where each pixel is associated with a value corresponding to either an object class label (e.g., human, building, car, animal, etc.) and/or a likelihood of belonging to that object class.”
“wherein the conditioned measurement data is used for supporting a stitching of conditioned image data.”
Donahoe does not explicitly teach utilizing image stitching. However, Donahoe does teach the following.
Donahoe teaches, (Paragraph [0159], Lines 7-12) “using computer vision processing, a system in accordance with the present teaching, can search for dense correspondence between images with overlapping FOV (e.g., images taken during sequential time steps and/or stereoscopic images taken at the same time step).”
Donahoe additionally teaches, (Paragraph [0045], Lines 7-10) “The perception inputs 306 may include images received from one or more image capture devices 114/115, results of processing such images (e.g., disparity images, depth values, semantic data, etc.)”
Nielsen does explicitly teach image stitching.
Nielsen teaches, (Paragraph [0011]) “FIG. 6 is a block diagram illustrating an example UAV control system of a UAV such as a drone.”
Nielsen additionally teaches, (Abstract) “A method and system for creating a point cloud are disclosed. A first image is captured by a first camera sensor and a second image is captured by a second camera sensor. The first and the second image have an area of overlap … Based on the area of overlap, the first and the second image are stitched to create a composite stitched image. In one aspect, depth information from the area of overlap is extracted based on the predetermined location and a point cloud is created from otherwise to be discarded image data in the area of overlap.”
Nielsen additionally teaches, (Paragraph [0065], Lines 17-21) “The output components 852 may include visual components (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)).”
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed inventio to combine the UAV system capable of capturing and conditioning images from multiple cameras and displaying the end result to a user on a display, with the explicit image stitching technique described in Nielsen, in order to yield predictable results.
The rationale for combining with Nielsen would be to achieve the wider views and/or increased resolution pertaining to stitched images with depth information. As Nielsen describes, (Paragraph [0034], Lines 48-51) “The greater the overlapping regions, the fuller and more encompassing is the depth information picture of the surrounding environment.”
RELEVANT BUT NOT CITED PRIOR ART
The prior art made of record and not relied upon is considered pertinent to applicant'sdisclosure.
Frei et al (US 20200019752 A1) teaches, (Abstract) “The method involves obtaining images of a target geographical area captured by an image-collection vehicle. The capture height of the image-collection vehicle above ground is determined when the corresponding image is captured. An image position of the corresponding image within the target geographical area is determined. The homography transform is performed on the corresponding image to generate a uniform-pixel-distance image. The first pixel locations of the identified objects (120) within the uniform-pixel-distance image are determined. The reverse homography transform is performed on the corresponding first pixel location to determine a corresponding second pixel location. The positions of the identified objects within the target geographical area are determined based on the corresponding second pixel location within the corresponding image. The determined positions of the one or more identified objects are stored.”
Chiu et al. (US 20200300637 A1) teaches, (Abstract) “During GPS-denied/restricted navigation, images proximate a platform device are captured using a camera, and corresponding motion measurements of the platform device are captured using an IMU device. Features of a current frame of the images captured are extracted. Extracted features are matched and feature information between consecutive frames is tracked. The extracted features are compared to previously stored, geo-referenced visual features from a plurality of platform devices. If one of the extracted features does not match a geo-referenced visual feature, a pose is determined for the platform device using IMU measurements propagated from a previous pose and relative motion information between consecutive frames, which is determined using the tracked feature information. If at least one of the extracted features matches a geo-referenced visual feature, a pose is determined for the platform device using location information associated with the matched, geo-referenced visual feature and relative motion information between consecutive frames.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEXANDER V. GENTILE whose telephone number is (703)756-1501. The examiner can normally be reached Monday - Friday 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kito R. Robinson can be reached at (571)270-3921. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEXANDER V GENTILE/ Examiner, Art Unit 3664
/KITO R ROBINSON/ Supervisory Patent Examiner, Art Unit 3664