Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20230161338 A1 to Shah et al. (“Shah”) and US 20190250601 A1 to Donahue et al. (“Donahue”).
As to claim 1, Shah teaches a video processing device, comprising: at least one memory and at least one processor, wherein: the at least one memory stores executable instruction; and when executing the executable instruction stored in the at least one memory, the at least one processor is configured to: acquire video image data collected by an image acquirer of a flying object and a flight speed set of the flying object (¶0038, alter the position, direction, speed, and/or orientation of the drone as it flies the computed spline. These changes may cause the drone to depart from the computed spline, or change direction as it travels along the spline. The pilot may cause the drone to speed up or slow down along segments of the spline or at points along the spline. The pilot may also modify the operation of a camera, such as zooming in or out or making adjustments to the exposure as the drone flies the spline. In other example, the pilot may cause the drone to reverse course along the computed spline or to “snap-to” a new position on the spline without traveling along the spline. The pilot may also cause the drone to stop and hover at point on the computed spline, for example, to add a keyframe at that location. Any or all of these modifications may be saved for subsequent use with the spline or as a new version of the spline. In some implementations, the pilot may control the orientation of the drone along the spine and/or the gimbal position as the drone travels along the computed spine, i.e., “Free Look Mode.” In this manner, the pilot can focus on camera angles and positions without having to actively pilot the drone.); determine at least one adjustment speed (¶0051, The UI may include virtual or physical control mechanisms such as rocker switches or sliders to adjust the drone speed along a segment of the flight path or at a point on the flight path); determine a playback rate of the video image data based on the flight speed set and the at least one adjustment speed (¶0100, FIGS. 13A-13D, the UI of the autonomous flight control application displays playback track 1320, which is linear graphical representation of the computed spline. On playback track 1320, keyframes are represented as diamonds 1322, and the current position of the drone along the spline is also shown as arrowhead 1324. The relative distance between the keyframes is indicated by the proportional spacing of diamonds 1322 on playback track 1320. FIG. 13B illustrates the addition of virtual speed control slider 1330 to the touchscreen display by which the UI receives manual input(s) causing the application to speed up, slow down, or hover the drone as it traverses computed spline 1260. Using speed control slider 1330, the pilot can control drone speed between keyframes or across the entire spline. Note that in this illustration, arrowhead 1324 changes color to indicate when the drone is in motion. In an implementation, the relative time to travel between keyframes is indicated by the proportional spacing of diamonds 1322 on playback track 1320); and play the video image data at the playback rate (¶0049, the user interface on the remote control displays a linear playback track or timeline representation of a predetermined flight plan. The linear timeline representation may include keyframes identified by a distinctive shape, such as a diamond, on the timeline. Distances along the timeline may be proportional to actual flight distances. As the drone flies a predetermined flight path, the timeline may indicate the drone's progress along the flight path using one color to show the completed portion and a second color to show the portion remaining; optionally, an arrow or other symbol may travel along the timeline as the drone flies to show in real time the drone's travel along the flight path). Donahue teaches the flight speed and flight pattern (¶0024, allows the user to input high-level commands that are interpreted and translated into complex control commands that guide the UAV's flight. The environment sensing system provides information on the surrounding environment, particularly where safe areas of surrounding space to fly to are and where areas are that are dangerous, for example, by being occupied by an object or by lacking information on the area. The information from the sensing system is combined with the user's commands by the motion planning system. In certain embodiments, the user's commands can be very general, such as a command to follow a particular person or object as it moves, or very specific, such as to go up or down. The motion planning system generates a path or planned trajectory based on the environment sensing system's data combined with the user's commands. The user can provide input via a GUI that is presented at a digital device such as a smartphone or tablet, on a controller, or on any other type of device onboard the aircraft or remotely located from the aircraft). In view of the teachings of Donahue, it would have been obvious before the effective filing date of the invention to modify the teachings of Shah. The suggestion/motivation would be effectively capturing video or other data during a flight.
As to claim 2, Shah and Donahue teaches the video processing device according to claim 1, wherein the determining at least one adjustment speed comprises: acquiring a flight altitude set of the flying object; and determining the at least one adjustment speed based on the flying altitude set, wherein the video image data, a flight speed in the flight speed set and the at least one adjustment speed have a corresponding relationship in time (¶0026, a default motion of the UAV can take place in a plane parallel with, but above the ground (i.e., an XY plane), mimicking the behavior of ground vehicles, but in the air. A separate slider element presented in the GUI may allow a user to provide basic inputs to control the altitude. In such an embodiment, the user can use simple touch gestures, for example, input using a single finger to fly the aircraft around in the XY plane at a particular altitude off the ground. The user can then use other touch gestures to control altitude, when necessary. The GUI and associated motion planning systems may utilize data from onboard sensors to prevent the user from steering the aircraft into detected obstacles.).
As to claim 3, Shah and Donahue teaches the video processing device according to claim 2, wherein the determining the at least one adjustment speed based on the flight altitude set comprises: determining a speed corresponding to each flight altitude in the flight altitude set to obtain the at least one adjustment speed according to a preset relationship between flight altitudes and speeds, wherein in the preset relationship between flight altitudes and speeds, the flight altitudes are positively correlated with the speeds (¶0095, camera view 1210 from an onboard camera. The UI displays text display 1203 to indicate the current mode: “KeyFrame Mode.” Flight parameter set 1202 is a graphic in the upper left corner of the touchscreen displaying drone flight speed, drone distance from the dock, drone elevation, and gimbal angle of the camera relative to level flight. At the bottom center of the touchscreen, the UI displays virtual buttons by which the pilot can define keyframes to be used in generating the spline: Add button 1251 causes the application to add a keyframe at the drone's current location, Undo button 1250 reverses the action triggered by Add button 1251 (i.e., undoes adding the most recently added keyframe), and Done button 1252 terminates the addition of keyframes. At the upper right corner of the touchscreen is graphic 1204 for pausing KeyFrame Mode so the pilot can stop autonomous flight and take manual control. Note that camera view 1210 is darkened to enhance the visibility of text display).
As to claim 4, Shah and Donahue teaches the video processing device according to claim 1, wherein the determining at least one adjustment speed comprises: determining an image acquirer movement mode set when the image acquirer acquires the video image data; and determining the at least one adjustment speed based on the image acquirer movement mode set (Fig. 15, ¶0109, a sequence of images illustrating yet another implementation of the UI of an autonomous flight control application of drone operating in KeyFrame Mode during playback. Images 1510-1530 illustrate the display on a drone remote control. In image 1510, the display shows a first-person camera view is a live-feed from an onboard camera. At the bottom of image 1510, the drone arrow indicator is traversing the playback track traveling from right to left and shows the drone just as it approaches keyframe 4. As the drone approaches keyframe 4, on the AR overlay of the computed spline, a translucent diamond marking keyframe 4's location dynamically grows in size as the drone approaches it, then disappears (in image 1520) to simulate the drone passing through the AR keyframe diamond. Next, image 1530 shows the first-person view of the drone as it continues on the computed spline but with the drone pivoting starboard to track the paddleboarder. Having pivoted away from a forward-facing orientation, the AR representation of the computed spline is no longer visible, ostensibly because it is out of camera view to the left of the screen).
As to claim 5, Shah and Donahue teaches the video processing device according to claim 4, wherein the determining the at least one adjustment speed based on the image acquirer movement mode set comprises: determining at least one preset speed corresponding to each image acquirer movement mode in the image acquirer movement mode set according to a relationship between preset image acquirer movement modes and speeds, and obtaining the at least one adjustment speed based on the at least one preset speed (¶0094, Fig. 12B).
As to claim 6, Shah and Donahue teaches the video processing device according to claim 1, wherein the determining at least one adjustment speed comprises: detecting a user operation to adjust the adjustment speed; and determining the at least one adjustment speed based on the user operation (¶0103, UI of the autonomous flight control application displays a number of flight and operational commands which cause the application to: adjust the speed of the drone's travel along the spline; stop or reverse the drone along the spline; jump or “snap” to a keyframe out of order on the spline; add new keyframes, delete keyframes; or edit speed or orientation settings at any keyframe. Note that on the AR representation of computed spline 1260, the size of each of keyframe markers 1262 grows larger as the drone approaches the keyframe location. The UI displays computed spline 1260 in a color which is highly visible from the background (first-person) view on the touchscreen. This color can be programmatically chosen using an algorithm which detects the range of colors of camera view 1210, or the color may be set manually the pilot).
As to claim 7, Shah and Donahue teaches the video processing device according to claim 6, wherein the detecting the user operation to adjust the adjustment speed comprises: displaying a virtual button for setting the adjustment speed in a display area corresponding to the video processor; and detecting a touch pressing track of a touch object on the virtual button (¶0106, virtual button).
As to claim 8, Shah and Donahue teaches the video processing device according to claim 6, wherein the detecting the user operation to adjust the adjustment speed comprises: displaying, in a display area corresponding to the video processing device, a flight speed curve of the flying object corresponding to the flight speed set; and detecting a touch operation by selecting a point on the flight speed curve in the touch area (¶0078).
As to claim 9, Shah and Donahue teaches the video processing device according to claim 1, wherein the determining the playback rate of the video image data based on the flight speed set and the at least one adjustment speed comprises: determining an analysis speed corresponding to each moment based on the flight speed set; and determining a playback rate corresponding to each moment based on the analysis speed corresponding to each moment and an adjustment speed corresponding to the moment among the at least one adjustment speed (¶0103).
As to claim 10 , Shah and Donahue teaches the video processing device according to claim 9, wherein the playing the video image data at the playing rate comprises: playing the video image data at the playing rate corresponding to each moment (¶0100, On playback track 1320, keyframes are represented as diamonds 1322, and the current position of the drone along the spline is also shown as arrowhead 1324. The relative distance between the keyframes is indicated by the proportional spacing of diamonds 1322 on playback track 1320. FIG. 13B illustrates the addition of virtual speed control slider 1330 to the touchscreen display by which the UI receives manual input(s) causing the application to speed up, slow down, or hover the drone as it traverses computed spline 1260. Using speed control slider 1330, the pilot can control drone speed between keyframes or across the entire spline. Note that in this illustration, arrowhead 1324 changes color to indicate when the drone is in motion. In an implementation, the relative time to travel between keyframes is indicated by the proportional spacing of diamonds 1322 on playback track 1320).
As to claim 11 , Shah and Donahue teaches the video processing device according to claim 9, wherein the determining the analysis speed corresponding to each moment based on the flight speed set comprises: segmenting the video image data to obtain one or more segments of sub-video image data (¶0077, FIG. 5A demonstrates an exemplary flight path 508 comprising straight line segments defined by a set of eight keyframes); obtaining a flight speed subset corresponding to each sub-video image data from the flight speed set (¶0095, one flight speed, drone distance from the dock, drone elevation, and gimbal angle of the camera relative to level flight. At the bottom center of the touchscreen, the UI displays virtual buttons by which the pilot can define keyframes to be used in generating the spline); and determining the analysis speed corresponding to each moment based on each of the flight speed subsets (¶0095, one flight speed, drone distance from the dock, drone elevation, and gimbal angle of the camera relative to level flight. At the bottom center of the touchscreen, the UI displays virtual buttons by which the pilot can define keyframes to be used in generating the spline).
As to claim 12 , Shah and Donahue teaches the video processing device according to claim 11, wherein the segmenting the video image data to obtain the one or more segments of sub-video image data comprises: segmenting the video image data according to a preset segment duration to obtain the one or more segments of sub-video image data; or segmenting the video image data according to a segmentation rule that a fluctuation range of the flight speed in a continuous time period is within a preset threshold range to obtain the one or more segments of the sub-video image data (¶0105, marking the locations of keyframes in proportion to their distances along the spline or in proportion to the time to travel between the keyframes).
As to claim 13 , Shah and Donahue teaches the video processing device according to claim 12, wherein the determining the playback rate corresponding to each moment based on the analysis speed corresponding to each moment and the adjustment speed corresponding to the moment among the at least one adjustment speed comprises: calculating a ratio of the adjustment speed corresponding to each moment to the analysis speed corresponding to the moment; and determining the playback rate corresponding to each moment based on the ratio corresponding to each moment (¶0052, a virtual slider controlling drone speed is displayed in multiple colors to show multiple zones of dynamic feasibility. Starting at the slower end of the slider range, green may indicate the range of speeds which are dynamically feasible for the drone to fly; yellow may indicate the range of speeds pushing the operating envelope of the drone; and red may indicate the range of speeds which are not dynamically feasible for the drone to fly).
As to claim 14 , Shah and Donahue teaches the video processing device according to claim 11, wherein the determining the analysis speed corresponding to each moment based on each of the flight speed subsets comprises: determining a nth flight speed from each of the flight speed subsets as the analysis speed, wherein n is less than or equal to a total number of elements included in each of the flight speed subsets; or calculating a speed average of the flight speeds included in each of the flight speed subsets and determining the speed average as the analysis speed (¶0052, drone speed is displayed in multiple colors to show multiple zones of dynamic feasibility. Starting at the slower end of the slider range, green may indicate the range of speeds which are dynamically feasible for the drone to fly; yellow may indicate the range of speeds pushing the operating envelope of the drone; and red may indicate the range of speeds which are not dynamically feasible for the drone to fly. For example, where a predetermined flight path indicates a turn, the red portion of the slider would correspond to the speeds at which the drone would be unable to navigate the turn without flying off the flight path.).
As to claim 15, Shah and Donahue teaches the video processing device according to claim 13, wherein the determining the corresponding playback rate based on the ratio corresponding to each moment comprises: if the ratio corresponding to each moment is greater than 1, subjecting the image frame data at the corresponding moment to frame skipping playback processing according to the ratio to obtain the playback rate corresponding to the ratio; and if the ratio corresponding to each moment is less than 1, subjecting the image frame data at the corresponding moment to frame supplementation according to the ratio to obtain the playback rate corresponding to the ratio (Donahue, ¶0080, the UAV 100 may remain at a particular altitude (i.e., within a particular XY plane parallel to the ground plane) when responding to the pan/tilt input. For example, depending on a currently selected control mode, the GUI module 404 may interpret a substantially lateral dragging motion or gesture as a pan/tilt command regardless of whether the user's dragging motion is perfectly level. Based on this interpretation, a pan and/or tilt objective may be generated that causes the UAV 100 to either rotate or move in the XY plane while maintaining a constant altitude. In some embodiments, vertical motion in the user's finger 610 may result in a gimbaled image capture device 115 panning or tilting up or down while the UAV 100 remains at a constant altitude).
As to claim 16, Shah and Donahue teaches the video processing device according to claim 1, wherein the video processing device is at least partially integrated onto the flying object or the video processing device is a terminal independent from the flying object (Fig. 1).
As to claim 17, Shah and Donahue teaches the video processing device according to claim 1, wherein the flying object is an unmanned aerial vehicle, and the image acquirer is a camera or a lens module with an image acquisition function (¶0035).
As to claim 18, see the rejection of claim 1.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTINE A KURIEN whose telephone number is (571)270-5694. The examiner can normally be reached M-F; 7:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Nathan Flynn can be reached at 571-272-1915. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHRISTINE A KURIEN/Examiner, Art Unit 2421 /NATHAN J FLYNN/Supervisory Patent Examiner, Art Unit 2421
Prosecution Insights