DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Application
Claims 1-12 have been amended. Claims 1-12 are currently pending in this application.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 8 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 8, one of ordinary skill would not understand what is being claimed when the claim, “calculate a control speed in the imaging direction based on (1) the tracking subject direction (2) a size of the tracking target” is read in light of the specification. One of ordinary skill in the art would not understand if the control speed is based on the tracking subject direction and the size of the tracking target or the control speed is based on the tracking subject direction or the size of the tracking target. Thus, the claim limitation is unclear and indefinite.
Response to Arguments
Presented arguments have been fully considered, but are rendered moot in view of new ground(s) of rejection necessitated by amendment(s) initiated by the applicant(s).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-4 and 6-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al. (Hereafter, “Zhu”) [US 11,048,277 B1] in view of NAGASAWA KAZUKI et al. (Hereafter, “Nagasawa”) [JP2020005122A].
In regards to claim 1, Zhu discloses a control device [Fig. 2] comprising: a memory storing a program; and a processor configured to, when executing the program ([Col. 4] For example, in some embodiments, the navigation system 120 and associated subsystems, may be implemented as instructions stored in memory and executable by one or more processors.), cause the control device to: calculate, in a case where a tracking target is detected from an image captured by an image capturing device ([Col. 3] In the example depicted in FIG. 1A, the image capture devices 114 and/or 115 are depicted capturing an object 102 in the physical environment that happens to be a person. [Col. 4] The UAV 100 can be configured to track one or more objects such as a human subject 102 through the physical environment based on images received via the image capture devices 114 and/or 115. [Col. 4 and 5] The tracking subsystem 140 may include one or more subsystems such as an object detection subsystem, an instance segmentation subsystem, an identity recognition subsystem, and any other subsystems (all not shown).), a tracking subject direction toward the tracking target from the image capturing device based on a position of the tracking target ([Col. 3] The image capture devices 114 may be configured to capture images for use by a visual navigation system in guiding autonomous flight by the UAV 100 and/or a tracking system for tracking other objects in the physical environment (e.g., as described with respect to FIG. 2). [Col. 5-6] the tracking system 140 may communicate with the motion planning system 130, for example, to maneuver the UAV 100 based on measured, estimated, and/or predicted positions, orientations, and/or trajectories of objects in the physical environment); perform, in a case where the tracking target is detected from the image, a first control for controlling an imaging direction of the image capturing device based on the calculated tracking subject direction ([Col. 4] The UAV 100 may be configured to automatically adjust an orientation of the image capture device 115 so as to track image capture of an object (e.g., human subject 102) as both the UAV 100 and object are in motion through the physical environment. [Col. 6] In some embodiments, the tracking system 140, operating separately or in conjunction with the motion planning system 130, is further configured to generate control commands configured to cause a mechanism to adjust an orientation of any image capture devices 114/115 relative to the body of the UAV 100 based on the tracking of one or more objects. Such a mechanism may include a mechanical gimbal or a hybrid digital-mechanical gimbal, as previously described. For example, while tracking an object in motion relative to the UAV 100, the tracking system 140 may generate control commands configured to adjust an orientation of an image capture device 115 so as to keep the tracked object centered in the field of view (FOV) of the image capture device 115 while the UAV 100 is in motion.); and perform, in a case where no tracking target is detected from the image while the first control is performed, a second control for controlling the imaging direction of the image capturing device based on a tracking subject direction calculated when the tracking target was detected.
Nagasawa discloses a control device ([0001] monitoring device) comprising: a memory storing a program ([0029] The storage unit 52 is a storage device including a hard disk drive (HDD), a read only memory (ROM), a random access memory (RAM), and various setting information, a program for operating the object detection unit 5, and the like is stored.); and a processor configured to, when executing the program, cause the control device to: calculate, in a case where a tracking target is detected from an image captured by an image capturing device ([0041] The imaging unit 6 includes a camera driving unit 61, an imaging unit 62, a communication unit 63, a storage unit 64, and a moving object detection unit 65. [0046] The moving object detection means 65 obtains a change area of the image by performing a difference process between frames of the shot image, and detects an object image moving in the shooting area based on the change area.), a tracking subject direction toward the tracking target from the image capturing device based on a position of the tracking target ([0013] The imaging unit includes a moving object detection unit that detects a moving object, and the imaging continuation processing is performed in the tracking imaging processing. The position of the moving object detected by the moving object detecting means within a predetermined range from the detection position of the moving object is estimated as the current position of the moving object to be tracked, and the shooting control is performed based on the current position. [0026] The object detection unit 5 includes an area sensor device, for example, performs a spatial scan of the monitoring area 4 at a predetermined cycle using a beam-like search signal, and reflects reflected light from an object (a person, a vehicle, or the like) on an optical path. The position of the object existing in the monitoring area 4 is detected. Further, the object detection unit 5 can calculate the moving speed and the moving direction of the object using the detection results of the positions at a plurality of times.); perform, in a case where the tracking target is detected from the image, a first control for controlling an imaging direction of the image capturing device based on the calculated tracking subject direction ([0040] The imaging unit 6 can change the shooting direction in order to track and shoot the moving object. Further, the imaging unit 6 can perform zoom control to obtain an object image having a sufficient resolution for an object or the like that is far from the monitoring device 2. That is, the imaging unit 6 of the present embodiment is configured using a camera (PTZ camera) capable of controlling pan, tilt, and zoom (PTZ), and is a control unit based on the tracking information of the moving object detected by the object detection unit 5. [0052] The camera control unit 73 generates a signal for giving a control instruction (a control amount of the PTZ or its speed) to the imaging unit 6 based on the detection position and the moving speed obtained from the tracking information of the object. [0063] Now, the camera control means 73 performs photographing control according to the position of the moving object. That is, the camera control unit 73 controls the shooting direction and zoom control. In the control of the photographing direction (pan, tilt), the object is basically taken at the center of the angle of view, and in the zoom control, the object is taken in a predetermined size.); and perform, in a case where no tracking target is detected from the image while the first control is performed, a second control for controlling the imaging direction of the image capturing device based on a tracking subject direction calculated when the tracking target was detected ([0007] When the moving object is no longer detected in the monitoring area, the tracking control is performed based on the current position of the moving object estimated from the last detected position. Then, an imaging continuation process for continuing tracking imaging of the moving object is executed. [0064] When the moving object is no longer detected by the object detection unit 5 in the monitoring area 4, the camera control unit 73 estimates the current position of the moving object (disappearing object) from the last detected position (disappearing position). In addition, instead of the position detected by the object detection unit 5, the continuation processing for performing the above-described photographic control based on the estimated current position (estimated position) is performed.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu with the teachings of Nagasawa in order to improve the continuous tracking of an object in a monitoring area [See Nagasawa].
In regards to claim 2, the limitations of claim 1 have been addressed. Zhu discloses wherein the tracking subject direction is calculated, in a case where the tracking target is detected from the image, further based on a current imaging direction of the image capturing device, and a zoom state of the image capturing device ([Col. 3] The image capture devices 114 may be configured to capture images for use by a visual navigation system in guiding autonomous flight by the UAV 100 and/or a tracking system for tracking other objects in the physical environment (e.g., as described with respect to FIG. 2). [Col. 5-6] the tracking system 140 may communicate with the motion planning system 130, for example, to maneuver the UAV 100 based on measured, estimated, and/or predicted positions, orientations, and/or trajectories of objects in the physical environment).
Nagasawa discloses wherein the tracking subject direction is calculated, in a case where the tracking target is detected from the image, further based on a current imaging direction of the image capturing device, and a zoom state of the image capturing device ([0063] Now, the camera control means 73 performs photographing control according to the position of the moving object. That is, the camera control unit 73 controls the shooting direction and zoom control. In the control of the photographing direction (pan, tilt), the object is basically taken at the center of the angle of view, and in the zoom control, the object is taken in a predetermined size. The zoom control is performed based on the position of the detected object (distance to the object). Basically, when the object moves away from the imaging unit 6, the zoom magnification is increased to control the object to zoom in. On the other hand, when the object approaches the imaging unit 6, the zoom control is performed. Control is performed to reduce the zoom magnification and zoom out. In the zoom control, the moving speed of the object can be considered. More specifically, the higher the speed of the object, the more zoom out the image can be taken.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu with the teachings of Nagasawa in order to improve the continuous tracking of an object in a monitoring area [See Nagasawa].
In regards to claim 3, the limitations of claim 1 have been addressed. Zhu fails to explicitly disclose wherein the processor is further configured to cause the control device to, in a case where the tracking target is detected from the image, calculate a control speed in the imaging direction based on the tracking subject direction.
Nagasawa discloses wherein the processor is further configured to cause the control device to, in a case where the tracking target is detected from the image, calculate a control speed in the imaging direction based on the tracking subject direction ([0052] The camera control unit 73 generates a signal for giving a control instruction (a control amount of the PTZ or its speed) to the imaging unit 6 based on the detection position and the moving speed obtained from the tracking information of the object.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu with the speed control instruction for the imaging unit based on the tracking information of the object as taught by Nagasawa in order to improve the continuous tracking of an object in a monitoring area [See Nagasawa].
In regards to claim 4, the limitations of claim 1 have been addressed. Zhu discloses wherein the processor is further configured to cause the control device to, in a case where the tracking target is detected from the image, calculate a control speed of a zoom of the image capturing device based on a size of the tracking target.
Nagasawa discloses wherein the processor is further configured to cause the control device to, in a case where the tracking target is detected from the image, calculate a control speed of a zoom of the image capturing device based on a size of the tracking target ([0052] The camera control unit 73 generates a signal for giving a control instruction (a control amount of the PTZ or its speed) to the imaging unit 6 based on the detection position and the moving speed obtained from the tracking information of the object. [0063] Now, the camera control means 73 performs photographing control according to the position of the moving object. That is, the camera control unit 73 controls the shooting direction and zoom control. In the control of the photographing direction (pan, tilt), the object is basically taken at the center of the angle of view, and in the zoom control, the object is taken in a predetermined size. The zoom control is performed based on the position of the detected object (distance to the object). Basically, when the object moves away from the imaging unit 6, the zoom magnification is increased to control the object to zoom in. On the other hand, when the object approaches the imaging unit 6, the zoom control is performed.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu with the speed control instruction for the PZT based on the size information of the object as taught by Nagasawa in order to improve the continuous tracking of an object in a monitoring area [See Nagasawa].
In regards to claim 6, the limitations of claim 1 have been addressed. Zhu fails to explicitly disclose wherein the processor is further configured to cause the control device to, in a case where no tracking target is detected from the image while the first control is performed, calculate a control speed in the imaging direction based on the tracking subject direction calculated when the tracking target was detected.
Nagasawa discloses wherein the processor is further configured to cause the control device to, in a case where no tracking target is detected from the image while the first control is performed, calculate a control speed in the imaging direction based on the tracking subject direction calculated when the tracking target was detected ([0052] The camera control unit 73 generates a signal for giving a control instruction (a control amount of the PTZ or its speed) to the imaging unit 6 based on the detection position and the moving speed obtained from the tracking information of the object.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu with the speed control instruction for the imaging unit based on the tracking information of the object as taught by Nagasawa in order to improve the continuous tracking of an object in a monitoring area [See Nagasawa].
In regards to claim 7, the limitations of claim 1 have been addressed. Zhu fails to explicitly disclose wherein the processor is further configured to cause the control device to, in a case where no tracking target is detected from the image while the first control is performed, calculate a control speed in the imaging direction based on a tracking subject direction predicted based on the tracking subject direction calculated when the tracking target was detected.
Nagasawa discloses wherein the processor is further configured to cause the control device to, in a case where no tracking target is detected from the image while the first control is performed, calculate a control speed in the imaging direction based on a tracking subject direction predicted based on the tracking subject direction calculated when the tracking target was detected ([0052] The camera control unit 73 generates a signal for giving a control instruction (a control amount of the PTZ or its speed) to the imaging unit 6 based on the detection position and the moving speed obtained from the tracking information of the object.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu with the speed control instruction for the imaging unit based on the tracking information of the object as taught by Nagasawa in order to improve the continuous tracking of an object in a monitoring area [See Nagasawa].
In regards to claim 8, the limitations of claim 1 have been addressed. Zhu fails to explicitly disclose wherein the processor is further configured to cause the control device to, in a case where no tracking target is detected from the image while the first control is performed, calculate a control speed in the imaging direction based on (1) the tracking subject direction (2) a size of the tracking target.
Nagasawa discloses wherein the processor is further configured to cause the control device to, in a case where no tracking target is detected from the image while the first control is performed, calculate a control speed in the imaging direction based on (1) the tracking subject direction (2) a size of the tracking target ([0052] The camera control unit 73 generates a signal for giving a control instruction (a control amount of the PTZ or its speed) to the imaging unit 6 based on the detection position and the moving speed obtained from the tracking information of the object. [0063] Now, the camera control means 73 performs photographing control according to the position of the moving object. That is, the camera control unit 73 controls the shooting direction and zoom control. In the control of the photographing direction (pan, tilt), the object is basically taken at the center of the angle of view, and in the zoom control, the object is taken in a predetermined size. The zoom control is performed based on the position of the detected object (distance to the object). Basically, when the object moves away from the imaging unit 6, the zoom magnification is increased to control the object to zoom in. On the other hand, when the object approaches the imaging unit 6, the zoom control is performed.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu with the speed control instruction for the imaging unit based on the tracking information of the object as taught by Nagasawa in order to improve the continuous tracking of an object in a monitoring area [See Nagasawa].
In regards to claim 9, the limitations of claim 1 have been addressed. Zhu discloses wherein the processor is further configured to cause the control device to: acquire the image ([Col. 3] The image capture devices 114 and/or 115 are depicted capturing an object 102 in the physical environment. The image capture devices 114 may be configured to capture images for use by a visual navigation system in guiding autonomous flight by the UAV 100 and/or a tracking system for tracking other objects in the physical environment (e.g., as described with respect to FIG. 2).); generate a command to change the imaging direction at the controlled control speed; and transmit the generated command to the image capture device ([Col. 4] The UAV 100 may be configured to automatically adjust an orientation of the image capture device 115 so as to track image capture of an object (e.g., human subject 102) as both the UAV 100 and object are in motion through the physical environment. [Col. 6] In some embodiments, the tracking system 140, operating separately or in conjunction with the motion planning system 130, is further configured to generate control commands configured to cause a mechanism to adjust an orientation of any image capture devices 114/115 relative to the body of the UAV 100 based on the tracking of one or more objects. Such a mechanism may include a mechanical gimbal or a hybrid digital-mechanical gimbal, as previously described. For example, while tracking an object in motion relative to the UAV 100, the tracking system 140 may generate control commands configured to adjust an orientation of an image capture device 115 so as to keep the tracked object centered in the field of view (FOV) of the image capture device 115 while the UAV 100 is in motion.).
Nagasawa discloses wherein the processor is further configured to cause the control device to: acquire the image ([0022] The imaging unit 6 performs camera control in accordance with the camera control command from the control unit 7, and captures an image while following the movement of the moving object in the monitoring area 4.); generate a command to change the imaging direction at the controlled control speed; and transmit the generated command to the image capture device ([0052] The camera control unit 73 generates a signal for giving a control instruction (a control amount of the PTZ or its speed) to the imaging unit 6 based on the detection position and the moving speed obtained from the tracking information of the object.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu with the teachings of Nagasawa in order to improve the continuous tracking of an object in a monitoring area [See Nagasawa].
In regards to claim 10, the limitations of claim 1 have been addressed. Zhu discloses further comprising: a drive unit configured to change the imaging direction [Col. 6] In some embodiments, the tracking system 140, operating separately or in conjunction with the motion planning system 130, is further configured to generate control commands configured to cause a mechanism to adjust an orientation of any image capture devices 114/115 relative to the body of the UAV 100 based on the tracking of one or more objects. Such a mechanism may include a mechanical gimbal or a hybrid digital-mechanical gimbal, as previously described.).
Nagasawa discloses further comprising: a drive unit configured to change the imaging direction ([0042] The camera driving means 61 includes a driving mechanism for changing the photographing direction (pan, tilt) of the camera and a driving mechanism for driving the zoom lens of the camera to change the angle of view. Operate the drive mechanism to perform pan / tilt control and zoom control.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu with the teachings of Nagasawa in order to improve the continuous tracking of an object in a monitoring area [See Nagasawa].
Claim 11 lists all the same elements of claim 1, but in method form rather than device form. Therefore, the supporting rationale of the rejection to claim 1 applies equally as well to claim 11.
Claim 12 lists all the same elements of claim 1, but in non-transitory computer-readable storage medium form rather than device form. Therefore, the supporting rationale of the rejection to claim 1 applies equally as well to claim 12.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhu in view of Nagasawa in further view of Stanard et al. (Hereafter, “Stanard”) [US 10,917,557 B2].
In regards to claim 5, the limitations of claim 1 have been addressed. Zhu fails to explicitly disclose wherein the processor is further configured to cause the control device to, in a case where no tracking target is detected from the image while the first control is performed, change the imaging direction in response to a period of time in which no tracking target is detected after a control speed in the imaging direction has been zero lasting for a predetermined period of time.
Nagasawa discloses wherein the processor is further configured to cause the control device to, in a case where no tracking target is detected from the image while the first control is performed, change the imaging direction in response to a period of time in which no tracking target is detected after a control speed in the imaging direction has been zero lasting for a predetermined period of time ([0065] When the disappearance position is within the outer edge area 100 in the monitoring area 4 shown in FIG. 3, that is, when the moving object to be tracked is no longer detected in a portion near the boundary in the monitoring area 4, the tracking object is It is highly probable that it has moved out of the monitoring area 4. Therefore, in this case, the camera control unit 73 may not perform the photographing continuation process. For example, the camera control unit 73 may maintain the photographing control at the vanishing position for a predetermined time and wait for the return of the lost object. The direction and zoom magnification are set to the home position stored in the storage means 64.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu with the determination to return the camera to the home position after a predetermined time after the disappearance of the tracking object as taught by Nagasawa in order to improve the continuous tracking of an object in a monitoring area [See Nagasawa].
Stanard discloses wherein the processor is further configured to cause the control device to, in a case where no tracking target is detected from the image while the first control is performed, change the imaging direction in response to a period of time in which no tracking target is detected after a control speed in the imaging direction has been zero lasting for a predetermined period of time ([Col. 18] An occlusion that happens in a shorter duration can still be tracked, but, but as the time grows longer, the object is more considered a lost track. The characterization of an occlusion situation is shown in the TABLE 6 below. FIG. 7 is a video depiction of an image 700 of a full occlusion of the object by trees. The feature and motion confidence dips below the threshold and cannot be detected. [Col. 19] Lost Track: The lost track mode is diagnosed when the features and motion models of the object are unable to find the object after some time. Both confidence values will be low. What differs from the occlusion mode is the duration of the lost track. This threshold between occlusion and lost track can be determined by the user as a temporal threshold. In some cases, even though the tracker may lose the features and motion of the object, the tracker has not permanently lost the track of the object and may still reacquire it. The characterization of the mode is shown in the TABLE 7 below.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu with the teachings of Stanard in order to improve the accuracy of the moving object tracking system [See Stanard].
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Kaitlin A Retallick whose telephone number is (571)270-3841. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chris Kelley can be reached at (571) 272-7331. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KAITLIN A RETALLICK/Primary Examiner, Art Unit 2482
Prosecution Insights