DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. JP 2021-185014, filed on 11/12/2021.
Amendment
Applicant submitted amendments on 05/14/2026. The Examiner acknowledges the amendment and has reviewed the claims accordingly.
Overview
Claims 1-13 are cancelled.
Claims 14-20 are pending.
Claims 14-20 are rejected.
Applicant’s Arguments
In regards to Argument 1, Applicant/s state/s “the rejection to independent Claim 14 under 35 U.S.C. 102(a)(2) based on Zhang, as discussed during the interview, Zhang fails to disclose or suggest Applicant's specifically claimed movable target having a cooling/heating-generator and claimed performing the recited operations. Reconsideration and withdrawal of the rejection under 35 U.S.C. 102(a)(2) based on Zhang is therefore respectfully requested.”
Examiner’s Responses
In response to Argument 1, see remarks filed 05/14/2026, regarding Applicant’s argument that “Zhang fails to disclose or suggest Applicant’s specifically claimed movable target having a heating/cooling-generator and claimed performing the recited operations” has been considered and is persuasive. Therefore, the rejection has been withdrawn due to the amendment. However, upon further consideration, a new ground(s) of rejection is made for Claim 1 under 35 U.S.C 103 in view of Zhang (U.S. Patent Pub No. 2020/0043184 A1, hereafter referred to as Zhang) in view of Ragucci et al. (U.S. Patent Pub. No. 2009/0321636 A1, hereafter referred to as Ragucci) in further view of Lei (U.S. Patent Pub. No. 2022/0191389 A1, hereafter referred to as Lei).
The Examiner finds that Ragucci reads on the claim limitation “regarding a target having a cooling/heating-generator and movable relative to an aerial drone.”
Specifically, regarding the limitation, “regarding a target having a cooling/heating-generator and movable relative to an aerial drone,” Ragucci teaches a method for aerial searching for a thermal target in a search area, such as searching for a single person in open water (Abstract). The Examiner interprets a person in open water to be “movable” relative to the aerial drone since a person is capable of swimming and/or moving around in the water. Under Broadest Reasonable Interpretation (BRI), the Examiner interprets a person in open water to be “a cooling/heating-generator” since a person (human) is capable of generating heat, such as body heat. In addition, Ragucci states “if the average temperature for a particular sector is anomalously high relative to nearby sectors, that indicates that there may be a relevant heat source, such as a human, in that sector” (Paragraph [0025]). Therefore, the Examiner interprets the person/human to be a heat source, hence, “heating/cooling generator.” The primary reference, Zhang, discloses a tracking method and device for tracking a target object through a first camera and a second camera (Abstract) mounted on a UAV (Paragraph [0007]). The UAV may obtain RGB color information based on the first image that includes the target object captured by the first camera “ordinary camera” (Paragraph [0026]). The UAV may also obtain heat distribution information of the target object based on the second image information captured by an infrared camera (Paragraph [0028]). The UAV may fuse the location information of the target object relative to the first camera and the location information of the target object relative to the second camera to obtain a fused location information of the target object relative to the first camera and the second camera. The UAV may also adjust the attitudes of the gimbals, thereby changing the attitudes of the first camera and the second camera, such that the location of the target object relative to the first camera and the second camera is best. The UAV may simultaneously track the target based on RGB information and heat distribution information through the first camera and the second camera. (Paragraph [0033]). In addition, the UAV may track the target object by fusing different advantageous characteristics of different cameras so that better tracking of the object can be achieved. Both Ragucci and Zhang are in the field of art of target object tracking. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Zhang by replacing the general “target object” taught by Zhang with a “thermal target” such as a human “heat source” that is taught by Ragucci, to make the invention that is capable of identifying thermal targets for search and rescue operations such as open water searches (Ragucci, Paragraph [0006]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (U.S. Patent Pub No. 2020/0043184 A1, hereafter referred to as Zhang) in view of Ragucci et al. (U.S. Patent Pub. No. 2009/0321636 A1, hereafter referred to as Ragucci) in further view of Lei (U.S. Patent Pub. No. 2022/0191389 A1, hereafter referred to as Lei).
Regarding Claim 14, Zhang teaches an information processing method regarding a target (Paragraph [0148], Zhang teaches a target object tracking method realized through a computer software program instructing related hardware.) (Paragraphs [0007], [0026], [0028], Fig. 1, Zhang teaches an Unmanned Aerial Vehicle (UAV) which includes a first camera and a second camera. The first camera may be an “ordinary camera.” The ordinary camera may obtain RGB color information of the target object. The second camera may be an infrared camera configured to obtain heat distribution information of the target object.),
PNG
media_image1.png
315
402
media_image1.png
Greyscale
the method comprising; providing a command to start automatically following the target by a shooting region of visible image capture for the RGB camera (Paragraph [0026], Zhang teaches when the UAV receives a control command (or in response to receiving a control command), the UAV may control the first camera (ordinary/RGB camera) based on an instruction included in the control command to acquire first image information that includes the target object, thereby obtaining the first characteristic information of the target object based on the first image information that includes the target object. The first characteristic information may include color information (RGB information). The first camera may be an “ordinary camera.” The ordinary camera may obtain the RGB color information of the target object.); controlling the automatic following with respect to the target in the shooting region of the RGB camera for the visible image (Paragraphs [0034], [0045], Zhang teaches the UAV may first track the target object based on the first characteristic information (RGB information) through the first camera (RGB camera). The UAV may control the first camera (RGB camera) based on an instruction included in the control command to acquire first image information that includes the target object.), including setting the shooting region of the RGB camera (Paragraphs [0049], [0026], Zhang teaches the UAV may adjust the attitudes of the first gimbal and the second gimbal, thereby changing attitude information of the first camera and attitude information of the second camera. The first camera may be an ordinary camera. The ordinary camera may obtain the RGB color information of the target object.) so as to cause the target (Paragraphs [0048-50], Fig. 3, Zhang teaches the UAV may fuse the location information of the target object relative to the first camera (RGB/ordinary camera) and the location information of the target object relative to the second camera (infrared camera) to obtain fused location information of the target object relative to the first camera and the second camera. The fused location information of the target object relative to the first camera and the second camera indicates a best location of the target object relative to the first camera and the second camera. For example, the UAV may fuse the RGB color information and the heat distribution information for tracking the target object.) and remain in the shooting region of the RGB at all times during the automatic following (Paragraphs [0039-40], [0033], Zhang teaches the first camera (RGB camera) and second camera (IR camera) may maintain a linked mode. The first camera and second camera may maintain parallel perspectives. Attitude of the second gimbal may change correspondingly as the attitude of the first gimbal changes. Therefore, the attitude of the second gimbal is consistent with the attitude of the first camera. The UAV may simultaneously track the target object based on first characteristic information (color/RGB information) and the second characteristic information (heat distribution information) through the first camera and the second camera.), the thermographic image data being data obtained by shooting an image in response to the command to start the automatic following (Paragraph [0028], Zhang teaches the UAV may obtain second characteristic information (heat distribution information) of the target object through the second camera mounted on the second gimbal of the UAV. For example, when the UAV receives a control command (or in response to receiving a control command), the UAV may control the second camera (IR camera) based on an instruction included in the control command to acquire second image information (heat distribution information) that includes the target object. The second camera may be an infrared camera configured to obtain the heat distribution information of the target object.), (Paragraphs [0035], [0059], Zhang teaches if the image of the other regions captured by the second camera (infrared camera) does not include the target object, then when the UAV receives the tracking switch command (or in response to receiving the tracking switch command), the UAV may first obtain location information of the target object relative to the first camera (RGB camera), and then obtain location information of the target object relative to the second camera based on the location information of the target object relative to the first camera and a characteristic parameter of the second camera. The UAV may perform tracking initialization based on location information of the target object relative to the second camera. The tracking initialization may include: adjusting attitude information of the second camera, tuning a parameter of the second camera, and controlling the second camera to observe or track the target object, such that the second camera can capture an image of the target object. Finally, the UAV may obtain the third characteristic information of the target object through the second camera, and track the target based on the third characteristic information through the second camera. The third characteristic information may include heat distribution information.), under a condition where the characteristic temperature distribution is in the shooting region of the IR camera (Paragraphs [0033], [0028], Zhang teaches the UAV may obtain location information of the target object relative to the second camera based on the second image information that includes the target object that is acquired by the second camera (infrared camera) and the obtained second characteristic information (heat distribution information) of the target object.), deriving the position corresponding to the characteristic temperature distribution (Paragraphs [0061], [0059], Zhang teaches the UAV may calculate the location information of the target object in the second camera (infrared camera) based on an azimuth angle of the target object relative to the first camera and a characteristic parameter of the second camera. The second camera (infrared) may be configured to obtain the heat distribution information of the target object. The Examiner interprets the object corresponds to the heat distribution information captured by the IR camera, and therefore, deriving the position of the target object in the second camera (infrared camera) is also deriving the position corresponding to the heat distribution.), and after setting the shooting region of the RGB camera (Paragraph [0049], Zhang teaches adjusting the attitude of the first camera (RGB camera), thereby changing attitude information of the first camera such that the location relative to the first and second cameras is best.), changing the shooting region of the RGB camera responsive to movement of the target (Paragraph [0051], Zhang teaches when the UAV is used to assist police in tracking a criminal suspect (moving target), the UAV may combine the advantageous characteristics of the different cameras to track the same target. For example, tracking the target is simultaneously performed using a thermal imaging camera, heat distribution information of the tracking target may be obtained through the thermal imaging camera. It then becomes easy to track the target and not easy to lose the track of the target.) determined based on change in the position corresponding to the characteristic temperature distribution included in the thermographic image data from the IR camera (Paragraph [0051], Zhang teaches using a dual gimbal two camera (RGB & infrared) UAV to assist police in tracking a criminal suspect (moving target). Tracking the target is simultaneously performed using a thermal imaging camera to obtain heat distribution information of the tracking target through the thermal imaging camera. The UAV may fuse the RGB color information and the heat distribution for tracking the target object to compensate for the disadvantages of the other camera type.)
Zhang does not explicitly disclose having a cooling/heating-generator movable relative to an aerial drone and setting the shooting region of the RGB camera so as to cause the target located in a position corresponding to a characteristic temperature distribution included in the thermographic image data to be included in the shooting region of the RGB camera and determining whether or not the characteristic temperature distribution is in a shooting region of the IR camera, under a condition where the characteristic temperature distribution is not in the shooting region of the IR camera, changing the shooting region of the IR camera until the characteristic temperature distribution is in the shooting region of the IR camera.
Ragucci is in the same field of art of performing aerial searching/tracking of a thermal target using a thermal/infrared imager. Further, Ragucci teaches (a target) having a cooling/heating-generator and movable relative to (an aerial drone) (Paragraphs [0004-0005], [0025], [0017], Ragucci teaches locating thermal targets such as a human swimmer, mammals, vehicles. Further, paragraph [0025] states that a human is an example of a “heat source.” Under Broadest Reasonable Interpretation (BRI), the Examiner interprets a “thermal target” such as a human swimmer, a mammal, vehicle, etc. to be a “heating/cooling generator” since these “objects” have the ability to change temperature. For example, a human’s body temperature generates heat and can also “cool down” when the human’s body temperature decreases. Additionally, the Examiner interprets a swimmer or a mammal for example, to be movable relative to a drone since these “objects” can move around in the water, etc.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Zhang by tracking a thermal target such as a person or a mammal that is taught by Ragguci, to make the invention that locates thermal targets based on their heat signatures; thus, one of ordinary skilled in the art would be motivated to combine the references since heat distribution/thermal information is not sensitive to interference by light in the environment, which can compensate for the disadvantages of a sole RGB camera, which is sensitive to change in the environment (Zhang, Paragraph [0050]). In addition, it is often necessary to locate small thermal targets, such as mammals, vehicles, humans, etc., amidst a comparatively vast expanse of open land or water. Humans are the most common subject of searches, such as during a search and rescue operation following an aircraft/watercraft accident in open water (Ragucci, Paragraphs [0004], [0006]).
Zhang in view of Ragucci does not explicitly disclose setting the shooting region of the RGB camera so as to cause the target located in a position corresponding to a characteristic temperature distribution included in the thermographic image data to be included in the shooting region of the RGB camera and determining whether or not the characteristic temperature distribution is in a shooting region of the IR camera, under a condition where the characteristic temperature distribution is not in the shooting region of the IR camera, changing the shooting region of the IR camera until the characteristic temperature distribution is in the shooting region of the IR camera.
Lei is in the same field of art of target tracking using both a visible light camera and an infrared camera to improve target tracking performance. Further, Lei teaches setting the shooting region of the RGB camera (Paragraphs [0124-125], [0111], Lei teaches when the target object is located outside the field of view of the visible light camera, the visible light is controlled to re-lock the target object according to second tracking information (such as a location in the infrared image of the target object) and continue to perform visual tracking.) so as to cause the target located in a position corresponding to a characteristic temperature distribution included in the thermographic image data (Paragraphs [0021-22], [0112], [011], Lei teaches extracting a heat signature of the target object from the infrared image according to the location of the target object.) to be included in the shooting region of the RGB camera (Paragraphs [0124-127], [0111], Lei teaches during tracking, in a case of determining that the target object is lost in the visible light camera, for example, when the target object is shielded or the target object is located outside a field of view of the visible light camera, step (130) is performed. Step 130 controls the visible light camera to re-lock the target object according to the second tracking information (such as a location of the object in the infrared image of the target object) and continue to perform visual tracking. The visible light camera continues to use a current location of a target object as a photographing center based on a second location information (such as a location in the infrared image of the target object) fed back by the infrared camera, and re-locks the target object after the target object is not shielded. A photographing angle of the visible light camera may be adjusted according to the second location information of the target object, to enable the target object to fall within the field of view range of the visible light camera.) and determining whether or not the characteristic temperature distribution is in a shooting region of the IR camera (Paragraphs [0128-130], [0112], Lei teaches determining that the target object is lost in the infrared camera. The target object may be recognized by the infrared camera based on some given features such as heat signature.), under a condition where the characteristic temperature distribution is not in the shooting region of the IR camera (Paragraphs [0124], [0128-130], Lei teaches in the case of determining that the target object is lost in the infrared camera, for example, when the infrared camera cannot explore the heat signature of the target object, step 140 is performed. In the case of determining that the target object is lost in the infrared camera, the first tracking information recorded by the visible light camera at the current moment may be fed back to the infrared camera, so that the camera re-locks the target object according to the first tracking information. If the extracted heat signature does not match with the heat signature of the target object, manual operation is requested.), changing the shooting region of the IR camera until the characteristic temperature distribution is in the shooting region of the IR camera (Paragraph [0130], Lei teaches the infrared camera may quickly position a current location of the target object according to the first location information (from visible light camera) and extract a heat signature at this location. If the extracted heat signature matches with the heat signature of the target object, the target object is re-locked and infrared tracking continues to be performed.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Zhang in view of Ragucci by changing the shooting regions of the (IR and visible light) cameras to re-lock (visually track) the target object when the target object is lost in one of the cameras by using tracking information from the camera which is still able to track the target that is taught by Lei, to make the invention that utilizes tracking information obtained from the camera (either IR or visible light) in which the target object has not been “lost” to accurately track the target; thus, one of ordinary skilled in the art would be motivated to combine the references since cooperating two types of cameras can improve the tracking performance of the UAV and expand the scope of application (Lei, Paragraph [0131]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 15, Zhang in view of Ragucci in further view of Lei teaches the method according to claim 14, wherein the method further comprises providing image data corresponding to the visible image captured by the RGB camera (Paragraph [0026], Zhang teaches the first camera may obtain the RGB information to the target object.) to a host device remote from the target and the aerial drone (Paragraphs [0029], [0150], Lei teaches transmitting a visible light image acquired by the visible light camera to a remote control device of the UAV.).
In regards to Claim 16, Zhang in view of Ragucci in further view of Lei teaches the method according to claim 14, wherein said changing the shooting region of the RGB camera keeps the target in a middle portion of the shooting region of the RGB (Paragraph [0127], Lei teaches the visible light camera continues to use a current location of the target as a photographing center.).
In regards to Claim 17, Zhang in view of Ragucci teaches the method according to claim 14, wherein said changing the shooting region of the RGB camera includes changing at least one of direction, angle of view, and/or position of the RGB camera (Paragraphs [0033], [0026], Zhang teaches changing the attitude of the first gimbal and second gimbal, thereby changing attitude information of the first camera and second camera, such that the location of the target object relative to the first camera and the second camera is best. The first camera is mounted to the first gimbal and the first camera may be an ordinary camera which obtains RGB color information of the target object. The Examiner interprets “at least one of” and “and/or” to mean only one of the listed items (direction or angle of view or position) is required to meet the claim limitation.).
Claims 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (U.S. Patent Pub No. 2020/0043184 A1, hereafter referred to as Zhang) in view of Ragucci et al. (U.S. Patent Pub. No. 2009/0321636 A1, hereafter referred to as Ragucci) in further view of Lei (U.S. Patent Pub. No. 2022/0191389 A1, hereafter referred to as Lei) in further view of Yu et al. (U.S. Patent Pub. No. 2008/0259163 A1, hereafter referred to as Yu).
Regarding Claim 18, Zhang in view of Ragucci in further view of Lei discloses the method according to claim 14 wherein said controlling the automatic following is with respect to the target (Abstract, Paragraphs [0026], [0028], Zhang teaches tracking a target object based on at least one of the first characteristic information (such as color information, i.e., RGB color information) or the second characteristic information (heat distribution information).).
Zhang in view of Ragucci in further view of Lei does not explicitly disclose at least one additional target in the shooting region of the RGB camera for the visible image.
Yu is in the same field of art of tracking objects in image data. Further, Yu teaches (wherein said controlling the automatic following is with respect to) at least one additional target in the shooting region of the RGB camera for the visible image (Abstract, Paragraph [0047], Fig. 1, Yu teaches distributed tracking of multiple targets. Multiple targets to be tracked by a plurality of trackers are detected in a frame. Fig. 1 shows three “targets” in the “shooting region.” See the computer 104 showing the video frame. The frames are captured by a video surveillance camera.).
PNG
media_image2.png
538
720
media_image2.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Zhang in view of Ragucci in further view of Lei by tracking multiple targets simultaneously that is taught by Yu, to make the invention that is able to track multiple objects such as multiple humans in crowded areas, where multiple objects are present; thus, one of ordinary skilled in the art would be motivated to combine the references to monitor and track human activities in crowded areas (with multiple targets (humans, etc.)), such as airports so that any unusual activities may be detected and possible damage may be prevented via monitoring and tracking (Yu, Paragraph [0003]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
[AltContent: arrow][AltContent: arrow]In regards to Claim 19, Zhang in view of Ragucci in view of Lei in further view of Yu discloses the method according to claim 18, wherein the target and the at least one additional target move independent of each other (Paragraphs [0002], [0018], Figs. 5A & 5B, Yu teaches tracking multiple targets in a surveillance system. Fig. 1 shows three targets being tracked (102a-102c). The Examiner interprets the three targets i.e., humans “move independent of each other” as they are not physically connected and are free to move. See Fig. 5B, the targets each move differently. Target 502a moves downward, target 502b moves upward, and so on.).
PNG
media_image3.png
493
462
media_image3.png
Greyscale
In regards to Claim 20, Zhang in view of Ragucci in view of Lei in further view of Yu discloses the method according to claim 18, wherein said controlling the automatic following is with respect to the target and at least one additional target in the shooting region of the RGB camera for the visible image (Paragraph [0047], Yu teaches each tracker 502a-502d is associated with a different target present in the frame. Fig. 5B shows the next frame received from the video surveillance camera. Each of the targets has moved from its earlier position in Fig. 5A. Each of the trackers has maintained an association with its corresponding target.) is responsive to respective commands (Paragraph [0026], Zhang teaches providing control commands to the UAV to control the first camera (RGB camera) based on an instruction in the control command to acquire image information that includes the target object.) to start automatically following the targets (Paragraph [0021], Yu teaches tracking module 212 which tracks the targets by using a plurality of trackers that are initialized after the multiple targets are detected.).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SYDNEY L BLACKSTEN whose telephone number is (571)272-7120. The examiner can normally be reached 8:30am-4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached at 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SYDNEY L BLACKSTEN/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights