DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 14-26 filed on 27 SEP 2024 are currently pending and have been examined.
Priority
The pending application 18/851,828, filed on 27 SEP 2024, is a national stage application filed under 35 U.S.C. 371 of PCT/JP2023/006794, filed on 24 FEB 2023, and claims priority from foreign application JP2022-052014, filed on 28 MAR 2022.
Information Disclosure Statement
The information disclosure statements (IDSs) submitted on 18 DEC 2024 and 23 JUN 2025 have been considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 14, 20, 22-23 and 26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhaoyu et al. (JP 2021140764 A, cited by applicant in IDS dated 18 DEC 2024) in view of Kirino et al. (US 2018/0375185 A1, cited by applicant in IDS dated 23 JUN 2025).
Regarding claim 14 (New), Zhaoyu et al. discloses:
[Note: what is not explicitly taught by Zhaoyu et al. has been struck-through]
A monitoring system (Zhaoyu et al. “To achieve intelligent monitoring in traffic systems, millimeter-wave radar and video camera heads may be used simultaneously to monitor vehicles, pedestrians, and other objects.” - ¶ [0002]) comprising:
a radar (Zhaoyu et al. “To achieve intelligent monitoring in traffic systems, millimeter-wave radar and video camera heads may be used simultaneously to monitor vehicles, pedestrians, and other objects.” - ¶ [0002]); and
at least one camera (Zhaoyu et al. “To achieve intelligent monitoring in traffic systems, millimeter-wave radar and video camera heads may be used simultaneously to monitor vehicles, pedestrians, and other objects.” - ¶ [0002]), the radar includes:
an antenna (Zhaoyu et al. obvious component of a radar);
at least one (Zhaoyu et al. processor 1101, Fig. 11); and
a (Zhaoyu et al. memory unit 1102, Fig. 11) storing
transmit, by the antenna, an electromagnetic wave to a first monitoring area (Zhaoyu et al. field of view of radar) and receive, by the antenna, a reflected wave of the electromagnetic wave (Zhaoyu et al. obvious steps performed by the radar in order to obtain detection results - ¶ [0012]);
detect, by the at least one integrated circuit, a presence or absence of an object in the first monitoring area (Zhaoyu et al. “Radar detection results include information such as the object's velocity and position coordinates…” - ¶ [0012]) on a basis of the reflected wave; and
generate detected object attribute information indicating an attribute of a detected object (Zhaoyu et al. velocity of radar target 11, Fig. 1) detected by the at least one integrated circuit and generate radar position information (Zhaoyu et al. “Radar detection results include information such as the object's velocity and position coordinates…” - ¶ [0012]) indicating a position of the detected object (Zhaoyu et al. “Radar information includes the position and velocity of an object in world coordinates…” - ¶ [0051]) on a basis of first installation information (Zhaoyu et al. in order to determine the position of the radar target in world coordinates, the position of the radar must be known in world coordinates) including installation position (Zhaoyu et al. “the coordinates of the radar in the world coordinate system are (ro<sub> x </sub>, ro<sub> y </sub>)” - ¶ [0054]) of the antenna and direction of the antenna and information of field of view of the antenna (Zhaoyu et al. in order to determine the position of the radar target in world coordinates, the direction of the antenna of the radar and the field of view of the antenna must be known); and
the at least one camera includes:
an imaging unit includes a lens and an image sensor (Zhaoyu et al obvious components of a camera);
at least one (Zhaoyu et al. processor 1101, Fig. 11); and
a (Zhaoyu et al. memory unit 1102, Fig. 11) storing camera instructions that, when execute by the at least one (Zhaoyu et al. Of these, the memory unit 1102 can store various types of data, and can also store the data fusion program 1103, and can execute the program 1103 under the control of the processor 1101, and can also store various types of data, such as predetermined values and predetermined conditions.” - ¶ [0126]), cause the at least one
capture, by the image sensor, a second monitoring area (Zhaoyu et al. field of view of the camera) at least partially overlapping the first monitoring area (Zhaoyu et al. “although they are detecting the same target, the radar detection results and video detection results do not perfectly match.” - ¶ [0014]); and
obtain imaging position information indicating a position of an imaged object included in a captured image (Zhaoyu et al. a frame of video) of the second monitoring area on a basis of second installation information (Zhaoyu et al. in order to determine the position of the camera target in world coordinates, the position of the camera must be known in world coordinates) including an installation position (Zhaoyu et al. “the coordinates of the camera head in the world coordinate system are ( co<sub> x </sub>, co<sub> y </sub>)” - ¶ [0054]) of the imaging unit and an imaging direction of the imaging unit and information of a field of view of the imaging unit (Zhaoyu et al. in order to determine the position of the camera target in world coordinates, the direction of the imaging direction of the imaging unit and the field of view of the imaging unit must be known); and obtain imaged object attribute information indicating an attribute of the imaged object (Zhaoyu et al. velocity of camera target 12, Fig. 1) on a basis of the captured image (Zhaoyu et al. “determine the position and velocity of the object in the world coordinate system based on the video information” - ¶ [0021]), wherein
determine whether or not the detected object and the imaged object are an identical object (Zhaoyu et al. “If a video target that satisfies the matching conditions can be found together with the radar target, then the radar target and the video target may correspond to the same object in the real world. If a video target that satisfies the matching conditions cannot be found together with the radar target, then the radar target does not have an associated or corresponding video target.” - ¶ [0026]) on a basis of the radar position information and the imaging position information (Zhaoyu et al. “In some embodiments, the video target associated with a radar target can be determined based on the distance between the radar target and the video target.” - ¶ [0027]);
notify a user of notification information (Zhaoyu et al. “By fusing the targets detected and the collected data from each set of sensing devices to generate an electronic map, targets in each direction on the road can be displayed in real time.” - ¶ [0009]; where the user is notified of notification information via the electronic map),
in a case where the detected object and the imaged object are an identical object, the notification information is information in which a first identifier for identifying the detected object (Zhaoyu et al. “one object number may be assigned to each matched radar target and video target…” - ¶ [0046]), the detected object attribute information, the captured image, and the imaged object attribute information at least are associated together (Zhaoyu et al. “based on the marking results of 204, the position and velocity of each marked object can be determined based on the video information and/or radar information corresponding to that object.” - ¶ [0050]), and
in a case where the detected object and the imaged object are not an identical object, the notification information is information based on at least one of the radar position information (Zhaoyu et al. position based on radar detection results - ¶ [0051]), the detected object attribute information (Zhaoyu et al. velocity based on radar detection results - ¶ [0051]), the imaging position information (Zhaoyu et al. position based on video information - ¶ [0051]), or the imaged object attribute information (Zhaoyu et al. velocity based on video information - ¶ [0051]; “one object number may be assigned to each unmatched radar target, and one object number may be assigned to each unmatched video target.)” - ¶ [0046]).
Kirino et al. discloses:
a radar (Kirino et al. radar system 510, Fig. 32); and
at least one camera (Kirino et al. camera system 700, Fig. 32),
the radar includes:
an antenna (Kirino et al. array antenna AA, Fig. 22);
at least one integrated circuit (Kirino et al. “The transmission line module 100 of the present embodiment further includes a circuit board 126 and a monolithic microwave integrated circuit (MMIC) 127.” - ¶ [0097]);
at least one radar processor (Kirino et al. processor PR, Fig. 28) ; and
a radar memory (Kirino et al. memory device MD, Fig. 28) storing radar instructions that, when execute by the at least one radar processor (Kirino et al. “In the signal processing circuit 560 with this construction, too, a computer program that is stored in the memory device MD may fulfill the functions of the reception intensity calculation section 532, the DBF processing section 535, the distance detection section 533, the velocity detection section 534, the azimuth detection section 536, the target link processing section 537, the matrix generation section 538, and the arriving wave estimation unit AU shown in FIG. 25.” - ¶ [0221]), cause the at least one radar processor to:
transmit, by the antenna, an electromagnetic wave (Kirino et al. “The transmission antenna Tx radiates a transmission wave…” - ¶ [0192]) to a first monitoring area (Kirino et al. defined by the field of view of the radar) and receive, by the antenna, a reflected wave of the electromagnetic wave (Kirino et al. “The reception antenna Rx that is dedicated to reception only outputs a reception signal in response to one or plural arriving waves (e.g., a millimeter wave(s)).” - ¶ [0192]);
detect, by the at least one integrated circuit, a presence or absence of an object in the first monitoring area on a basis of the reflected wave (Kirino et al. “From the signal intensities of beat frequencies, the reception intensity calculation section 532 detects any signal intensity that exceeds a predefined value (threshold value), thus determining the presence of a target.” - ¶ [0226]); and
the at least one camera includes:
an imaging unit includes a lens and an image sensor (Kirino et al. obvious components of cameras; “an optical sensor (e.g., a camera)” - ¶ [0311]);
at least one camera processor (Kirino et al. image processing circuit 720, Fig. 24); and
capture, by the image sensor, a second monitoring area at least partially overlapping the first monitoring area (Kirino et al. “Note that the each of the following detection devices is to be installed in the vehicle, and at least includes a millimeter wave radar detection section, an image detection section (e.g., a camera) which is oriented in a direction overlapping the direction of detection by the millimeter wave radar detection section, and a matching section.” - ¶ [0350])
It would have been obvious to someone with ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Kirino et al. into the invention of Zhaoyu et al. to yield the invention of claim 14 above. Both Zhaoyu et al. and Kirino et al. are considered analogous arts to the claimed invention as they both disclose detecting objects by fusing data obtained from radar sensors and cameras. Zhaoyu et al. discloses the limitations of claim 14 outlined above. However, Zhaoyu et al. fails to explicitly disclose the specific components of the radar and camera. This feature is disclosed by Kirino et al. where the radar system 510 includes antenna array AA, MMICs, memory and processor, and the camera system 700 includes an image processing circuit 720 (Kirino et al. ¶ [0097], [0192], [0221], [226], [0311], [0350]; Figs. 22, 24, 25, 28). The combination of Zhaoyu et al. and Kirino et al. would be obvious with a reasonable expectation of success to “provide a small-sized, highly efficient, and high-performance fusion apparatus.” (Kirino et al. ¶ [0316]) and improve the accuracy of target recognition (Kirino et al. ¶ [0316]).
Regarding claim 20 (New), Zhaoyu et al. as modified above discloses:
The monitoring system according to claim 14, further comprising:
an information processing apparatus (Zhaoyu et al. information processing device 1100, Fig. 11) includes at least one processor (Zhaoyu et al. CPU 1101, Fig. 11), a memory (Zhaoyu et al. memory 1102, Fig. 11) storing instructions that, when execute by the at least one processor (Zhaoyu et al. Of these, the memory unit 1102 can store various types of data, and can also store the data fusion program 1103, and can execute the program 1103 under the control of the processor 1101, and can also store various types of data, such as predetermined values and predetermined conditions.” - ¶ [0126]), cause the at least one processor to:
superimpose and display a position of the detected object and a position of the imaged object (Zhaoyu et al. data fusion program 1102, Fig. 11) on a map image (Zhaoyu et al. “By fusing the targets detected and the collected data from each set of sensing devices to generate an electronic map, targets in each direction on the road can be displayed in real time.” - ¶ [0009]) including the first monitoring area and the second monitoring area as the notification processing (Zhaoyu et al. where the user is notified of notification information via the electronic map).
Regarding claim 22 (New), Zhaoyu et al. as modified above discloses:
The monitoring system according to claim 20, wherein the map image is a two-dimensional image (Zhaoyu et al. “By fusing the targets detected and the collected data from each set of sensing devices to generate an electronic map, targets in each direction on the road can be displayed in real time.” - ¶ [0009]) or a three-dimensional image relating to an area including the first monitoring area and the second monitoring area (Zhaoyu et al. the electronic map includes the first monitoring area and the second monitoring area, Fig. 1).
Regarding claim 23 (New), Zhaoyu et al. as modified above discloses:
The monitoring system according to claim 14, wherein
the monitoring system includes a plurality of the cameras (Zhaoyu et al. “each set of sensing devices including one camera head and one radar” - ¶ [0003]; “by installing multiple sets of sensing devices and performing detection, information from multiple directions and all directions can be obtained… the sensing ranges of these sensing devices can overlap.” - ¶ [0074]), and
in a case where there are a plurality of captured images of the imaged object determined to be identical to the detected object, the notification information is information in which the first identifier corresponding to the detected object (Zhaoyu et al. “the fusion of all targets detected by multiple sets of sensing devices is completed. The same object can be labeled using the same object number after fusion.” - ¶ [0078]), a plurality of captured images relating to the imaged object determined to be identical to the detected object (Zhaoyu et al. where the overlapping sensing ranges includes overlapping fields of view of the cameras of the multiple sets of sensing devices and therefore a plurality of capture images) and imaged object attribute information relating to the imaged object determined to be identical to the detected object are associated together (Zhaoyu et al. “In embodiments of the present invention, in addition to performing fusion on targets detected by multiple sets of sensing devices based on the location of the target detected by each set of sensing devices (the target after fusing radar detection results and video detection results), the speed of the target detected by each set of sensing devices may also be used as a reference.” - ¶ [0079]; the speed of the target detected by the radar and the camera are also associated together).
Regarding claim 26 (New), Zhaoyu et al. discloses:
A monitoring method (Zhaoyu et al. “To achieve intelligent monitoring in traffic systems, millimeter-wave radar and video camera heads may be used simultaneously to monitor vehicles, pedestrians, and other objects.” - ¶ [0002]) comprising:
transmitting, by an antenna (Zhaoyu et al. obvious component of a radar), an electromagnetic wave to a first monitoring area (Zhaoyu et al. field of view of radar) and receiving, by the antenna, a reflected wave of the electromagnetic wave (Zhaoyu et al. obvious steps performed by the radar in order to obtain detection results - ¶ [0012]);
executing detection processing to detect a presence or absence of an object in the first monitoring area (Zhaoyu et al. “Radar detection results include information such as the object's velocity and position coordinates…” - ¶ [0012]) on a basis of the reflected wave;
generating detected object attribute information indicating an attribute of a detected object (Zhaoyu et al. velocity of radar target 11, Fig. 1) detected by at least one integrated circuit that executes the detection processing on a basis of a result of the detection processing and generating radar position information (Zhaoyu et al. “Radar detection results include information such as the object's velocity and position coordinates…” - ¶ [0012]) indicating a position of the detected object (Zhaoyu et al. “Radar information includes the position and velocity of an object in world coordinates…” - ¶ [0051]) on a basis of first installation information (Zhaoyu et al. in order to determine the position of the radar target in world coordinates, the position of the radar must be known in world coordinates) including installation position (Zhaoyu et al. “the coordinates of the radar in the world coordinate system are (ro<sub> x </sub>, ro<sub> y </sub>)” - ¶ [0054]) of the antenna and direction of the antenna and information of field of view of the antenna (Zhaoyu et al. in order to determine the position of the radar target in world coordinates, the direction of the antenna of the radar and the field of view of the antenna must be known);
imaging, with an imaging unit, a second monitoring area (Zhaoyu et al. field of view of the camera) at least partially overlapping the first monitoring area (Zhaoyu et al. “although they are detecting the same target, the radar detection results and video detection results do not perfectly match.” - ¶ [0014]);
obtaining imaging position information indicating a position of an imaged object included in a captured image (Zhaoyu et al. a frame of video) of the second monitoring area on a basis of second installation information (Zhaoyu et al. in order to determine the position of the camera target in world coordinates, the position of the camera must be known in world coordinates) including an installation position (Zhaoyu et al. “the coordinates of the camera head in the world coordinate system are ( co<sub> x </sub>, co<sub> y </sub>)” - ¶ [0054]) of the imaging unit and an imaging direction of the imaging unit and information of a field of view of the imaging unit (Zhaoyu et al. in order to determine the position of the camera target in world coordinates, the direction of the imaging direction of the imaging unit and the field of view of the imaging unit must be known) and obtaining imaged object attribute information indicating an attribute of the imaged object (Zhaoyu et al. velocity of camera target 12, Fig. 1) on a basis of the captured image (Zhaoyu et al. “determine the position and velocity of the object in the world coordinate system based on the video information” - ¶ [0021]);
executing determination processing to determine whether or not the detected object and the imaged object are an identical object (Zhaoyu et al. “If a video target that satisfies the matching conditions can be found together with the radar target, then the radar target and the video target may correspond to the same object in the real world. If a video target that satisfies the matching conditions cannot be found together with the radar target, then the radar target does not have an associated or corresponding video target.” - ¶ [0026]) on a basis of the radar position information and the imaging position information (Zhaoyu et al. “In some embodiments, the video target associated with a radar target can be determined based on the distance between the radar target and the video target.” - ¶ [0027]); and
executing notification processing to notify a user of notification information (Zhaoyu et al. “By fusing the targets detected and the collected data from each set of sensing devices to generate an electronic map, targets in each direction on the road can be displayed in real time.” - ¶ [0009]; where the user is notified of notification information via the electronic map), wherein
in a case where the detected object and the imaged object are an identical object, the notification information is information in which a first identifier for identifying the detected object (Zhaoyu et al. “one object number may be assigned to each matched radar target and video target…” - ¶ [0046]), the detected object attribute information, the captured image, and the imaged object attribute information at least are associated together (Zhaoyu et al. “based on the marking results of 204, the position and velocity of each marked object can be determined based on the video information and/or radar information corresponding to that object.” - ¶ [0050]), and
in a case where the detected object and the imaged object are not an identical object, the notification information is information based on at least one of the radar position information (Zhaoyu et al. position based on radar detection results - ¶ [0051]), the detected object attribute information (Zhaoyu et al. velocity based on radar detection results - ¶ [0051]), the imaging position information (Zhaoyu et al. position based on video information - ¶ [0051]), or the imaged object attribute information (Zhaoyu et al. velocity based on video information - ¶ [0051]; “one object number may be assigned to each unmatched radar target, and one object number may be assigned to each unmatched video target.)” - ¶ [0046]).
Kirino et al. discloses:
A monitoring method comprising:
transmitting, by an antenna (Kirino et al. array antenna AA, Fig. 22), an electromagnetic wave (Kirino et al. “The transmission antenna Tx radiates a transmission wave…” - ¶ [0192]) to a first monitoring area (Kirino et al. defined by the field of view of the radar) and receiving, by the antenna, a reflected wave of the electromagnetic wave (Kirino et al. “The reception antenna Rx that is dedicated to reception only outputs a reception signal in response to one or plural arriving waves (e.g., a millimeter wave(s)).” - ¶ [0192]);
executing detection processing to detect a presence or absence of an object in the first monitoring area on a basis of the reflected wave (Kirino et al. “From the signal intensities of beat frequencies, the reception intensity calculation section 532 detects any signal intensity that exceeds a predefined value (threshold value), thus determining the presence of a target.” - ¶ [0226]);
imaging, with an imaging unit (Kirino et al. camera system 700, Fig. 32), a second monitoring area (Kirino et al. defined by the field of view of the camera) at least partially overlapping the first monitoring area (Kirino et al. “Note that the each of the following detection devices is to be installed in the vehicle, and at least includes a millimeter wave radar detection section, an image detection section (e.g., a camera) which is oriented in a direction overlapping the direction of detection by the millimeter wave radar detection section, and a matching section.” - ¶ [0350]);
It would have been obvious to someone with ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Kirino et al. into the invention of Zhaoyu et al. to yield the invention of claim 26 above. Both Zhaoyu et al. and Kirino et al. are considered analogous arts to the claimed invention as they both disclose detecting objects by fusing data obtained from radar sensors and cameras. Zhaoyu et al. discloses the limitations of claim 26 outlined above. However, Zhaoyu et al. fails to explicitly disclose the specific components of the radar and camera. This feature is disclosed by Kirino et al. where the radar system 510 includes antenna array AA, MMICs, memory and processor, and the camera system 700 includes an image processing circuit 720 (Kirino et al. ¶ [0097], [0192], [0221], [226], [0311], [0350]; Figs. 22, 24, 25, 28). The combination of Zhaoyu et al. and Kirino et al. would be obvious with a reasonable expectation of success to “provide a small-sized, highly efficient, and high-performance fusion apparatus.” (Kirino et al. ¶ [0316]) and improve the accuracy of target recognition (Kirino et al. ¶ [0316]).
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhaoyu et al. (JP 2021140764 A, cited by applicant in IDS dated 18 DEC 2024) in view of Kirino et al. (US 2018/0375185 A1, cited by applicant in IDS dated 23 JUN 2025) as applied to claim 14 above, and further in view of Hanna et al. (US 2010/0013917 A1, cited by applicant in IDS dated 23 JUN 2025).
Regarding claim 15 (New), Zhaoyu et al. as modified above discloses:
[Note: what is not explicitly taught by Zhaoyu et al. has been struck-through]
The monitoring system according to claim 14
in a case where the detected object and the imaged object are an identical object, the notification information is information in which a monitoring position captured image (Zhaoyu et al. a frame of video captured by cameras) obtained by the imaging unit imaging the object included in the second monitoring area and imaged object attribute information of an object (Zhaoyu velocity of the object based on video information) included in the monitoring position captured image are associated with the first identifier corresponding to the detected object (Zhaoyu et al. “one object number may be assigned to each matched radar target and video target…” - ¶ [0046]; “determining, for each labeled object, the position and velocity of the object in the world coordinate system based on video information and/or radar information corresponding to the object.” - ¶ [0139]).
Hanna et al. discloses:
the at least one camera includes:
a motor that adjusts the imaging direction (Hanna et al. “the system 100 comprises a camera pan/tilt/zoom (PTZ) module 102 that controls the pan/tilt/zoom parameters of at least one imaging sensor 104 (e.g., a visible or infrared camera)” - ¶ [0024]), wherein the at least one camera processor configured to execute the camera instructions to:
drive the motor on a basis of a relationship between a position of the motor and an imaging range of the imaging unit in real space and the second installation information so that the imaging unit images an object, among the detected objects, included in the second monitoring area (Hanna et al. “That is, the method 1200 issues a control signal or ESD (e.g., to a camera/PTZ module 102) that causes the second sensor to adjust its field of view so that the field of view includes the detected object… Thus, in one embodiment, the second sensor is a PTZ camera, and the control signal causes the PTZ camera to pan, tilt and/or zoom such that the PTZ camera captures images of the detected object.” - ¶ [0063]), and
in a case where the detected object and the imaged object are an identical object (Hanna et al. “In this embodiment, slewing of the second sensor to the detected object includes mapping the received location information to the three-dimensional (e.g., PTZ) coordinates of the detected object in the real world…” - ¶ [0063]).
It would have been obvious to someone with ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Hanna et al. into the invention of Zhaoyu et al. to yield the invention of claim 15 above. Zhaoyu et al., Kirino et al. and Hanna et al. are considered analogous arts to the claimed invention as they disclose detecting objects by fusing data obtained from radar sensors and cameras. Zhaoyu et al. as modified above discloses the monitoring system of claim 14. However, Zhaoyu et al. fails to explicitly disclose the at least one camera includes a motor. This feature is disclosed by Hanna et al. where “the system 100 comprises a camera pan/tilt/zoom (PTZ) module 102 that controls the pan/tilt/zoom parameters of at least one imaging sensor 104 (e.g., a visible or infrared camera)” (Hanna et al. ¶ [0024]). The combination of Zhaoyu et al., Kirino et al. and Hanna et al. would be obvious with a reasonable expectation of success to “adjust its field of view so that the field of view includes the detected object” (Hanna et al. ¶ [0063]) in order to provide a robust alert capability (Hanna et al. ¶ [0069]).
Claim(s) 16-19, 21, and 24-25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhaoyu et al. (JP 2021140764 A, cited by applicant in IDS dated 18 DEC 2024) in view of Kirino et al. (US 2018/0375185 A1, cited by applicant in IDS dated 23 JUN 2025) as applied to claim 14 above, and further in view of Deyle et al. (US 2020/0053325 A1).
Regarding claim 16 (New), Zhaoyu et al. as modified above discloses:
[Note: what is not explicitly taught by Zhaoyu et al. has been struck-through]
The monitoring system according to claim 14
Deyle et al. discloses:
The monitoring system according to claim 14, wherein
the at least one radar processor configured to execute the radar instructions to, or the at least one camera processor configured to execute the camera instructions to:
execute tracking processing to track an object on a basis of a result of the detection processing (Deyle et al. “Examples of scanners include laser scanners or rangefinders, radar… In addition to detecting obstacles, objects, and individuals, the scanners can track moving objects or individuals, for instance to follow the movement of suspicious individuals… Likewise, information captured by the scanners can be used by or provided to other components of the robot or an entity external to the robot (such as a component of FIG. 2), for instance via the communication interface 714.” - ¶ [0114]);
execute abnormal determination processing (Deyle et al. “A security anomaly can be identified based on accessed information describing a characteristic of an environment contrary to an expected characteristic of the environment.” - ¶ [0165]) to determine whether or not the detected object or the imaged object is an object for which an alarm is to be activated on a basis of an abnormal determination condition including at least one of the imaged object attribute information or a movement state of the detected object or the imaged object (Deyle et al. “In the event the individual persists and attempts to or succeeds in bypassing the robot, the robot can perform an enhanced security operation, for instance by notifying security personnel 250, by following the individual, by triggering an alarm…” - ¶ [0153]), and
in a case where the imaged object attribute information corresponding to an object targeted for the tracking processing (Deyle et al. “In the event that an identified individual being followed, escorted, or tracked performs an unauthorized action or commits a security violation, the robot 100 can perform one or more security operations in response.” - ¶ [0151]) or a movement state of the detected object targeted for the tracking processing or the imaged object targeted for the tracking processing is determined to correspond to the abnormal determination condition (Deyle et al. where the abnormal condition is a security violation), the notification processing includes an alarm (Deyle et al. “For instance, during the performance of any of the security operations below, a robot 100 can perform one or more of the following operations… sounding or triggering an alarm… tracking or following an object or individual…” - ¶ [0130]).
It would have been obvious to someone with ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Deyle et al. into the invention of Zhaoyu et al. to yield the invention of claim 16 above. Zhaoyu et al., Kirino et al. and Deyle et al. are considered analogous arts to the claimed invention as they disclose monitoring systems comprising radar sensors and cameras. Zhaoyu et al. as modified above discloses the monitoring system of claim 14. However, Zhaoyu et al. fails to explicitly disclose tracking processing, determining an abnormal condition, and an alarm. This feature is disclosed by Deyle et al. where surveillance is performed by a mobile robot to track objects, determine an abnormal condition such as a security violation exists, and triggering an alarm (Deyle et al. ¶ [0114], [0130], [0151], [0153], [0165]. The combination of Zhaoyu et al., Kirino et al. and Deyle et al. would be obvious with a reasonable expectation of success to “provide a small-sized, highly efficient, and high-performance fusion apparatus.” (Kirino et al. ¶ [0316]) and improve the accuracy of target recognition (Kirino et al. ¶ [0316]) and reduce “the burden on an operator by automating robot functionality and consolidating robot information” and “allow the operator to prioritize her focus on high-risk or dangerous security events” (Deyle et al. ¶ [0323]).
Regarding claim 17 (New), Zhaoyu et al. as modified above discloses:
[Note: what is not explicitly taught by Zhaoyu et al. has been struck-through]
The monitoring system according to claim 16
Deyle et al. discloses:
wherein
the abnormal determination condition is at least one of entry of the detected object or the imaged object into a no-entry area where entry by an object is prohibited, the detected object or the imaged object lingering in the no-entry area, a specific behavior pattern relating to the detected object or the imaged object, an item left by the detected object or the imaged object, or an item taken by the detected object or the imaged object (Deyle et al. “As used herein, “suspicious activity” can refer to the identification of an unknown individual, an unauthorized individual, an individual performing an unauthorized or unusual activity, movement in areas or at times when movement is not expected, unusual network activity, sound in areas where sound is not expected, types of sound associated with security risks, objects in unexpected or unauthorized locations, unlocked or open doors or windows at unauthorized times, or any other activity associated with a potential security risk or threat or combination thereof.” - ¶ [0131]; see also ¶ [0138], [0142]-[0143], [0145], [0167], [0276]).
It would have been obvious to someone with ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Deyle et al. into the invention of Zhaoyu et al. to yield the invention of claim 17 above. Zhaoyu et al., Kirino et al. and Deyle et al. are considered analogous arts to the claimed invention as they disclose monitoring systems comprising radar sensors and cameras. Zhaoyu et al. as modified above discloses the monitoring system of claim 16. However, Zhaoyu et al. fails to explicitly disclose the abnormal determination conditions. This feature is disclosed by Deyle et al. where suspicious activity is detected by the robot including identification of an unauthorized individual and objects in unexpected or unauthorized locations. The combination of Zhaoyu et al., Kirino et al. and Deyle et al. would be obvious with a reasonable expectation of success to “provide a small-sized, highly efficient, and high-performance fusion apparatus.” (Kirino et al. ¶ [0316]) and improve the accuracy of target recognition (Kirino et al. ¶ [0316]) and reduce “the burden on an operator by automating robot functionality and consolidating robot information” and “allow the operator to prioritize her focus on high-risk or dangerous security events” (Deyle et al. ¶ [0323]).
Regarding claim 18 (New), Zhaoyu et al. as modified above discloses:
[Note: what is not explicitly taught by Zhaoyu et al. has been struck-through]
The monitoring system according to claim 16
Deyle et al. discloses:
wherein the alarm is a threat or a warning targeting an object or information for causing a drone or robot to issue the threat or the warning (Deyle et al. “a robot 100 can perform one or more of the following operations: warning one or more individuals…” - ¶ [0130]).
It would have been obvious to someone with ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Deyle et al. into the invention of Zhaoyu et al. to yield the invention of claim 18 above. Zhaoyu et al., Kirino et al. and Deyle et al. are considered analogous arts to the claimed invention as they disclose monitoring systems comprising radar sensors and cameras. Zhaoyu et al. as modified above discloses the monitoring system of claim 16. However, Zhaoyu et al. fails to explicitly disclose wherein the alarm is a threat or a warning targeting an object or information for causing a drone or robot to issue the threat or the warning. This feature is disclosed by Deyle et al. where “a robot 100 can perform one or more of the following operations: warning one or more individuals…” (Deyle et al. ¶ [0130]). The combination of Zhaoyu et al., Kirino et al. and Deyle et al. would be obvious with a reasonable expectation of success to “provide a small-sized, highly efficient, and high-performance fusion apparatus.” (Kirino et al. ¶ [0316]) and improve the accuracy of target recognition (Kirino et al. ¶ [0316]) and reduce “the burden on an operator by automating robot functionality and consolidating robot information” and “allow the operator to prioritize her focus on high-risk or dangerous security events” (Deyle et al. ¶ [0323]).
Regarding claim 19 (New), Zhaoyu et al. as modified above discloses:
[Note: what is not explicitly taught by Zhaoyu et al. has been struck-through]
The monitoring system according to claim 14
Deyle et al. discloses:
The monitoring system according to claim 14, wherein
the imaged object attribute information is information including at least one of type, gender, age bracket, height (Deyle et al. “Likewise, the robot can identify a height or size of the individual…” - ¶ [0141]) color of clothes, vehicle type, vehicle color, a score indicating accuracy of the attribute (Deyle et al. “An object within the captured image is classified 2605 by the robot, for instance by performing a classification operation that produces a candidate classification for the object and a confidence score representative of the classification.” - ¶ [0332]), or movement speed of the imaged object.
It would have been obvious to someone with ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Deyle et al. into the invention of Zhaoyu et al. to yield the invention of claim 19 above. Zhaoyu et al., Kirino et al. and Deyle et al. are considered analogous arts to the claimed invention as they disclose monitoring systems comprising radar sensors and cameras. Zhaoyu et al. as modified above discloses the monitoring system of claim 14. However, Zhaoyu et al. fails to explicitly disclose the imaged object attribute information is information including at least one of type, gender, age bracket, height, color of clothes, vehicle type, vehicle color, a score indicating accuracy of the attribute, or movement speed of the imaged object. This feature is disclosed by Deyle et al. where the height of an individual is detected and classification of objects is performed to provide candidate classifications and confidence scores (Deyle et al. ¶ [0141], [0332]). The combination of Zhaoyu et al., Kirino et al. and Deyle et al. would be obvious with a reasonable expectation of success to “provide a small-sized, highly efficient, and high-performance fusion apparatus.” (Kirino et al. ¶ [0316]) and improve the accuracy of target recognition (Kirino et al. ¶ [0316]) and reduce “the burden on an operator by automating robot functionality and consolidating robot information” and “allow the operator to prioritize her focus on high-risk or dangerous security events” (Deyle et al. ¶ [0323]).
Regarding claim 21 (New), Zhaoyu et al. as modified above discloses:
[Note: what is not explicitly taught by Zhaoyu et al. has been struck-through]
The monitoring system according to claim 20
Deyle et al. discloses:
wherein
in a case where the notification information includes the imaged object attribute information (Deyle et al. “In some embodiments, if the object is classified with a confidence score that is greater than a second confidence interval, the robot can present the captured image to the human operator with one or more “best guesses” of the classification of the object.” - ¶ [0326]), the information processing apparatus superimposes and displays the captured image (Deyle et al. image 3115, Fig. 31) and the imaged object attribute information (Deyle et al. “For instance, a facial recognition operation performed on the image of the person can output a number of candidate identities, each associated with a corresponding confidence score.” - ¶ [0371]; where the confidence score is considered to be the imaged object attribute information) on the map image (Deyle et al. “In embodiment of FIG. 31, the interface element is displayed as a pop-up window overlaid onto the map near a representation of the person displayed within the security interface.” - ¶ [0370]).
It would have been obvious to someone with ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Deyle et al. into the invention of Zhaoyu et al. to yield the invention of claim 21 above. Zhaoyu et al., Kirino et al. and Deyle et al. are considered analogous arts to the claimed invention as they disclose monitoring systems comprising radar sensors and cameras. Zhaoyu et al. as modified above discloses the monitoring system of claim 20. However, Zhaoyu et al. fails to explicitly disclose in a case where the notification information includes the imaged object attribute information, the information processing apparatus superimposes and displays the captured image and the imaged object attribute information on the map image. This feature is disclosed by Deyle et al. where notification information includes the captured image, the candidate classification of an object or the candidate identification of a person with confidence scores, and notification information is superimposed on the map image (Deyle et al. [0326], [0370], [0371]; Fig. 31) . The combination of Zhaoyu et al., Kirino et al. and Deyle et al. would be obvious with a reasonable expectation of success to “provide a small-sized, highly efficient, and high-performance fusion apparatus.” (Kirino et al. ¶ [0316]) and improve the accuracy of target recognition (Kirino et al. ¶ [0316]) and reduce “the burden on an operator by automating robot functionality and consolidating robot information” and “allow the operator to prioritize her focus on high-risk or dangerous security events” (Deyle et al. ¶ [0323]).
Regarding claim 24 (New), Zhaoyu et al. as modified above discloses:
[Note: what is not explicitly taught by Zhaoyu et al. has been struck-through]
The monitoring system according to claim 23
Deyle et al. discloses:
the notification information is information including a captured image (Deyle et al. image 3115, Fig. 31) corresponding to a predetermined criterion relating to the object (Deyle et al. “In response to the captured image indicating a security violation, the robot provides 3010 the captured image and information describing the security violation to a remote terminal for display to a human operator (for instance, via a security interface) for review.” - ¶ [0365]).
It would have been obvious to someone with ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Deyle et al. into the invention of Zhaoyu et al. to yield the invention of claim 24 above. Zhaoyu et al., Kirino et al. and Deyle et al. are considered analogous arts to the claimed invention as they disclose monitoring systems comprising radar sensors and cameras. Zhaoyu et al. as modified above discloses the monitoring system of claim 21. However, Zhaoyu et al. fails to explicitly disclose the notification information is information including a captured image corresponding to a predetermined criterion relating to the imaged object determined to be identical to the detected object and the imaged object attribute information. This feature is disclosed by Deyle et al. where in response to detecting a security violation, the robot provides the captured image and information describing the security violation to a security interface (Deyle et al. ¶ [0365]; Fig. 31). The combination of Zhaoyu et al., Kirino et al. and Deyle et al. would be obvious with a reasonable expectation of success to “provide a small-sized, highly efficient, and high-performance fusion apparatus.” (Kirino et al. ¶ [0316]) and improve the accuracy of target recognition (Kirino et al. ¶ [0316]) and reduce “the burden on an operator by automating robot functionality and consolidating robot information” and “allow the operator to prioritize her focus on high-risk or dangerous security events” (Deyle et al. ¶ [0323]).
Regarding claim 25 (New), Zhaoyu et al. as modified above discloses:
[Note: what is not explicitly taught by Zhaoyu et al. has been struck-through]
The monitoring system according to claim 24
Deyle et al. discloses:
wherein the predetermined criterion is set on a basis of at least one of a score indicating accuracy of the attribute (Deyle et al. “Many of these object detection algorithms classify a detected object by identifying a portion of the image in which the object is located (and may include a bounding box around the identified portion of the image), by identifying the object, and by providing a confidence score representative of how confident the object detection algorithm is that the object is correctly classified.” - ¶ [0324]; “In some embodiments, if the object is classified with a confidence score that is greater than a second confidence interval, the robot can present the captured image to the human operator with one or more “best guesses” of the classification of the object. For example, if the confidence score for an object being a table is 72%, and the confidence score for the object being a chair is 66%, the robot can present the captured image to a human operator with both candidate classifications (“table” and “chair”), along with the corresponding confidence scores.” - ¶ [0326]) or a size of an object.
It would have been obvious to someone with ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Deyle et al. into the invention of Zhaoyu et al. to yield the invention of claim 25 above. Zhaoyu et al., Kirino et al. and Deyle et al. are considered analogous arts to the claimed invention as they disclose monitoring systems comprising radar sensors and cameras. Zhaoyu et al. as modified above discloses the monitoring system of claim 24. However, Zhaoyu et al. fails to explicitly disclose wherein the predetermined criterion is set on a basis of at least one of a score indicating accuracy of the attribute or a size of an object. This feature is disclosed by Deyle et al. where the classification of the objects detected by the robot are provided with a confidence score (Deyle et al. ¶ [0324], [0326]). The combination of Zhaoyu et al., Kirino et al. and Deyle et al. would be obvious with a reasonable expectation of success to “provide a small-sized, highly efficient, and high-performance fusion apparatus.” (Kirino et al. ¶ [0316]) and improve the accuracy of target recognition (Kirino et al. ¶ [0316]) and reduce “the burden on an operator by automating robot functionality and consolidating robot information” and “allow the operator to prioritize her focus on high-risk or dangerous security events” (Deyle et al. ¶ [0323]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAOMI M WOLFORD whose telephone number is (571)272-3929. The examiner can normally be reached Monday - Friday, 8:30 am - 4:30 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Resha Desai can be reached at (571)270-7792. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
NAOMI M. WOLFORD
Examiner
Art Unit 3648
/N.M.W./Examiner, Art Unit 3648 29 MAY 2026
/VLADIMIR MAGLOIRE/Supervisory Patent Examiner, Art Unit 3648
Prosecution Insights