DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 9 and 13 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention. Specifically, claims 9 and 13 introduces “an overview camera” repeatedly without clarifying whether it is the same “overview camera” introduced in independent claim 1. Therefore, the claims are indefinite regarding antecedent basis. For examination, the “overview camera” in claims 9 and 13 will be treated as the same “overview camera” introduced in claim 1.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 and 14 are rejected are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., an abstract idea) without integration into a practical application or recitation of significantly more.
Independent claims 1 and 14 are directed to one of the four statutory categories of eligible subject matter (a process for independent claim 1; an apparatus for independent claim 14); thus, the claims pass Step 1 of the Subject Matter Eligibility Test (See flowchart in MPEP 2106).
Step 2A, prong 1 analysis:
Independent claims are directed to tracking objects in a plurality of image frames depicting the scene to produce current object tracks, detecting object candidates in an image frame subsequent to the plurality of image frames depicting the scene, calculating, for each object candidate, an association measure which is indicative of a likelihood that the object candidate is associated with a current object track, correlating a view with a heatmap of the scene, the heatmap of the scene providing data indicative of areas in the scene having an elevated degree of occurrence of historical object tracks, adjusting the association measure or an association threshold for object candidates which according to the heatmap are located in areas in the scene having an elevated degree of occurrence of historical object tracks so as to increase their probability of being associated with a current object track, and associating each object candidate with a current object track if the association measure is above an association threshold.
Each of the above steps can be performed mentally. In particular, a human uses their own vision to track and identify objects such as vehicles and moving pedestrians in a series of images or video and notes the path each object moves through time; the human figures out each movement path in the images/video and determines whether the objects seen such as vehicles or pedestrians are moving on the paths over time; the human creates a heat map (by hand with pen and paper) by observing historical data of objects moving in the area that is in the images/video and weights the objects (darker on the heat map means stronger weight) depending on how closely aligned with historical tracking traffic data movements the objects are; the objects that are aligned with historical movements with stronger weighting in the heart map are noted as being more primary objects for further tracking since they align with movement paths in line with historical movement patterns; if the paths of those specific objects are very closely aligned beyond a threshold then those objects (pedestrians or vehicles) are associated with that historical path.; this amounts to nothing more than a human observing traffic patterns and noting visual data and comparing the data to historical data for tracking objects with their human vision; therefore, this process can all be done mentally.
As such, the description in independent claims 1 and 14 is an abstract idea – namely, a mental process. Accordingly, the analysis under prong one of step 2A of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
Additional elements:
The additional element recited in independent claims 1 and 10 are an overview camera and a PTZ camera.
Step 2A, prong 2 analysis:
The above-identified additional elements do not integrate the judicial exception into a practical application. A computing system, a machine learning model, and data structures are generic computers that simply automate what a human doctor already does in prognosis and diagnosis of tumors looking at medical imaging data.
Each of the other additional elements (overview camera and a PTZ camera) amounts to merely using different devices as tools to perform the claimed mental process. Implementing an abstract idea on a computer or using known generic devices does not integrate a judicial exception into a practical application (See MPEP 2106.05(f)).
Moreover, the additional elements of the claims do not recite an improvement in the functioning of a computer or other technology or technical field, the claimed steps are not performed using a particular machine, the claimed steps do not effect a transformation, and the claims do not apply the judicial exception in any meaningful way beyond generically linking the use of the judicial exception to a particular technological environment (See MPEP 2106.04(d)). Therefore, the analysis under prong two of step 2A of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
Step 2B:
Finally, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Each of the other additional elements (overview camera and a PTZ camera) are generic computer features which perform generic computer functions that are well-understood, routine, and conventional and do not amount to more than implementing the abstract idea with a computerized system. Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea).
Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation, and mere implementation on a generic computer does not add significantly more to the claims. Accordingly, the analysis under step 2B of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
For all of the foregoing reasons, independent claims 1 and 14 do not recite eligible subject matter under 35 USC 101.
Claim 2 recites wherein the heatmap is generated using a PTZ-camera tracking objects in the scene over time and storing tracks followed by the objects. The human easily uses the PTZ camera and notes observations in the heat map made by hand; therefore, this process can all be done mentally.
Claim 3 recites wherein the heatmap is generated using a PTZ-camera tracking objects in the scene over time, and wherein the tracking is performed using intermittent or continuous reidentification to ensure a verified track from each individual object being tracked. Humans easily use the PTZ camera tracking objects and notes observations in the heat map made by hand; therefore, this process can all be done mentally.
Claim 4 recites wherein the heatmap includes position measurements for recorded object tracks. Humans create heat maps by hand the identify objects and observing a scene note the position of object; therefore, this process can all be done mentally.
Claim 5 recites wherein the heatmap includes velocity information for recorded object tracks. Humans use radar guns to observe traffic objects moving and records their movement paths alongside their velocities; therefore, this process can all be done mentally.
Claim 6 recites wherein the heatmap includes an object class or an object speed for recorded object tracks, so as to enable filtering on object class or object speed. Humans use radar guns to observe traffic objects moving and records their movement paths alongside their velocities and then includes the velocities of the objects in the heat map for each object tracking the object’s movements; therefore, this process can all be done mentally.
Claim 7 recites selecting heatmap data corresponding to an identified object class, or an identified object speed. Humans use radar guns to observe traffic objects moving and records their movement paths alongside their velocities and then includes the velocities of the objects in the heat map for each object tracking the object’s movements; therefore, this process can all be done mentally.
Claim 8 recites wherein the tracking of current objects comprises identifying an object class to which the tracked object belongs, or an object speed for the object. Humans uses radar guns to observe traffic objects moving and records their movement paths alongside their velocities and then includes the velocities of the objects in the heat map for each object tracking the object’s movements; therefore, this process can all be done mentally.
Claim 9 recites performed by a system comprising a PTZ-camera and an overview camera, wherein a view of overview camera is positionally calibrated with a view of the PTZ-camera. Humans calibrate cameras so their views are matched up to point at the same scene; therefore, this process can all be done mentally.
Claim 10 recites wherein during assembly of the heatmap tracking is also performed using image data from the overview camera, wherein an association measure for the tracking performed by the overview camera is monitored for tracks verified by the PTZ-camera. When drawing the heat map of observed objects in a traffic scene such as pedestrians or vehicles, humans use both an overview camera and PTZ camera to make observations where over time the movement paths of the objects from the overview camera is verified with the PTZ camera; therefore, this process can all be done mentally.
Claim 11 recites wherein the association threshold is lowered in areas of the heatmap comprising verified tracks. When the heatmap is made by humans, the threshold for comparing observed movement paths of objects in series of images/video over time is lowered, if objects move along tracks that are verified as similar to historical movement paths, then the threshold for indicating the objects as unique and require closer following is lowered because of the association with historical traffic data; therefore, this process can all be done mentally.
Claim 12 recites wherein the association measure is increased in areas of the heatmap comprising verified tracks, with distance between object detection and a path of the heatmap, or heatmap intensity, so as to enhance a likelihood of an object detection to be associated with a current track. If movement paths for observed traffic objects in series of images/video over time match historical traffic flow movements, the path is more strongly associated to the object/pedestrian/vehicle; therefore, this process can all be done mentally.
Claim 13 recites wherein a functionality of an overview camera is provided by a PTZ-camera in a zoomed-out mode. The human easily uses the PTZ camera and increases or decreases zoom accordingly as needed; therefore, this process can all be done mentally.
Therefore, dependent claims 2-13 recite the same abstract idea of a mental process which can be performed in the mind with the aid of pen and paper, and are therefore also rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No.: 2017/0236284 (Elliethy et al.) (hereinafter Elliethy), in view of U.S. Patent Application Publication No.: 2006/0222205 (Porikli et al.) (hereinafter Porikli).
Regarding claim 1, Elliethy teaches a method for improving tracking of objects in a scene, using an overview monitoring camera, comprising: (Elliethy, para. [0038]: “This disclosure focuses upon the registration of vector road map data with aerial imagery data, such as WAMI video frames, and includes novel algorithms that exploit vehicular motion for accurate and computationally efficient alignment of the respective data. Registering vector road map data with aerial imagery data leads to a rich source of geo-spatial information that can be used for many applications. One application of interest is moving vehicle detection and tracking. By registering the road network to aerial imagery, one can easily filter out false detections that occur off the road network. Another application is the detection and tracking of suspicious off-road traffic. These applications depend upon accurate alignments between aerial imagery and the road network, which is typically represented in a vector format that describes the roads mathematically as lines or curves connecting a series of geo-registered points. Such alignment of the aerial imagery with a geo-registered road network also directly provides an alignment between the aerial image and any prior geo-registered image”)
tracking objects in a plurality of image frames depicting the scene to produce current object tracks, detecting object candidates in an image frame subsequent to the plurality of image frames depicting the scene, calculating, for each object candidate, an association measure which is indicative of a likelihood that the object candidate is associated with a current object track (Elliethy, para. [0095]: “
PNG
media_image1.png
281
468
media_image1.png
Greyscale
PNG
media_image2.png
107
467
media_image2.png
Greyscale
”),
correlating a view of the overview camera with a heatmap of the scene (Elliethy, para. [0038]: “By registering the road network to aerial imagery, one can easily filter out false detections that occur off the road network”),
the heatmap of the scene providing data indicative of areas in the scene having an elevated degree of occurrence of object tracks (Elliethy, para. [0084]; para. [0095] (see above); para. [0039]:
PNG
media_image3.png
610
475
media_image3.png
Greyscale
;
“In general, successive WAMI video frames can be related by both global and local motions. Global motion arises from camera movement due to movement of the aerial platform, and can be parameterized as a homography between spatial coordinates for successive frames under the assumption that the captured scene is planar. Local motion arises due to local movement of objects within the captured scene. Local motion in WAMI datasets for urban areas is dominated by vehicle movements on the road network within the captured scene. Those movements can be exploited to develop an effective registration scheme for aligning vector road map data with WAMI video frames.”; one of ordinary skill in the art would understand that the road network provides data indicative of areas having an elevated degree of occurrence of object tracks and thus meets the broadest reasonable interpretation of the term “heatmap”)
adjusting the association measure or an association threshold for object candidates which according to the heatmap are located in areas in the scene having an elevated degree of occurrence of object tracks so as to increase their probability of being associated with a current object track (Elliethy, para. [0095]: “
Elliethy, para. [0095]: “
PNG
media_image1.png
281
468
media_image1.png
Greyscale
PNG
media_image2.png
107
467
media_image2.png
Greyscale
”;
Examiner is mapping the cited reference to “association measure” and not an association threshold), and
associating each object candidate with a current object track if the association measure is above an association threshold (Elliethy, para. [0096]: “
PNG
media_image4.png
312
463
media_image4.png
Greyscale
”).
Elliethy fails to teach
the heatmap of the scene providing data indicative of areas in the scene having an elevated degree of occurrence of historical object tracks.
Porikli teaches
the heatmap of the scene providing data indicative of areas in the scene having an elevated degree of occurrence of historical object tracks (Porikli, para. [0052]: “Alternatively, we select the initial set of possible kernel locations based on a pathway likelihood map that maintains a history of previously tracked objects. We increase the value of a pixel in the pathway likelihood map if the object kernel corresponds to the pixel. We keep updating the pathway likelihood map for each frame in the video. Thus, after objects have been tracked in a large number of frames, the pathway likelihood map indicates likely locations of objects.”).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the heatmap of the scene providing data indicative of areas in the scene having an elevated degree of occurrence of object tracks, as taught by Elliethy, provide data indicative of areas in the scene having an elevated degree of occurrence of historical object tracks, as taught by Porikli.
The suggestion/motivation for doing so would have been that a heatmap of a scene that aggregates historical object tracks translates raw, complex geospatial or surveillance data into a highly intuitive, color-coded visual layer; by highlighting areas with an "elevated degree of occurrence" (hotspots), this tool provides several critical benefits in computer vision, such as improved anomaly spotting, improved loitering and vulnerability assessment; a track-based historical monitoring heatmap aggregates how frequently moving entities pass through specific coordinates over time; one has the ability to distinguish a "bottleneck" (objects trying to move but stuck) from a "destination" (objects arriving and stopping intentionally) by including temporal/historical dimension of objects).
Elliethy, in view of Porikli, teaches adjusting the association measure or an association threshold for object candidates which according to the heatmap are located in areas in the scene having an elevated degree of occurrence of historical object tracks so as to increase their probability of being associated with a current object track (Elliethy, para. [0095]; Porikli, para. [0052]; incorporating the aspects of historical object tracks in heat maps from Porikli, into the heat map of Elliethy, allows for adjusting the association measure for object candidates so as to increase their probability of being associated with a current object track to improve and capture objects over time).
Therefore, it would have been obvious to combine Elliethy, with Porikli, to obtain the invention as specified in claim 1.
Regarding claim 4, Elliethy, in view of Porikli, teaches the method of claim 1, wherein the heatmap includes position measurements for recorded object tracks (Elliethy, para. [0084]; para. [0095] (see above); para. [0039]: see rejection of claim 1; the path map/road map indicates the locations where an object was tracked so the maps naturally include position measurements).
Regarding claim 12, Elliethy, in view of Porikli, teaches the method of claim 1, wherein the association measure is increased in areas of the heatmap comprising verified tracks (Elliethy, para. [0084]; para. [0095] (see above); para. [0039]; see rejection of claim 1 above regarding movement paths highlighted more strongly in the heatmap where traffic is more likely to flow).
Elliethy, in view of Porikli, fails to teach
verified tracks with distance between object detection and a path of the heatmap, or heatmap intensity, so as to enhance a likelihood of an object detection to be associated with a current track.
Porikli further teaches
verified tracks with distance between object detection and a path of the heatmap, or heatmap intensity, so as to enhance a likelihood of an object detection to be associated with a current track (Porikli, para. [0052]-[0053]: “Alternatively, we select the initial set of possible kernel locations based on a pathway likelihood map that maintains a history of previously tracked objects. We increase the value of a pixel in the pathway likelihood map if the object kernel corresponds to the pixel. We keep updating the pathway likelihood map for each frame in the video. Thus, after objects have been tracked in a large number of frames, the pathway likelihood map indicates likely locations of objects. For instance, for a traffic surveillance video, the likelihood pathway map has higher values for the pixels corresponding to the traffic lanes where objects, e.g., vehicles, are moving.”).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the verified tracks, as taught by Elliethy, in view of Porikli, be with distance between object detection and a path of the heatmap, or heatmap intensity, so as to enhance a likelihood of an object detection to be associated with a current track, as further taught by Porikli.
The suggestion/motivation for doing so would have been that in multi-object tracking systems, increasing the association measure in heatmap areas with verified tracks, based on distance to a path or heatmap intensity, directly improves tracking accuracy by reducing identity switches and false dismissals.
Therefore, it would have been obvious to combine Elliethy and Porikli, with Porikli further, to obtain the invention as specified in claim 12.
Claims 2-3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Elliethy, in view of Porikli, and in view of non-patent literature "Map interface for geo-registering and monitoring distributed events"; 13th International IEEE Conference on Intelligent Transportation Systems; IEEE, 2010 (Morris et al.) (hereinafter Morris).
Regarding claim 2, Elliethy, in view of Porikli, teaches the method of claim 1.
Elliethy, in view of Porikli, fails to teach
a PTZ-camera tracking objects in the scene over time and storing tracks followed by the objects.
Morris teaches
a PTZ-camera tracking objects in the scene over time and storing tracks followed by the objects (Morris, page 962, right-side col., para. 4; FIG. 3: “The main sensing modality for CANVAS are video cameras. Fig. 3 shows a map of UCSD along with images of the many camera nodes situated around campus. A variety of environments, both indoor and outdoor, with different coverage area, scale, and objects of interest are present. Both pan-tilt-zoom (PTZ) and wide are covering omni-directional cameras are utilized to monitor highway traffic along Interstate 5, human/vehicle interactions on campus roads, and people indoors. Most video processing is performed remotely by transmitting video data across the network. Non-streaming cameras with a local capture machine can be used to limit the bandwidth requirements necessary from very-large scale video networks by transmitting just archival data.”;
PNG
media_image5.png
456
971
media_image5.png
Greyscale
).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the method, as taught by Elliethy, in view of Porikli, to include the step of tracking objects in the scene over time and storing tracks followed by the objects using a PTZ -camera, as taught by Morris.
The suggestion/motivation for doing so would have been that a PTZ-camera (pan-tilt-zoom) replaces multiple fixed cameras by moving horizontally, vertically, and zooming in to eliminate blind spots across wide spaces for complete area coverage, and provides dynamic, close-up visual details of moving subjects across a vast area, whereas a regular (fixed) camera only captures a static field of view.
Elliethy, in view of Porikli, and in view of Morris teaches wherein the heatmap is generated using a PTZ-camera tracking objects in the scene over time and storing tracks followed by the objects (Elliethy, para. [0084]; para. [0095]; para. [0039]; Morris, page 962, right-side col., para. 4; FIG. 3; Elliethy teaches that the road network provides data indicative of areas having an elevated degree of occurrence of object tracks and thus meets the broadest reasonable interpretation of the term “heatmap”; the method of Elliethy operates if the images come from a PTZ camera, as taught by Morris; the main difference between a regular camera and a PTZ camera is a PTZ (Pan, Tilt, Zoom) camera can move left/right, up/down, and magnify images, whereas a regular (fixed) camera stays in one stationary position with a set field of view; as long as the moving target coordinates are projected into a static reference frame, then PTZ camera from Morris is easily adapted to the road network “heatmap” of Elliethy).
Therefore, it would have been obvious to combine Elliethy and Porikli, with Morris, to obtain the invention as specified in claim 2.
Regarding claim 3, Elliethy, in view of Porikli, teaches the method of claim 1, wherein the tracking is performed using intermittent or continuous reidentification to ensure a verified track from each individual object being tracked (Elliethy, para. [0074]: “Specifically, the third algorithm estimates, via an alternating optimization, vehicular trajectories over a multi-frame temporal window (typically 10-15 frames) and the best geometric transformation for aligning those trajectories with the road network. The algorithm may be implemented using a maximum a posteriori probability (MAP) formulation that penalizes trajectory deviations from the road network using a chamfer distance metric, appropriately modified for the problem setting to incorporate directionality, as well as a successive approach to identifying and extending reliable trajectories for individual vehicles based on detections in individual frames and the alignment of the oriented trajectories with the road network directionality.”; it is noted that monitoring an area is conventionally done in order to track specific vehicles (e.g. broken down vehicles, speeding, driving in the wrong direction), which is enables by para. [0074] of Elliethy above; the identification of a vehicle in a new frame to extend the track represents a continuous reidentification; a track is reestablished if, for example, in a case a track is interrupted (e.g. by a tree)”).
Elliethy, in view of Porikli, fails to teach
using a PTZ-camera tracking objects in the scene over time.
Morris teaches
using a PTZ-camera tracking objects in the scene over time (Morris, page 962, right-side col., para. 4; FIG. 3: “The main sensing modality for CANVAS are video cameras. Fig. 3 shows a map of UCSD along with images of the many camera nodes situated around campus. A variety of environments, both indoor and outdoor, with different coverage area, scale, and objects of interest are present. Both pan-tilt-zoom (PTZ) and wide are covering omni-directional cameras are utilized to monitor highway traffic along Interstate 5, human/vehicle interactions on campus roads, and people indoors. Most video processing is performed remotely by transmitting video data across the network. Non-streaming cameras with a local capture machine can be used to limit the bandwidth requirements necessary from very-large scale video networks by transmitting just archival data.”;
PNG
media_image5.png
456
971
media_image5.png
Greyscale
).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the method, as taught by Elliethy, in view of Porikli, to include the step of using a PTZ-camera tracking objects in the scene over time, as taught by Morris.
The suggestion/motivation for doing so would have been that a PTZ-camera (pan-tilt-zoom) replaces multiple fixed cameras by moving horizontally, vertically, and zooming in to eliminate blind spots across wide spaces for complete area coverage, and provides dynamic, close-up visual details of moving subjects across a vast area, whereas a regular (fixed) camera only captures a static field of view.
Elliethy, in view of Porikli, and in view of Morris teaches wherein the heatmap is generated using a PTZ-camera tracking objects in the scene over time (Elliethy, para. [0084]; para. [0095]; para. [0039]; Morris, page 962, right-side col., para. 4; FIG. 3; Elliethy teaches that the road network provides data indicative of areas having an elevated degree of occurrence of object tracks and thus meets the broadest reasonable interpretation of the term “heatmap”; the method of Elliethy operates if the images come from a PTZ camera, as taught by Morris; the main difference between a regular camera and a PTZ camera is a PTZ (Pan, Tilt, Zoom) camera can move left/right, up/down, and magnify images, whereas a regular (fixed) camera stays in one stationary position with a set field of view; as long as the moving target coordinates are projected into a static reference frame, then PTZ camera from Morris is easily adapted to the road network “heatmap” of Elliethy).
Therefore, it would have been obvious to combine Elliethy and Porikli, with Morris, to obtain the invention as specified in claim 3.
Regarding claim 13, Elliethy, in view of Porikli, teaches the method of claim 1.
Elliethy, in view of Porikli, fails to teach
wherein a functionality of an overview camera is provided by a PTZ-camera in a zoomed-out mode.
Morris teaches
wherein a functionality of an overview camera is provided by a PTZ-camera in a zoomed-out mode time (Morris, page 962, right-side col., para. 4; FIG. 3: “The main sensing modality for CANVAS are video cameras. Fig. 3 shows a map of UCSD along with images of the many camera nodes situated around campus. A variety of environments, both indoor and outdoor, with different coverage area, scale, and objects of interest are present. Both pan-tilt-zoom (PTZ) and wide are covering omni-directional cameras are utilized to monitor highway traffic along Interstate 5, human/vehicle interactions on campus roads, and people indoors. Most video processing is performed remotely by transmitting video data across the network. Non-streaming cameras with a local capture machine can be used to limit the bandwidth requirements necessary from very-large scale video networks by transmitting just archival data.”;
PNG
media_image5.png
456
971
media_image5.png
Greyscale
”;
A PZT camera is a Pan-Tilt-Zoom camera and therefore implicitly has zoom in and out capabilities).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the overview camera, as taught by Elliethy, in view of Porikli, to include functionality of an overview camera is provided by a PTZ-camera in a zoomed-out mode time, as taught by Morris.
The suggestion/motivation for doing so would have been that a PTZ-camera (pan-tilt-zoom) replaces multiple fixed cameras by moving horizontally, vertically, and zooming in to eliminate blind spots across wide spaces for complete area coverage, and provides dynamic, close-up visual details of moving subjects across a vast area, whereas a regular (fixed) camera only captures a static field of view.
Therefore, it would have been obvious to combine Elliethy and Porikli, with Morris, to obtain the invention as specified in claim 13.
Claims 5-8 are rejected under 35 U.S.C. 103 as being unpatentable over Elliethy, in view of Porikli, and in view of non-patent literature "Traffic-Net: 3D traffic monitoring using a single camera. arXiv 2021"; arXiv preprint arXiv:2109.09165 (2021) (Rezaei et al.) (hereinafter Rezaei).
Regarding claim 5, Elliethy, in view of Porikli, teaches the method of claim 1.
Elliethy, in view of Porikli, fails to teach
wherein the heatmap includes velocity information for recorded object tracks.
Rezaei teaches
wherein the heatmap includes velocity information for recorded object tracks (Rezaei, page 15, left-side col., para. 2-4; right-side col., para. 1; FIG. 17A;
PNG
media_image6.png
276
437
media_image6.png
Greyscale
;
PNG
media_image7.png
363
416
media_image7.png
Greyscale
PNG
media_image8.png
165
424
media_image8.png
Greyscale
”).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the heatmap, as taught by Elliethy, in view of Porikli, to include velocity information for recorded object tracks, as taught by Rezaei.
The suggestion/motivation for doing so would have been that Including velocity information in a heat map transforms a standard spatial density chart into a dynamic behavioral map, providing critical insights into how tracked objects move, accelerate, and interact over time; Standard heat maps look identical for an area where objects move slowly but steadily and an area where objects are completely stuck, while velocity allows you to isolate true "dwell time" (zero or near-zero speed) from continuous high-traffic throughput corridors.
Therefore, it would have been obvious to combine Elliethy and Porikli, with Rezaei, to obtain the invention as specified in claim 5.
Regarding claim 6, Elliethy, in view of Porikli, teaches the method of claim 1.
Elliethy, in view of Porikli, fails to teach
wherein the heatmap includes an object class or an object speed for recorded object tracks, so as to enable filtering on object class or object speed.
Rezaei teaches
wherein the heatmap includes an object class or an object speed for recorded object tracks, so as to enable filtering on object class or object speed (Rezaei, page 15, left-side col., para. 2-4; right-side col., para. 1; FIG. 17A; page 4, right-side col., para. 3-4; page 9, right-side col.:
PNG
media_image6.png
276
437
media_image6.png
Greyscale
;
PNG
media_image7.png
363
416
media_image7.png
Greyscale
PNG
media_image8.png
165
424
media_image8.png
Greyscale
”; “
PNG
media_image9.png
459
525
media_image9.png
Greyscale
”; “
PNG
media_image10.png
734
497
media_image10.png
Greyscale
”).
for road surveillance not only vehicles as objects but also pedestrians are detected and thus different likelihood pathways maps are generated; thus, dependent on the object, which one is tracked depends on what is appropriate at the time for one of ordinary skill in the art using the process).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the heatmap, as taught by Elliethy, in view of Porikli, to include includes an object class or an object speed for recorded object tracks, so as to enable filtering on object class or object speed, as taught by Rezaei.
The suggestion/motivation for doing so would have been that Including velocity information in a heat map transforms a standard spatial density chart into a dynamic behavioral map, providing critical insights into how tracked objects move, accelerate, and interact over time; Standard heat maps look identical for an area where objects move slowly but steadily and an area where objects are completely stuck, while velocity allows you to isolate true "dwell time" (zero or near-zero speed) from continuous high-traffic throughput corridors.
Therefore, it would have been obvious to combine Elliethy and Porikli, with Rezaei, to obtain the invention as specified in claim 6.
Regarding claim 7, Elliethy, in view of Porikli, teaches the method of claim 6, further comprising selecting heatmap data corresponding to an identified object class, or an identified object speed (Rezaei, page 15, left-side col., para. 2-4; right-side col., para. 1; FIG. 17A; page 4, right-side col., para. 3-4; page 9, right-side col.; see rejection of claim 6 above discussing the heatmaps and objects identified in the heat map along with their corresponding speed/velocities).
Regarding claim 8, Elliethy, in view of Porikli, teaches the method of claim 7, wherein the tracking of current objects comprises identifying an object class to which the tracked object belongs, or an object speed for the object (Rezaei, page 15, left-side col., para. 2-4; right-side col., para. 1; FIG. 17A; page 4, right-side col., para. 3-4; page 9, right-side col.; see rejection of claims 6-7 above; the different object classes include pedestrians and vehicles along side with identifying the speed/velocity associated with the object).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Elliethy, in view of Porikli, and in view of U.S. Patent Application Publication No.: 2019/0066335 (Dahlstrom et al.) (hereinafter Dahlstrom).
Regarding claim 9, Elliethy, in view of Porikli teaches the method of claim 1.
Elliethy, in view of Porikli, fails to teach
performed by a system comprising a PTZ-camera and an overview camera, wherein a view of overview camera is positionally calibrated with a view of the PTZ-camera.
Dahlstrom teaches
performed by a system comprising a PTZ-camera and an overview camera, wherein a view of overview camera is positionally calibrated with a view of the PTZ-camera (Dahlstrom, abstract: “A method of calibrating a pan, tilt, zoom (PTZ) camera with a fixed camera utilizing an overview image of a scene captured by the fixed camera, and an image of the scene captured by the PTZ camera when directed in a first direction. By matching features in the overview image and the PTZ camera image, a first calibration is carried out by correlating the first direction to matching features in the overview image. A mapping between the PTZ camera image and the overview image is defined based on the matching features. The mapping is used to map an object from the PTZ camera image to the overview image. Based on an appearance of the mapped object, a quality of the mapping is calculated. If the quality is not good enough, the PTZ camera is redirected to a second direction, and a further calibration is carried out by again.”).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the method, as taught by Elliethy, in view of Porikli, to be performed by a system comprising a PTZ-camera and an overview camera, wherein a view of overview camera is positionally calibrated with a view of the PTZ-camera, as taught by Dahlstrom).
The suggestion/motivation for doing so would have been that “the PTZ camera may thus be redirected until a mapping of good enough quality is achieved; in this way, the precision of the calibration may be further improved” (Dahlstrom, para. [0025).
Therefore, it would have been obvious to combine Elliethy and Porikli, with Dahlstrom, to obtain the invention as specified in claim 9.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Elliethy, in view of Porikli, and in view of Morris.
Regarding claim 14, Elliethy teaches a camera system comprising an overview camera, configured to perform a method for improving tracking of objects in a scene, comprising: (Elliethy, para. [0038]: “This disclosure focuses upon the registration of vector road map data with aerial imagery data, such as WAMI video frames, and includes novel algorithms that exploit vehicular motion for accurate and computationally efficient alignment of the respective data. Registering vector road map data with aerial imagery data leads to a rich source of geo-spatial information that can be used for many applications. One application of interest is moving vehicle detection and tracking. By registering the road network to aerial imagery, one can easily filter out false detections that occur off the road network. Another application is the detection and tracking of suspicious off-road traffic. These applications depend upon accurate alignments between aerial imagery and the road network, which is typically represented in a vector format that describes the roads mathematically as lines or curves connecting a series of geo-registered points. Such alignment of the aerial imagery with a geo-registered road network also directly provides an alignment between the aerial image and any prior geo-registered image”)
tracking objects in a plurality of image frames depicting the scene to produce current object tracks, detecting object candidates in an image frame subsequent to the plurality of image frames depicting the scene, calculating, for each object candidate, an association measure which is indicative of a likelihood that the object candidate is associated with a current object track (Elliethy, para. [0095]: “
PNG
media_image1.png
281
468
media_image1.png
Greyscale
PNG
media_image2.png
107
467
media_image2.png
Greyscale
”),
correlating a view of the overview camera with a heatmap of the scene (Elliethy, para. [0038]: “By registering the road network to aerial imagery, one can easily filter out false detections that occur off the road network”),
the heatmap of the scene providing data indicative of areas in the scene having an elevated degree of occurrence of object tracks (Elliethy, para. [0084]; para. [0095] (see above); para. [0039]:
PNG
media_image3.png
610
475
media_image3.png
Greyscale
;
“In general, successive WAMI video frames can be related by both global and local motions. Global motion arises from camera movement due to movement of the aerial platform, and can be parameterized as a homography between spatial coordinates for successive frames under the assumption that the captured scene is planar. Local motion arises due to local movement of objects within the captured scene. Local motion in WAMI datasets for urban areas is dominated by vehicle movements on the road network within the captured scene. Those movements can be exploited to develop an effective registration scheme for aligning vector road map data with WAMI video frames.”; one of ordinary skill in the art would understand that the road network provides data indicative of areas having an elevated degree of occurrence of object tracks and thus meets the broadest reasonable interpretation of the term “heatmap”)
adjusting the association measure or an association threshold for object candidates which according to the heatmap are located in areas in the scene having an elevated degree of occurrence of object tracks so as to increase their probability of being associated with a current object track (Elliethy, para. [0095]: “
Elliethy, para. [0095]: “
PNG
media_image1.png
281
468
media_image1.png
Greyscale
PNG
media_image2.png
107
467
media_image2.png
Greyscale
”;
Examiner is mapping the cited reference to “association measure” and not an association threshold), and
associating each object candidate with a current object track if the association measure is above an association threshold (Elliethy, para. [0096]: “
PNG
media_image4.png
312
463
media_image4.png
Greyscale
”).
Elliethy fails to teach
the heatmap of the scene providing data indicative of areas in the scene having an elevated degree of occurrence of historical object tracks.
Porikli teaches
the heatmap of the scene providing data indicative of areas in the scene having an elevated degree of occurrence of historical object tracks (Porikli, para. [0052]: “Alternatively, we select the initial set of possible kernel locations based on a pathway likelihood map that maintains a history of previously tracked objects. We increase the value of a pixel in the pathway likelihood map if the object kernel corresponds to the pixel. We keep updating the pathway likelihood map for each frame in the video. Thus, after objects have been tracked in a large number of frames, the pathway likelihood map indicates likely locations of objects.”).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the heatmap of the scene providing data indicative of areas in the scene having an elevated degree of occurrence of object tracks, as taught by Elliethy, provide data indicative of areas in the scene having an elevated degree of occurrence of historical object tracks, as taught by Porikli.
The suggestion/motivation for doing so would have been that a heatmap of a scene that aggregates historical object tracks translates raw, complex geospatial or surveillance data into a highly intuitive, color-coded visual layer; by highlighting areas with an "elevated degree of occurrence" (hotspots), this tool provides several critical benefits in computer vision, such as improved anomaly spotting, improved loitering and vulnerability assessment; a track-based historical monitoring heatmap aggregates how frequently moving entities pass through specific coordinates over time; one has the ability to distinguish a "bottleneck" (objects trying to move but stuck) from a "destination" (objects arriving and stopping intentionally) by including temporal/historical dimension of objects).
Elliethy, in view of Porikli, teaches adjusting the association measure or an association threshold for object candidates which according to the heatmap are located in areas in the scene having an elevated degree of occurrence of historical object tracks so as to increase their probability of being associated with a current object track (Elliethy, para. [0095]; Porikli, para. [0052]; incorporating the aspects of historical object tracks in heat maps from Porikli, into the heat map of Elliethy, allows for adjusting the association measure for object candidates so as to increase their probability of being associated with a current object track to improve and capture objects over time).
Elliethy, in view of Porikli, fails to teach
a camera system comprising a PTZ camera.
Morris teaches
a camera system comprising a PTZ camera (Morris, page 962, right-side col., para. 4; FIG. 3: “The main sensing modality for CANVAS are video cameras. Fig. 3 shows a map of UCSD along with images of the many camera nodes situated around campus. A variety of environments, both indoor and outdoor, with different coverage area, scale, and objects of interest are present. Both pan-tilt-zoom (PTZ) and wide are covering omni-directional cameras are utilized to monitor highway traffic along Interstate 5, human/vehicle interactions on campus roads, and people indoors. Most video processing is performed remotely by transmitting video data across the network. Non-streaming cameras with a local capture machine can be used to limit the bandwidth requirements necessary from very-large scale video networks by transmitting just archival data.”;
PNG
media_image5.png
456
971
media_image5.png
Greyscale
).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the camera system, as taught by Elliethy, in view of Porikli, to include a PTZ -camera, as taught by Morris.
The suggestion/motivation for doing so would have been that a PTZ-camera (pan-tilt-zoom) replaces multiple fixed cameras by moving horizontally, vertically, and zooming in to eliminate blind spots across wide spaces for complete area coverage, and provides dynamic, close-up visual details of moving subjects across a vast area, whereas a regular (fixed) camera only captures a static field of view.
Therefore, it would have been obvious to combine Elliethy, with Porikli and Morris, to obtain the invention as specified in claim 14.
Conclusion
Claims 10-11 are rejected under 35 U.S.C. 101 but not 35 U.S.C 102 and 103 prior art.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL ADAM SHARIFF whose telephone number is 571-272-9741. The examiner can normally be reached M-F 8:30-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached on 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL ADAM SHARIFF/
Examiner, Art Unit 2672
Prosecution Insights