DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgement is made of Applicant’s claim of priority from Foreign Application No. EP23184034.9, filed July 7, 2023.
Information Disclosure Statement
The information disclosure statement (“IDS”) filed on June 10, 2026 was reviewed and the listed references were noted.
Status of Claims
Claims 1-20 are pending. Claims 17-20 are newly added.
Response to Arguments
Applicant's arguments filed June 11, 2026 with respect to the 35 USC 103 rejections have been fully considered but they are not persuasive.
Applicant argues that the references do not teach the limitation “characterized in that the method comprises building a map of one or more occluded areas in the one or more video sequences, wherein the map comprises connections between a plurality of areas in the one or more video sequences, wherein a connection is established by successfully resuming a lost object track” of claim 1. Specifically, Applicant argues that the Das reference does not comprise connections established by successfully resuming lost object tracks, and that the map building and tracker resurrection of Das are functionally completely separate. Examiner respectfully disagrees. As described in Col. 14 line 47 – Col. 15 line 21 of Das, the tracker resurrection is used in times when full occlusions occur on the occlusion/motion map. In these cases, a tracker is lost and later a new object detection is created. To handle this scenario, Das uses a tracker resurrection algorithm to establish a connection between a new object detection and a lost object by resuming the track of the lost object as the new object. All historical information of the previous tracker are passed onto the new tracker, and essentially, the old tracker is resurrected. Thus, Examiner asserts that the tracker resurrection of Das is not completely separate from the map building, but rather occurs when an object is lost in the map. Additionally, Das is not relied upon to teach adding the resumed connection to the map, but rather determining a fist area where an object track is lost and a second area where the object track is resumed.
Applicant further argues that the Kobayashi reference cannot be relied upon to teach resuming a lost track by establishing a connection in a map of occluded areas. Examiner respectfully disagrees. As illustrated in Fig. 7 of Kobayashi when a player is occluded for a time, a connection can be made between the time when the tracking was lost and the time when it was resumed such that the map identifies that an object being lost in a first area has been resumed in the second area. Regardless of whether this connection of Kobayashi is generated based on GPS data of the lost object, in combination with the teachings of Das to resurrect a lost tracker without the use of the GPS data, the references are sufficient to teach the limitation because under the broadest reasonable interpretation of ” wherein the map comprises connections between a plurality of areas in the one or more video sequences, wherein a connection is established by successfully resuming a lost object track” Kobayashi’s map (Fig. 7) establishes connections between a lost object track and displays this in a map. It would have been obvious to one having ordinary skill in the art to combine this with Das’s resurrection of lost object trackers. Thus, the 35 USC 103 rejection of claim 1 is maintained.
Applicant further argues that dependent claims 2-15 are not taught by the prior art by nature of their dependency on claim 1, and because none of the Tokmakov, Chen, Beller, Li, Garoutte and Mande references cure the deficiencies of Das, Coffman and Kobayashi. Examiner respectfully disagrees. Claim 1 is taught by the prior art of record as discussed above (see p. 2-4, Paragraph 5) and as shown in the 35 USC 103 rejections below. Thus, the 35 USC 103 rejection of the claims is maintained, and consequently, THIS ACTION IS FINAL.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 16-17 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Das et al. (US 11,069,072 B2) in view of Coffman et al. (US 2011/0116684 A1) further in view of Kobayashi et al. (US 2018/0374222 A1).
Regarding claim 1, Das teaches a computer-implemented method for detecting one or more occluded areas in one or more video sequences analyzed by an object tracking system, the method comprising:
providing one or more video sequences, wherein the one or more video sequences are depicting a same scene, the one or more video sequences comprising a plurality of objects (Das, Col. 8, lines 16-41, the videos captured by motion capturing devices may be offline videos for which analytics may be performed by the system for detecting and tracking moving objects);
determining a plurality of object tracks in the one or more video sequences (Das, Col. 12, lines 54-67, The optical flow tracks are accumulated during the course of the lifetime of an object in view),
characterized in that the method comprises building a map of one or more occluded areas in the one or more video sequences (Das, Col. 13, lines 32-41, the one or more hardware processors iteratively generate, using an occlusion map technique, a motion map to identify one or more moving objects along with depth information from the plurality of moving objects of interest), and wherein building the map comprises:
a) upon determining that a first object track among the plurality of object tracks is lost, the first object track corresponding to a first object among the plurality of objects, running a re-identification algorithm on at least one video sequence of the one or more video sequences to try to resume the first object track (Das. Col. 14 line 50 – Col. 15 line 21, when the system fails to detect/track (or iteratively redetect) the object-of-interest (or moving object(s)), possibly due to environmental variations, low contrast etc., the tracker for that object is lost. When even the object-of-interest is re-detected at a later point in time, it creates a new tracker object. This scenario is undesirable because the system essentially counts the object twice and its previous tracked history is lost. To handle such scenarios, the concept of resurrection model/tracker resurrection algorithm is introduced. When a tracker is deleted in an area of the image where a natural termination of a tracker object is not expected, the particular tracker object is added to a list of trackers needing potential resurrection. Similarly, when a new detection is observed in an area where sudden new detection are not expected, all new detections are initially tracked temporarily and after confirmation these are added to a list of possible resurrection matches. The two lists are compared at any time to check the spatial, temporal, and dynamic properties (speed, direction of motion, aspect ratio, area, time and location of old tracker (outdated or expired) and new tracker generated and the like) to confirm whether the two trackers are essentially the same physical object. If confirmed so, all the historical information of the previous tracker are passed onto the new tracker, and essentially, the old tracker is resurrected);
b) upon successfully resuming the first object track:
determining a first area where the first object track is lost, wherein the first area is determined in the one or more video sequences among the plurality of areas in the one or more video sequences, and a second area where the first object track is resumed, wherein the second area is determined in the one or more video sequences among the plurality of areas in the one or more video sequences (Das, Col. 14 line 50 – Col. 15 line 21, when a tracker is deleted in an area of the image where a natural termination of a tracker object is not expected, the particular tracker object is added to a list of trackers needing potential resurrection (i.e., first area where the first object track is lost). Similarly, when a new detection is observed in an area where sudden new detection are not expected (i.e., second area where the first object track is resumed), all new detections are initially tracked temporarily and after confirmation these are added to a list of possible resurrection matches),
wherein the one or more video sequences are associated with a base coordinate system to which objects and areas depicted in image frames of the each video sequence are transformed (Das, Col. 19, lines 17-34, The camera calibration system (comprised in the system 100) derives the perspective corrected scales for x, y and z coordinates in the image. This scales allow it to convert the pixel shift into real world units like distance-per-time).
Although Das teaches capturing one or more videos (Das, Col. 8, lines 16-41), Das does not explicitly teach “wherein each area of the plurality of areas in the one or more video sequences refers to an area in a 2D representation of a 3D area of the scene as captured by the one or more video sequences”. However, in an analogous field of endeavor, Coffman teaches a visual sensor that may produce of 2D image of a 3D object (i.e., areas in video sequences refer to an area in a 2D representation of a 3D area) (Coffman, Para. [0050]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date to modify the method of Das with the teachings of Coffman by including that the areas captured in the images are 2D representations of 3D areas. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for tracking objects in the presence of occlusions, as recognized by Coffman.
Although Das in view of Coffman teaches determining the first and second areas in the video stream (Das, Col. 14 line 50 – Col. 15 line 21), they do not explicitly teach “wherein the map comprises connections between a plurality of areas in the one or more video sequences, wherein a connection is established by successfully resuming a lost object track” and “adding a connection between the first and the second area to the map of one or more occluded areas, such that the map identifies that an object track being lost in the first area has been resumed in the second area”. However, in an analogous field of endeavor, Kobayashi teaches there is a time period in which a player having a position sensor is temporarily not in the video captured by a camera because of occlusion, for example, a degree of overlap between a single second trajectory L2_1 mapped in the field coordinate system and two first trajectories L1_1 and L1_2 is high, and the two first trajectories L1_1 and L1_2 have continuity through the second trajectory L2_1 as illustrated in FIG. 7. In this case, the estimation function 16 integrates the two first trajectories L1_1 and L1_2 with the second trajectory L2_1 as a motion trajectory of one player in the field coordinate system. With this integration, the motion trajectory of the player in a time period of disconnection between the two first trajectories L1_1 and L1_2 that are motion trajectories of the player, that is, the motion trajectory of the player in a time period in which the player is not detected from the video because of occlusion and thus the first trajectory L1 is not generated, can be interpolated by using the second trajectory L2_1 mapped in the field coordinate system (Kobayashi, Para. [0039]; Fig. 7).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date to modify the method of Das in view of Coffman with the teachings of Kobayashi by including a map includes connections made after resuming an object track and adding a connection between the first area (i.e., L1_1 in Fig. 7) and the second area (i.e., L1_2 in Fig. 7) such that the map identifies that an object track being lost in the first area has been resumed in the second area (i.e., L2_1 in Fig. 7). One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for correctly analyzing the motion of a moving object, as recognized by Kobayashi. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Regarding claim 2, Das in view of Coffman further in view of Kobayashi teaches the method of claim 1, wherein building the map comprises performing step a) and b) on a plurality of lost object tracks among the plurality of object tracks (Das, Col. 14 line 50 – Col. 15 line 21, when a tracker is deleted in an area of the image where a natural termination of a tracker object is not expected, the particular tracker object is added to a list of trackers needing potential resurrection. Similarly, when a new detection is observed in an area where sudden new detection are not expected, all new detections are initially tracked temporarily and after confirmation these are added to a list of possible resurrection matches (i.e., plurality of lost object tracks)).
Claim 16 recites a computer-readable storage medium storing a program with instructions corresponding to the steps recited in Claim 1. Therefore, the recited programming instructions of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Das, Coffman and Kobayashi references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Das, Coffman and Kobayashi references discloses a computer readable storage medium (Das, Col. 21, lines 31-41, a computer-usable or computer readable medium can be any apparatus that can comprise, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device).
Regarding claim 17, Das in view of Coffman further in view of Kobayashi teaches the method of claim 1, wherein the map of one or more occluded areas are used to improve object tracking when a threshold number of object tracks have been lost and resumed (Das, Col. 13, lines 32-54, the one or more hardware processors iteratively generate, using an occlusion map technique, a motion map to identify one or more moving objects along with depth information from the plurality of moving objects of interest. The motion map is generated not only to identify moving objects, but also to handle fully or partially occluded objects).
Regarding claim 19, Das in view of Coffman further in view of Kobayashi teaches the method of claim 1, wherein the map of one or more occluded areas is built and adjusted in real time while the object tracking system tracks objects in the scene (Das, Col. 8, lines 14-41, the plurality of motion streams may be captured in real-ti8me and processed by the system for video analytics and moving objects detection and tracking in real-time).
Regarding claim 20, Das in view of Coffman further in view of Kobayashi teaches the method of claim 1, further comprising: upon determining that a second object is occluded, determining an estimated trajectory of the second object while being occluded based on the map of one or more occluded areas (Das, Col. 14 line 50 – Col. 15 line 21, when a tracker is deleted in an area of the image where a natural termination of a tracker object is not expected, the particular tracker object is added to a list of trackers needing potential resurrection. Similarly, when a new detection is observed in an area where sudden new detection are not expected, all new detections are initially tracked temporarily and after confirmation these are added to a list of possible resurrection matches).
Claims 3-4 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Das et al. (US 11,069,072 B2) in view of Coffman et al. (US 2011/0116684 A1) further in view of Kobayashi et al. (US 2018/0374222 A1), as applied to claims 1-2, 16-17 and 19-20 above, and further in view of Tokmakov et al. (US 12,236,688 B2).
Regarding claim 3, Das in view of Coffman further in view of Kobayashi teaches the method of claim 2, as described above.
Although Das in view of Coffman further in view of Kobayashi teaches determining possible resurrection matches after occlusion (Das, Col. 14 line 50 – Col. 15 line 21), they do not explicitly teach “wherein the map of one or more occluded areas further indicates a probability that an object track being lost in the first area is resumed in the second area”. However, in an analogous field of endeavor, Tokmakov teaches that in cases of occlusion, there are many potential paths through the graph that link the object’s locations before and after occlusion. By minimizing the RAM objective, the probabilities are shifted towards the paths which are most likely to result in correctly localizing the target object when it re-appears without enforcing any single hypothesis (Tokmakov, Col. 10, lines 14-27).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date to modify the method of Das in view of Coffman further in view of Kobayashi with the teachings of Tokmakov by including probabilities that an object track being lost in a first area is resumed in the second area. One having ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to combine these references because doing so would allow for estimating a trajectory of an occluded object without ignoring or breaking the tracking during occlusion, as recognized by Tokmakov. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Regarding claim 4, Das in view of Coffman further in view of Kobayashi and Tokmakov teaches the method of claim 3, wherein the map indicates that an object track being lost in the first area has been resumed in a subset of areas in the one or more video sequences, the subset comprising at least two areas in the one or more video sequences among the plurality of areas in the one or more video sequences (Das, Col. 14 line 50 – Col. 15 line 21, when a tracker is deleted in an area of the image where a natural termination of a tracker object is not expected, the particular tracker object is added to a list of trackers needing potential resurrection. Similarly, when a new detection is observed in an area where sudden new detection are not expected, all new detections are initially tracked temporarily and after confirmation these are added to a list of possible resurrection matches),
wherein the map further indicates for each area in the subset, the probability that an object track being lost in the first area is resumed in that area in the subset (Tokmakov, Col. 10, lines 14-27, in cases of occlusion, there are many potential paths through the graph that link the object’s locations before and after occlusion. By minimizing the RAM objective, the probabilities are shifted towards the paths which are most likely to result in correctly localizing the target object when it re-appears without enforcing any single hypothesis).
The proposed combination as well as the motivation for combining the Das, Coffman, Kobayashi and Tokmakov references presented in the rejection of Claim 3, apply to Claim 4 and are incorporated herein by reference. Thus, the method recited in Claim 4 is met by Das in view of Coffman further in view of Kobayashi and Tokmakov.
Regarding claim 11, Das in view of Coffman further in view of Kobayashi and Tokmakov teaches the method of claim 4, wherein the sensor data is one of: radar data, lidar data, or video data (Coffman, Para. [0050], the visual sensor includes a sensor that may produce a 2D image of a 3D object and includes passive staring visual sensors such as, for example, the common visible-light camera and video camera and other sensors that capture images in the visible light spectrum).
The proposed combination as well as the motivation for combining the Das, Coffman, Kobayashi and Tokmakov references presented in the rejection of Claim 3, apply to Claim 11 and are incorporated herein by reference. Thus, the method recited in Claim 11 is met by Das in view of Coffman further in view of Kobayashi and Tokmakov.
Claims 5 and 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over Das et al. (US 11,069,072 B2) in view of Coffman et al. (US 2011/0116684 A1) further in view of Kobayashi et al. (US 2018/0374222 A1), as applied to claims 1-2, 16-17 and 19-20 above, and further in view of Chen et al. (US 12,190,588 B2) and Beller et al. (US 2022/0185267 A1).
Regarding claim 5, Das in view of Coffman further in view of Kobayashi teaches the method of claim 1, further comprising the steps of:
providing sensor data capturing the scene (Coffman, Para. [0050], the visual sensor includes a sensor that may produce a 2D image of a 3D object);
tracking a second object in the sensor data to determine a second object track (Das, Col. 12, lines 54-67, The optical flow tracks are accumulated during the course of the lifetime of an object in view (i.e., multiple tracks for multiple objects)).
The proposed combination as well as the motivation for combining the Das, Coffman and Kobayashi references presented in the rejection of Claim 1, apply to Claim 5 and are incorporated herein by reference.
Although Das in view of Coffman further in view of Kobayashi teaches determining an area where a tracked object is occluded (Das, Col. 14 line 50 – Col. 15 line 21), they do not explicitly teach “upon determining that the second object is untrackable in the sensor data, determining a position in the sensor data of a latest observation of the second object in the second object track, and upon determining that the position of the latest observation of the second object in the second object track is within a threshold distance from the first area”. However, in an analogous field of endeavor, Chen teaches an occlusion center may be predicted based on a set of candidate occluding locations for a set of other objects within a threshold distance of the estimated region, each location of the set of occluding locations overlapping with the estimated region (Chen, Col. 9, lines 48-67).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Das in view of Coffman further in view of Kobayashi with the teachings of Chen by including determining the position of a latest observation of the second object (i.e., candidate occluding locations for a set of other objects) is within a threshold distance of the first area (i.e., estimated region). One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for tracking multiple objects, as recognized by Chen.
Although Das in view of Coffman further in view of Kobayashi and Chen teaches determining a distance between a second object and the occluded area (i.e., first area) (Chen, Col. 9, lines 48-67), they do not explicitly teach “identifying, from the map, an area in the one or more video sequences where an object track being lost in the first area has been resumed, and determining a distance between the first area and the identified area, and setting a coasting period for the second object track based on the determined distance, wherein a larger distance results in a comparably longer coasting period”. However, in an analogous field of endeavor, Beller teaches that for occluded regions associated with probabilities that meet or exceed a threshold, the model can determine a distance between a closest point of each occluded region and a location of the vehicle along the candidate trajectory at the future time (i.e., determining a distance between the first area and the identified area) (Beller, Para. [0017]). The model can determine one or more of a time, a velocity, and/or a distance to intersection between the object and the vehicle (i.e., a coasting period for the second object based on the distance) (Beller, Para. [0039]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Das in view of Coffman further in view of Kobayashi and Chen with the teachings of Beller by including determining a distance between the first area (i.e., occluded region) and the identified area (i.e., the vehicle at the future time) and determining a time to intersection between the occluded region and the vehicle (i.e., the coasting period for the second object). One having ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to combine these references because doing so would allow for predicting possible behaviors for dynamic objects, as recognized by Beller. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Regarding claim 8, Das in view of Coffman further in view of Kobayashi, Chen and Beller teaches the method of claim 5, wherein setting the coasting period comprises:
determining a speed of the second object at the latest observation of the second object in the second object track (Beller, Para. [0039], the model can determine one or more of a time, a velocity, and/or a distance to intersection between the object and the vehicle); and
setting the costing period for the second track further based on the speed of the second object, wherein a higher speed results in a comparably shorter coasting period (Beller, Para. [0022], the model can determine risk for an occluded region (e.g., a likelihood for an occluded region to have an object, an intersection time and/or an intersection point between the vehicle and the object at a future time) based at least in part on features of the object comprising one or more of: a predetermined lateral acceleration, a predetermined longitudinal acceleration, a predetermined vertical acceleration, a predetermined speed, or a predetermined change in direction for a given speed).
The proposed combination as well as the motivation for combining the Das, Coffman, Kobayashi, Chen and Beller references presented in the rejection of Claim 5, apply to Claim 8 and are incorporated herein by reference. Thus, the method recited in Claim 8 is met by Das in view of Coffman further in view of Kobayashi, Chen and Beller.
Regarding claim 9, Das in view of Coffman further in view of Kobayashi, Chen and Beller teaches the method of claim 8, wherein
upon determining that the second object is untrackable in the sensor data:
predicting a position of the second object while being untrackable in the sensor data based on the determined speed (Beller, Para. [0022], the model can determine risk for an occluded region (e.g., a likelihood for an occluded region to have an object, an intersection time and/or an intersection point between the vehicle and the object at a future time) based at least in part on features of the object comprising one or more of: a predetermined lateral acceleration, a predetermined longitudinal acceleration, a predetermined vertical acceleration, a predetermined speed, or a predetermined change in direction for a given speed (i.e., intersection point is position of second object while being untrackable)).
The proposed combination as well as the motivation for combining the Das, Coffman, Kobayashi, Chen and Beller references presented in the rejection of Claim 5, apply to Claim 9 and are incorporated herein by reference. Thus, the method recited in Claim 9 is met by Das in view of Coffman further in view of Kobayashi, Chen and Beller.
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Das et al. (US 11,069,072 B2) in view of Coffman et al. (US 2011/0116684 A1) further in view of Kobayashi et al. (US 2018/0374222 A1) and Tokmakov et al. (US 12,236,688 B2), as applied to claims 3-4 and 11 above, and further in view of Chen et al. (US 12,190,588 B2) and Beller et al. (US 2022/0185267 A1).
Regarding claim 6, Das in view of Coffman further in view of Kobayashi and Tokmakov teaches the method of claim 4, further comprising the steps of:
providing sensor data capturing the scene (Coffman, Para. [0050], the visual sensor includes a sensor that may produce a 2D image of a 3D object);
tracking a second object in the sensor data to determine a second object track Das, Col. 12, lines 54-67, The optical flow tracks are accumulated during the course of the lifetime of an object in view (i.e., multiple tracks for multiple objects)).
The proposed combination as well as the motivation for combining the Das, Coffman, Kobayashi and Tokmakov references presented in the rejection of Claim 3, apply to Claim 6 and are incorporated herein by reference.
Although Das in view of Coffman further in view of Kobayashi and Tokmakov teaches determining an area where a tracked object is occluded (Das, Col. 14 line 50 – Col. 15 line 21), they do not explicitly teach “upon determining that the second object is untrackable in the sensor data, determining a position in the sensor data of a latest observation of the second object in the second object track, and upon determining that the position of the latest observation of the second object in the second object track is within a threshold distance from the first area”. However, in an analogous field of endeavor, Chen teaches an occlusion center may be predicted based on a set of candidate occluding locations for a set of other objects within a threshold distance of the estimated region, each location of the set of occluding locations overlapping with the estimated region (Chen, Col. 9, lines 48-67).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Das in view of Coffman further in view of Kobayashi and Tokmakov with the teachings of Chen by including determining the position of a latest observation of the second object (i.e., candidate occluding locations for a set of other objects) is within a threshold distance of the first area (i.e., estimated region). One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for tracking multiple objects, as recognized by Chen.
Although Das in view of Coffman further in view of Kobayashi, Tokmakov and Chen teaches determining a distance between a second object and the occluded area (i.e., first area) (Chen, Col. 9, lines 48-67), they do not explicitly teach “identifying, from the map, an area in the one or more video sequences where an object track being lost in the first area has been resumed, and determining a distance between the first area and the identified area, and setting a coasting period for the second object track based on the determined distance, wherein a larger distance results in a comparably longer coasting period”. However, in an analogous field of endeavor, Beller teaches that for occluded regions associated with probabilities that meet or exceed a threshold, the model can determine a distance between a closest point of each occluded region and a location of the vehicle along the candidate trajectory at the future time (i.e., determining a distance between the first area and the identified area) (Beller, Para. [0017]). The model can determine one or more of a time, a velocity, and/or a distance to intersection between the object and the vehicle (i.e., a coasting period for the second object based on the distance) (Beller, Para. [0039]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Das in view of Coffman further in view of Kobayashi, Tokmakov and Chen with the teachings of Beller by including determining a distance between the first area (i.e., occluded region) and the identified area (i.e., the vehicle at the future time) and determining a time to intersection between the occluded region and the vehicle (i.e., the coasting period for the second object). One having ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to combine these references because doing so would allow for predicting possible behaviors for dynamic objects, as recognized by Beller. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Regarding claim 7, Das in view of Coffman further in view of Kobayashi, Tokmakov, Chen and Beller teaches the method of claim 6, wherein the step of identifying from the map, an area in the one or more video sequences where an object track being lost in the first area has been resumed comprises:
selecting the area in the one or more video sequences from the subset having the largest probability as indicated by the map (Tokmakov, Col. 10, lines 14-27, in cases of occlusion, there are many potential paths through the graph that link the object’s locations before and after occlusion. By minimizing the RAM objective, the probabilities are shifted towards the paths which are most likely to result in correctly localizing the target object when it re-appears without enforcing any single hypothesis).
The proposed combination as well as the motivation for combining the Das, Coffman, Kobayashi, Tokmakov, Chen and Beller references presented in the rejection of Claim 6, apply to Claim 7 and are incorporated herein by reference. Thus, the method recited in Claim 7 is met by Das in view of Coffman further in view of Kobayashi, Tokmakov, Chen and Beller.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Das et al. (US 11,069,072 B2) in view of Coffman et al. (US 2011/0116684 A1) further in view of Kobayashi et al. (US 2018/0374222 A1), Chen et al. (US 12,190,588 B2) and Beller et al. (US 2022/0185267 A1), as applied to claims 5 and 8-9 above, and further in view of Li et al. (US 2023/0401799 A1, with foreign priority to CN 202110221723.4, filed February 27, 2021, US PGPub used herein for translation and mapping purposes).
Regarding claim 10, Das in view of Coffman further in view of Kobayashi, Chen and Beller teaches the method of claim 9, further as described above.
Although Das in view of Coffman further in view of Kobayashi, Chen and Beller teaches predicting the position of the second object based on speed (Beller, Para. [0022]), they do not explicitly teach “upon determining that the second object is untrackable in the sensor data: determining an angle between the first area and the identified area in the one or more video sequences” and “predicting the position of the second object while being untrackable in the sensor data further based on the angle”. However, in an analogous field of endeavor, Li teaches obtaining an occlusion relationship (i.e., position of the second object while untrackable) between a third and first object based on an included angle between an orientation of the first object and an orientation from a central point of the front face of the first object to an origin of the three-dimensional coordinate system (Li, Para. [0229]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Das in view of Coffman further in view of Kobayashi, Chen and Beller with the teachings of Li by including determining the position of the second object (i.e., occlusion relationship) based on an angle between the first object and an origin of the coordinate system. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for determining occluded objects, as recognized by Li. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Das et al. (US 11,069,072 B2) in view of Coffman et al. (US 2011/0116684 A1) further in view of Kobayashi et al. (US 2018/0374222 A1), as applied to claims 1-2, 16-17 and 19-20 above, and further in view of Maurice V. Garoutte (US 2005/0213815 A1).
Regarding claim 12, Das in view of Coffman further in view of Kobayashi teaches the method of claim 1, as described above.
Although Das in view of Coffman further in view of Kobayashi teaches determining a plurality of occluded areas in an image (Das, Col. 14 line 50 – Col. 15 line 21), they do not explicitly teach “wherein the plurality of areas in the one or more video sequences comprises a plurality of predetermined areas in the one or more video sequences”. However, in an analogous field of endeavor, Garoutte teaches a video scene under analysis is divided into a grid having a predetermined number of areas (Garoutte, Para. [0041]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Das in view of Coffman further in view of Kobayashi with the teachings of Garoutte by including determining a plurality of areas by dividing the video sequence into a grid of a predetermined number of areas. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for monitoring the activity of vehicles and pedestrians, as recognized by Garoutte. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Claims 13 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Das et al. (US 11,069,072 B2) in view of Coffman et al. (US 2011/0116684 A1) further in view of Kobayashi et al. (US 2018/0374222 A1), as applied to claims 1-2, 16-17 and 19-20 above, and further in view of Mande et al. (US 2017/0344855 A1).
Regarding claim 13, Das in view of Coffman further in view of Kobayashi teaches the method of claim 1, as described above.
Although Das in view of Coffman further in view of Kobayashi teaches adding a tracker to a list of trackers when it is lost and when it reappears (Das. Col. 14 line 50 – Col. 15 line 21), they do not explicitly teach “wherein the one or more video sequences is iteratively divided into the plurality of areas in the one or more video sequences, based on positions in the one or more video sequences where an object track among the plurality of object tracks is lost and resumed”. However, in an analogous field of endeavor, Mande teaches clustering reference trajectories according to position data. Each resulting cluster corresponds to a path available to be taken by a vehicle in the associated class. Fig. 7 conceptually illustrates a hierarchy of five clusters of reference trajectories (Mande, Para. [0078]; Fig. 7).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date to modify the method of Das in view of Coffman further in view of Kobayashi with the teachings of Mande by including dividing the video sequence into a plurality of areas (see Fig. 7) based on clustering the plurality of object trajectories as taught by Mande based on areas where object tracks are lost and resumed as taught by Das. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for enabling prediction of occlusion or vehicle collision, as recognized by Mande. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Regarding claim 14, Das in view of Coffman further in view of Kobayashi and Mande teaches the method of claim 13, wherein the step of iteratively dividing the one or more video sequences into the plurality of areas in the one or more video sequences comprises:
determining a first plurality of positions, each position indicating a position in the one or more video sequences where an object track among the plurality of object tracks is lost, clustering the first plurality of positions into a first plurality of clusters (Das. Col. 14 line 50 – Col. 15 line 21, when a tracker is deleted in an area of the image where a natural termination of a tracker object is not expected, the particular tracker object is added to a list of trackers needing potential resurrection), and
for each cluster of the first plurality of clusters, determine an area in the one or more video sequences based on the positions in the cluster track (Mande, Para. [0078]; Fig. 7, each resulting cluster corresponds to a path available to be taken by a vehicle in the associated class); and
determining a second plurality of positions, each position indicating a position in the one or more video sequences where an object track among the plurality of object tracks is resumed, clustering the second plurality of positions into a second plurality of clusters (Das. Col. 14 line 50 – Col. 15 line 21, Similarly when a new detection is observed in an area where sudden new detection are not expected, all new detections are initially tracked temporarily and after confirmation these are added to a list of possible resurrection matches), and
for each cluster of the second plurality of clusters, determine an area in the one or more video sequences based on the positions in the cluster (Mande, Para. [0078]; Fig. 7, each resulting cluster corresponds to a path available to be taken by a vehicle in the associated class).
The proposed combination as well as the motivation for combining the Das, Coffman, Kobayashi and Mande references presented in the rejection of Claim 13, apply to Claim 14 and are incorporated herein by reference. Thus, the method recited in Claim 14 is met by Das in view of Coffman further in view of Kobayashi and Mande.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Das et al. (US 11,069,072 B2) in view of Coffman et al. (US 2011/0116684 A1) further in view of Kobayashi et al. (US 2018/0374222 A1), as applied to claims 1-2, 16-17 and 19-20 above, and further in view of Chen et al. (US 12,190,588 B2).
Regarding claim 15, Das in view of Coffman further in view of Kobayashi teaches the method of claim 1, further comprising the steps of:
providing sensor data capturing the scene (Coffman, Para. [0050], the visual sensor includes a sensor that may produce a 2D image of a 3D object);
tracking a third object in the sensor data to determine a third object track (Das, Col. 12, lines 54-67, The optical flow tracks are accumulated during the course of the lifetime of an object in view (i.e., multiple tracks for multiple objects)).
The proposed combination as well as the motivation for combining the Das, Coffman and Kobayashi references presented in the rejection of Claim 1, apply to Claim 5 and are incorporated herein by reference.
Although Das in view of Coffman further in view of Kobayashi teaches determining an area where a tracked object is occluded (Das, Col. 14 line 50 – Col. 15 line 21), they do not explicitly teach “determining that a location of the third object in the sensor data is within a threshold distance from the first area in the one or more video sequences” and “determining a feature vector for the third object for the purpose of object re-identification in the object tracking application”. However, in an analogous field of endeavor, Chen teaches an occlusion center may be predicted based on a set of candidate occluding locations for a set of other objects within a threshold distance of the estimated region, each location of the set of occluding locations overlapping with the estimated region (Chen, Col. 9, lines 48-67). Chen further teaches each detected object may be represented by a feature vector indicating a set of computational features of an image (Chen, Col. 5 line 60 – Col. 6, line 5). The re-identification system uses the feature vector (Chen, Col. 7, lines 39-53).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Das in view of Coffman further in view of Kobayashi with the teachings of Chen by including determining the position of a latest observation of the second object (i.e., third object) is within a threshold distance of the first area (i.e., estimated region) and determining a feature vector for the object for the purpose of object re-identification. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for tracking multiple objects, as recognized by Chen.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Das et al. (US 11,069,072 B2) in view of Coffman et al. (US 2011/0116684 A1) further in view of Kobayashi et al. (US 2018/0374222 A1), as applied to claims 1-2, 16-17 and 19-20 above, and further in view of Kim et al. (US 2018/0342081 A1).
Regarding claim 18, Das in view of Coffman further in view of Kobayashi teaches the method of claim 1, as described above.
Although Das in view of Coffman further in view of Kobayashi teaches X, they do not explicitly teach “wherein the connection between the first area and the second area is verified when a threshold number of object tracks that were lost in the first area were resumed in the second area”. However, in an analogous field of endeavor, Kim teaches determining occlusion when the number of times occlusions that an obstacle covers a moving object (e.g., a person) is greater than a threshold value (i.e., an occlusion is verified when a threshold number of object tracks are occluded). (Kim, Para. [0092]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date to modify the method of Das in view of Coffman further in view of Kobayashi with the teachings of Kim by including determining an occlusion region when a number of occlusions is greater than a threshold. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for continued tracking of a moving object, as recognized by Kim. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Emma Rose Goebel whose telephone number is (703)756-5582. The examiner can normally be reached Monday - Friday 7:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Emma Rose Goebel/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights