DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner note regarding Drawings
It is the examiner’s opinion any form of photograph image shown in Figure 7 are necessary and “are the only practicable medium for illustrating the claimed invention” per 37 CFR 1.84(b)(1) because the invention pertains to image-processing technology, and more particularly to “tracking partially occluded objects in a monitored environment” (specification ¶ [0002]). In the figure, an original image is processed to change details of the image, which may not be captured with sufficient detail in a line drawing to demonstrate the applicant’s invention. Therefore, no drawing objection is raised.
Claim Objections
Applicant is advised that should claims 11-13 be found allowable, claims 18-20 will be objected to under 37 CFR 1.75 as being a substantial duplicate thereof. When two claims in an application are duplicates or else are so close in content that they both cover the same thing, despite a slight difference in wording, it is proper after allowing one claim to object to the other as being a substantial duplicate of the allowed claim. See MPEP § 608.01(m).
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 4-7, 9-12, 14-19 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Divakaran et al (US 2014/0347475).
Regarding Claim 1, Divakaran et al teach a method (method of applying object detecting and tracking computing system 100; Fig 1 and ¶ [0023]) comprising:
identifying a reference point of a first object in an environment based on one or more characteristics pertaining to the first object (a detected object may include a reference location, based on corresponding OT nodes 118, 120, 122, which is applied to tracking; Fig 1 and ¶ [0023]-[0025]),
wherein a portion of the first object is occluded by a second object in the environment relative to a perspective of a camera component associated with a set of image frames depicting the first object and the second object (occlusions can be identified for a tracked object, based on corresponding OT nodes 118, 120, 122, from the given camera field of view in a video stream 216 with occlusion caused by a static or dynamic object; Fig 1, 2 and ¶ [0026]-[0028], [0039]-[0041]);
updating a set of coordinates of a multi-dimensional model for the first object based on the identified reference point (geo-coordinates 140 are used to track the object, which coordinates at each time instance are updated to track the node (nodes 118, 120, 122 represented as OT node 200 embodiment) of the object in the OT module 110 which can track with multi-dimensional video data and update the coordinate at each time instant with a track stream 224; Fig 1-3 and ¶ [0024], [0034]-[0037], [0045]),
wherein the updated set of coordinates indicate a region in at least one image frame of the set of image frames that includes the occluded portion of the first object relative to the identified reference point (a scene awareness module 210 maintains occlusion maps 212, 214 associated with static and dynamic objects causing occlusions for the field of view of the camera when the given tracked object node is occluded and updates the tracking data; Fig 1-3 and ¶ [0040]-[0045]); and
causing a location of the first object to be tracked in the environment based on the updated set of coordinates of the multi-dimensional model (a background modeling module 310 generates the background and foreground regions 326 of the scene based on the input video stream 216 and used by the feature computation module 312 to generate the spatial arrangement of the objects to be tracked; Fig 1-5D and ¶ [0027], [0030], [0043]-[0050]).
Regarding Claim 2, Divakaran et al teach the method of claim 1 (as described above), wherein updating the set of coordinates of the multi-dimensional model for the first object (geo-coordinates 140 are used to track the object, which coordinates at each time instance are updated to track the node (nodes 118, 120, 122 represented as OT node 200 embodiment) of the object in the OT module 110 which can track with multi-dimensional video data with a track stream 224; Fig 1-3 and ¶ [0024], [0034]-[0037], [0045]) comprises:
determining a first coordinate of the multi-dimensional model that corresponds to the identified reference point of the first object (the OT Node 118 can correspond to a first coordinate from the first video camera stream 112 and local tracking video 124; Fig 1 and ¶ [0024]);
determining, based on the determined first coordinate, a second coordinate of the multi-dimensional model that corresponds to a portion of the first object that is depicted by the set of image frames (the OT Node 120 can correspond to a second coordinate from the second video camera stream 114 and local tracking video 126; Fig 1 and ¶ [0026]);
generating a mapping between the second coordinate of the multi-dimensional model and an additional region of the set of image frames including the depicted portion of the first object (the object tracker manager 130 of the OT module 110 (with 2D or 3D tracking ¶ [0024], [0042]) maps the local tracks/video 124, 126, 128 for the corresponding nodes for each detected object to a single global track/video 132 for the tracked object based on converting subset geocoordinates 140 to 142 (additional region in frames) and updating coordinates to a global coordinate in order to link local tracks; Fig 1 and ¶ [0026]-[0027]; and
updating the first coordinate based on the generated mapping (the OT manager 130 may determine the detected object’s initial local track in the first camera 112 node 118 video 124 is the same as a second node 120 camera 114 track 126, resulting in updating the local track associated with the same object (such as updating the first track to match the second track); Fig 1 and ¶ [0026]-[0029]),
wherein the updated value of the first coordinate represents a corrected location of the reference point of the first object in view of the generated mapping between the second coordinate of the multi-dimensional model and the additional region of the set of image frames including the depicted portion of the first object (the object node 118 in local track 124 can be assigned a new local ID based on the second object track node 120 based on the global ID and accounted subset geo-coordinates 142 by the OT manager 130; Fig 1 and ¶ [0026]-[0029]),
wherein the updated set of coordinates comprises at least the updated first coordinate and the second coordinate (the updated coordinates of the local tracks are set to the global ID in order to link the local tracks to one another; Fig 1 and ¶ [0027]).
Regarding Claim 4, Divakaran et al teach the method of claim 2 (as described above), further comprising: determining, based on the determined first coordinate, a third coordinate of the multi-dimensional model that corresponds to the occluded portion of the first object (three object tracking nodes 118, 120, 122 may be based on geo-locations of the object (such as a person) and occlusions may be based on select parts of the object (human form); Fig 1, 3, 4A-D and ¶ [0044], [0047]),
wherein the updated set of coordinates further comprises the determined third coordinate (node regions may be updated by the occlusion reasoning engine 316 within the OT manager 130; Fig 1, 3 and ¶ [0026]-[0029], [0045], [0047].
Regarding Claim 5, Divakaran et al teach the method of claim 4 (as described above), wherein the first object comprises at least a top portion, a center portion, and a bottom portion (the object (such as a person) may be mapped into a head, torso and foot section; Fig 1, 3, 4A-D and ¶ [0031]-[0032], [0042], [0047]), and
wherein the first coordinate corresponds to the center portion of the first object, the second coordinate corresponds to the top portion of the first object, and the third coordinate corresponds to the bottom portion of the first object (object tracking nodes 118, 120, 122 may be based on geo-locations of the object (such as a person) with using a centroid torsos a first position and using distance to determine the head and feet or using human detectors 314 with head 334, torso 336 and foot detector 338 for respective regions 414, 416, 418; Fig 1, 3, 4A-D and ¶ [0031]-[0032], [0042], [0047]).
Regarding Claim 6, Divakaran et al teach the method of claim 1 (as described above), wherein identifying the reference point of the first object based on the one or more characteristics pertaining to the first object (OT nodes 118, 120, 122 are identified and associated with features of the object (such as the head, torso and feet of a person); Fig 1-4D and ¶ [0024]-[0025], [0030], [0044]-[0047]) comprises:
determining a type of the first object (the OT managing module 130 includes feature information of the detected object to identify the object as a particular object, such as a person; Fig 1-3 and ¶ [0024], [0030], [0044]-[0047]); and
identifying a set of pre-defined characteristics for the first object in view of the determined type (the algorithm to identify an object may be based on a multi-scale intrinsic motion structure (MIMS) feature; fig 1-3 and ¶ [0030], [0044]-[0047]),
wherein the one or more characteristics comprise at least one of the identified set of pre-defined characteristics (detected features in an image may be used to generate an inference of certainty of a particular identified object; Fig 1-3 and ¶ [0030], [0044]-[0047]).
Regarding Claim 7, Divakaran et al teach the method of claim 6 (as described above), wherein the set of pre-defined characteristics comprises a pre-defined size of objects corresponding to the determined type and a location of the reference point for the objects relative to the pre-defined size of the objects (features of the object (for a person in example) accounts for hypothesized regions of a particular shape of particular predefined proportions based on give dimensions; Fig 1-4A-D and ¶ [0030], [0042], [0046]-0047]).
Regarding Claim 9, Divakaran et al teach the method of claim 1 (as described above), 9. The method of claim 1, wherein causing the location of the first object to be tracked in the environment (the feature computation module 312 generates the spatial arrangement of the objects to be tracked within the OT manager 130 via tracking stream 132; Fig 1-5D and ¶ [0027], [0030] [0043]-[0050]) comprises providing the updated set of coordinates to at least one of (interpreted as only one of the following list of alternatives): an object tracking engine to track a location of objects detected within the environment across a sequence of subsequent image frames generated by the camera component, an object location engine to track a location of the object relative to real-world geographic coordinates associated with the environment, or a tracking correction engine to associate newly detected objects in the environment with previously detected objects in the environment (the OT manager generates a track stream 132 to track the object over a series of subsequent frames , which may be associated with geographic information service data 158 and may be displayed by a OT GUI 154 to provide direct correlation with real-time activity and geo-coordinates; ¶ [0026]-[0027], [0030]-[0031], [0035]).
Regarding Claim 10, Divakaran et al teach the method of claim 1 (as described above), wherein the camera component is associated with a computing system (cameras 112, 114, 116 are associated with object tracking module 110 of computing system 100 in system 1000; Fig 1 and ¶ [0023]) comprised by at least one of (interpreted as only one of the following list of alternatives): a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for three-dimensional (3D) assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for performing operations using a large language model (LLM); a system for performing operations using a vision language model (VLM); a system for performing operations using a multi-modal language model; a system for performing synthetic data generation; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (the OT module 110 may be used in associated with an aerial vehicle, an unmanned aerial vehicle or robots; Fig 1, 10 and ¶ [0024], [0083], [0087]).
Regarding Claim 11, Divakaran et al teach a system (computing system 1100; Fig 11 and ¶ [0090]) comprising: a set of one or more processing devices to perform operations (processor 1112 of computing device 1110 in system 1100); Fig 1, 11 and ¶ [0090], [0092]) comprising: steps identical to claim 1 (as described above).
Regarding Claim 12, Divakaran et al teach the system of claim 11 (as described above), with further limitations identical to claim 2 (as described above).
Regarding Claim 14, Divakaran et al teach the system of claim 12 (as described above), with further limitations identical to claim 4 (as described above).
Regarding Claim 15, Divakaran et al teach the system of claim 14 (as described above), with further limitations identical to claim 5 (as described above).
Regarding Claim 16, Divakaran et al teach the system of claim 11 (as described above), with further limitations identical to claim 6 (as described above).
Regarding Claim 17, Divakaran et al teach the system of claim 16 (as described above), with further limitations identical to claim 7 (as described above).
Regarding Claim 18, Divakaran et al teach a processor (computing system 1100 with processor 1112; Fig 11 and ¶ [0090], [0092]) comprising: a set of one or more processing units (processor 1112 of computing device 1110 in system 1100); Fig 1, 11 and ¶ [0090], [0092]) to: perform steps identical to claim 1 (as described above).
Regarding Claim 19, Divakaran et al teach the processor of claim 18 (as described above), with further limitations identical to claim 2 (as described above).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3, 13, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Divakaran et al (US 2014/0347475) in view of McKennoch (US 2021/0097701).
Regarding Claim 3, Divakaran et al teach the method of claim 2 (as described above).
Divakaran et al does not explicitly teach determining an angle of the perspective of the camera component associated with the set of image frames, wherein the second coordinate of the multi-dimensional model is further determined based on the determined angle of the perspective of the camera component.
McKennoch is analogous art pertinent to the technological problem addressed in the current application and teaches determining an angle of the perspective of the camera component associated with the set of image frames (multiple cameras 112A, 112B are used to record an environment 102 with objects 106 and fields of view 104A-104B from different angles, determined by epipolar geometry; Fig 1 and ¶ [0022]-[0026]),
wherein the second coordinate of the multi-dimensional model is further determined based on the determined angle of the perspective of the camera component (the real-world coordinates are determined for the given position based on the parallel lines 110A-110C and epipolar geometry; Fig 1 and ¶ [0022]-[0026]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Divakaran et al with McKennoch including determining an angle of the perspective of the camera component associated with the set of image frames, wherein the second coordinate of the multi-dimensional model is further determined based on the determined angle of the perspective of the camera component. By tracking the motion of objects in 3D space depicted in a sequence of 2D images, matching and distinguishing homogeneous objects is improved with efficient matching, as recognized by McKennoch (¶ [0020]).
Regarding Claim 13, Divakaran et al teach the system of claim 12 (as described above), with further limitations identical to claim 3 (as described above).
Regarding Claim 20, Divakaran et al teach the processor of claim 18 (as described above), with further limitations identical to claim 3 (as described above).
Claims 8 is rejected under 35 U.S.C. 103 as being unpatentable over Divakaran et al (US 2014/0347475) in view of Koivisto et al (US 2019/0258878).
Regarding Claim 8, Divakaran et al teach the method of claim 1 (as described above).
Although Divakaran et al teach a confidence value associated with a bounding box associated with the giving region of the detected object (¶ [0047]), Divakaran et al does not explicitly teach determining a value of a visibility metric indicating a degree of visibility of the first object in the set of image frames based on bounding box data for the first object in the environment and the updated set of coordinates of the multi-dimensional model for the first object; and determining whether the value of the visibility metric satisfies one or more visibility criteria, wherein causing the location of the first object to be tracked in the environment is performed responsive to a determination that the value of the visibility metric satisfies the one or more visibility criteria.
Koivisto et al is analogous art pertinent to the technological problem addressed in the current application and teaches determining a value of a visibility metric indicating a degree of visibility of the first object in the set of image frames based on bounding box data for the first object in the environment and the updated set of coordinates of the multi-dimensional model for the first object (object detector 106 uses a set of bounding boxes and coordinates for detected object regions 250A, 250B, 250C, 250D, including object occlusion from the sensors angles, which such visibility data 320, 322 may be associated with a given value; Fig 2A, 3, 9 and ¶ [0064]-[0067], [0087]-[0091], [0177]); and
determining whether the value of the visibility metric satisfies one or more visibility criteria (the visibility value may be used as a sigmoid activation function and used as thresholds for filtering 116A the detected object based on various criteria (such as size of the region detected); Fig 3 and ¶ [0091]-[0094]),
wherein causing the location of the first object to be tracked in the environment is performed responsive to a determination that the value of the visibility metric satisfies the one or more visibility criteria (the outputs of the detector 106 filtered to size predict the detected object, thereby allowing the detected object based on size to be tracked; Fig 3 and ¶ [0081]-[0083], [0093]-[0095]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Divakaran et al with Koivisto et al including determining a value of a visibility metric indicating a degree of visibility of the first object in the set of image frames based on bounding box data for the first object in the environment and the updated set of coordinates of the multi-dimensional model for the first object; and determining whether the value of the visibility metric satisfies one or more visibility criteria, wherein causing the location of the first object to be tracked in the environment is performed responsive to a determination that the value of the visibility metric satisfies the one or more visibility criteria. By using a visibility metric to determine objects of interest, the objects are detected with higher probabilities, thereby resulting in improved computations associated with tracking, as recognized by Koivisto et al (¶ [0003]-[0004]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Das et al (US 2021/0181758) teach a method and system for object detection and tracking including detection between multiple sensors with associated object classification and pose.
Risinger et al (US 2017/0083766) teach a method and system for object detection and tracking based on data analytics of successive pixels over multiple frames.
Holzer et al (US 2021/0225017) teach a mapping system to identify viewpoints of an object between multiple images.
Cayon et al (US 2022/0406003) teach a method and system for viewpoint stabilization based on transformations from a first coordinate to a second coordinate between the first and second image.
Chen et al (US 2023/0134690, application 17/981770) was reviewed for claim construction in parallel to claims of this application.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHLEEN M BROUGHTON whose telephone number is (571)270-7380. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATHLEEN M BROUGHTON/Primary Examiner, Art Unit 2661
Prosecution Insights