DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference characters "550" (see specification [0083]) and "555" (see Fig. 5) have both been used to designate “display”. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “550” has been used to designate both “display” (see specification [0083]) and “interface” (see specification [0081]). Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-14 are rejected under 35 U.S.C. 103 as being unpatentable over Mirza et al. (US 2021/0124927), herein Mirza in view of Black et al. (“Multi camera image tracking”), herein Black.
Regarding claim 1, Mirza discloses a multi-camera object tracking system, the system comprising:
at least one processor (see Mirza [0396]-[0400], where one or more processors are disclosed ); and
at least one memory including computer program code (see Mirza [0396]-[0400], where memory storing programs and instructions for implementing the disclosed teachings);
wherein the at least one processor, at least one memory and the computer program code are configured (see Mirza [0396]-[0400], where the one or more processors execute instructions stored in memory to implement the disclosed teachings) to allow the system to:
receive an identifier and tracking information associated with each detected object within an overlapping portion of a first video stream from a first camera and a second video stream from a second camera, the overlapping portion associated with partially overlapping fields of view of the first and second cameras (see Mirza Fig. 10 and [0149]-[0151], where an object identifier is assigned to each person as they enter the space that is tracked by a plurality of adjacent sensors with overlapping fields of view, where a first frame from a first sensor partially overlaps with a second frame from a second sensor);
determine positional information of each detected object relative to a common coordinate system using the tracking information and a homography matrix associated with the partially overlapping fields of view of the first and second cameras (see Mirza [0157] and [0159], where the a first homography is applied to the first pixel location to determine a first coordinate in the global plane for the first person, and a homography that is associated with the second sensor and translates between pixel locations in the second frame and the coordinates in the global plane is applied to the first coordinate); and
record the associated identifiers in a correspondence table for object tracking analysis (see Mirza [0160], where the second pixel location is stored with the object identifier for the first person in the second tracking list).
Mirza does not explicitly disclose match one detected object in the first video stream with another detected object in the second video stream, based on a criterion of minimum distance between the detected objects, using the positional information of each detected object; and associate the identifiers of the matched objects.
Black teaches in a related and pertinent method for multi-camera image tracking in image surveillance (see Black Abstract), where a homography alignment method is used to correspond 2D object tracks between two camera views and the object tracks are able to be merged from separate camera views into a global world coordinate view (see Black sect. 2.3. Viewpoint correspondence (two views) and sect. 3. Tracking in 3D), where a Mahalanobis distance table is used to assess the quality of matches between each tracked object and observation, and for each 3D object observation in a frame, a Mahalanobis distance table is used to sort the distance measure, where the Mahalanobis distances values are subject to a threshold, and for each existing tracked object, the observation which has the largest likelihood of being a match is selected (see Black sect. 3.1 Data association and sect. 3.2. Outline of 3D tracking algorithm).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Black to the teachings of Mirza, such that tracked objects between two adjacent sensor field of view can be matched and associated in the global plane coordinates, where the matching is based on the best Mahalanobis distance less than a threshold.
This modification is rationalized as an application of a known technique to a known system ready for improvement to yield predictable results.
In this instance, Mirza discloses a base system for object tracking in a space with a plurality of adjacent sensors with overlapping fields of view, where a respective homography can be applied to translate pixel locations from the respective image frames to coordinates in the global plane.
Black teaches a known technique for tracking objects in 3D, where homography alignment method is used to correspond 2D object tracks between two camera views and the object tracks are able to be merged from separate camera views into a global world coordinate view, and a Mahalanobis distance table is used to assess the quality of matches between each tracked object and observation, where for each 3D object observation in a frame, a Mahalanobis distance table is used to sort the distance measure, where the Mahalanobis distances values are subject to a threshold, and for each existing tracked object, the observation which has the largest likelihood of being a match is selected.
One of ordinary skill in the art would have recognized that by applying Black’s technique would allow for the system of Mirza to match tracked objects between two adjacent sensor field of view and associated in the global plane coordinates, where the matching is based on the best Mahalanobis distance less than a threshold, predictable leading to an improved object tracking system which matches tracked objects according to a best distance measure between tracked objects and corresponding observations.
Regarding claim 2, please see the above rejection of claim 1. Mirza and Black disclose the system as claimed in claim 1, wherein, to receive the identifier and tracking information associated with each detected object within the overlapping portion of the first video stream from the first camera and the second video stream from the second camera, the system is configured to:
receive the first video stream from the first camera and the second video stream from the second camera (see Mirza Fig. 10 and [0153], where first frames from a first sensor is received; see Mirza [0159], where second frames from a second sensor are used);
detect objects in the first and the second video streams within the overlapping portion of the first and the second video streams (see Mirza [0156], where the object is determined within the overlap region between the first sensor and the second sensor);
determine tracking information associated with each detected object (see Mirza [0158], where the object identifier associated with the first person is identified); and
record the identifier and the tracking information associated with each detected object (see Mirza [0158], where the object identifier associated with the first person is stored in a second tracking list associated with the second sensor).
Regarding claim 3, please see the above rejection of claim 2. Mirza and Black disclose the system as claimed in claim 2, wherein the tracking information comprises
a detection box (see Mirza [0154], where a bounding box that encloses the collection of pixels representing the first person) and wherein, to determine the tracking information associated with each detected object, the system is configured to:
map an element of the detection box of each detected object within the overlapping portion in the first and second video streams to the common coordinate system using the homography matrix associated with the partially overlapping fields of view of the first and second cameras (see Mirza [0157] and [0159], where the first homography is applied to the first pixel location to determine a first coordinate in the global plane for the first person, and a homography that is associated with the second sensor and translates between pixel locations in the second frame and the coordinates in the global plane is applied to the first coordinate; suggesting that the bounding box for representing the first person are also mapped to the global plane coordinates using the respective homographies).
Regarding claim 4, please see the above rejection of claim 3. Mirza and Black disclose the system as claimed in claim 3, wherein, to match the one detected object in the first video stream with the another detected object in the second video stream based on the minimum distance therebetween using the positional information of each detected object (see Black sect. 3.1 Data association and sect. 3.2. Outline of 3D tracking algorithm, for each 3D object observation in a frame, a Mahalanobis distance table is used to sort the distance measure, where the Mahalanobis distances values are subject to a threshold, and for each existing tracked object, the observation which has the largest likelihood of being a match is selected), the system is configured to:
calculate, for each detected object in the first video stream, a distance between the element and a corresponding element of a detected object in the second video stream (see Black sect. 3.1 Data association and sect. 3.2. Outline of 3D tracking algorithm, where the Mahalanobis distance table is used to assess the quality of matches between each tracked object and observation);
determine a pair comprising an element in the first video stream and a corresponding element in the second video stream with a minimum distance therebetween (see Black sect. 3.1 Data association and sect. 3.2. Outline of 3D tracking algorithm, where a Mahalanobis distance is suggested to be determined for each existing tracked object and 3D observation); and
match the detected object in the first video stream associated with the element with the detected object in the second video stream associated with the corresponding element (see Black sect. 3.1 Data association and sect. 3.2. Outline of 3D tracking algorithm, where, for each existing tracked object, the observation which has the largest likelihood of being a match is selected).
Regarding claim 5, please see the above rejection of claim 1. Mirza and Black disclose the system as claimed in claim 1, wherein the system is further configured to:
associate the matched objects with a unique global identifier (see Mirza [0158], where an object identifier is associated with the tracked first person; see Black sect. 3.1 Data association and sect. 3.2. Outline of 3D tracking algorithm, where existing tracked objects are matched with observations with the largest likelihood and the tracked object is updated using the observation); and
record the global identifier in the correspondence table for object tracking analysis (see Mirza [0158], where the object identifier associated with the first person is stored in a second tracking list associated with the second sensor).
Regarding claim 6, please see the above rejection of claim 5. Mirza and Black disclose the system as claimed in claim 5, wherein, to associate the matched objects with the unique global identifier, the system is configured to:
determine if any of the identifiers of the matched objects correspond to an existing global identifier (see Mirza [0160], where the second pixel location is stored with the object identifier for the first person in the second tracking list, and after storing the second pixel location, the person movement is tracked within the field of view of the second sensor); and
in response to a positive determination that at least one of the identifiers of the matched objects correspond to an existing unique identifier, verify that the matched objects in the first and second video stream associated with the existing global identifier meet the criterion of minimum distance between the detected objects using the positional information of each detected object (see Mirza [0160], where an object identifier is associated with the tracked first person; see Black sect. 3.1 Data association and sect. 3.2. Outline of 3D tracking algorithm, where existing tracked objects are matched with observations with the largest likelihood where the Mahalanobis distance is less than a threshold and the tracked object is updated using the observation; suggesting that matched objects with associated object identifiers between fields of view meet a distance threshold ).
Regarding claim 7, please see the above rejection of claim 6. Mirza and Black disclose the system as claimed in claim 6, wherein the system is further configured to:
in response to a positive determination that the matched objects in the first and second video stream do not meet the criterion of minimum distance, re-match one detected object in the first video stream with another detected object in the second video stream based on the criterion of minimum distance between the detected objects, using the positional information of each detected object (see Black sect. 3.1 Data association and sect. 3.2. Outline of 3D tracking algorithm, where existing tracked objects are matched with observations with the largest likelihood where the Mahalanobis distance is less than a threshold; suggesting that if tracked objects and observations have Mahalanobis distances greater than the threshold, the respective tracked object and observations would not match and would separately match with other tracked objects and observations that meet the distance threshold);
associate the identifiers of the matched objects (see Mirza [0158], where an object identifier is associated with the tracked first person; see Black sect. 3.1 Data association and sect. 3.2. Outline of 3D tracking algorithm, where existing tracked objects are matched with observations with the largest likelihood and the tracked object is updated using the observation); and
update the global identifier in the correspondence table for object tracking analysis (see Mirza [0158], where the object identifier associated with the first person is stored in a second tracking list associated with the second sensor; see also Mirza [0163], where the movement of the first person is continued to be tracked until the first person leaves the field of view of the second sensor).
Regarding claim 8, it recites a method performing the system functions of claim 1. Mirza and Black teach the method by performing the system functions of claim 1. Please see above for detailed claim analysis.
Please see the above rejection for claim 1, as the rationale to combine the teachings of Mirza and Black are similar, mutatis mutandis.
Regarding claim 9, see above rejection for claim 8. It is a method claim reciting similar subject matter as claim 2. Please see above claim 2 for detailed claim analysis as the limitations of claim 9 are similarly rejected.
Regarding claim 10, see above rejection for claim 9. It is a method claim reciting similar subject matter as claim 3. Please see above claim 3 for detailed claim analysis as the limitations of claim 10 are similarly rejected.
Regarding claim 11, see above rejection for claim 10. It is a method claim reciting similar subject matter as claim 4. Please see above claim 4 for detailed claim analysis as the limitations of claim 11 are similarly rejected.
Regarding claim 12, see above rejection for claim 8. It is a method claim reciting similar subject matter as claim 5. Please see above claim 5 for detailed claim analysis as the limitations of claim 12 are similarly rejected.
Regarding claim 13, see above rejection for claim 12. It is a method claim reciting similar subject matter as claim 6. Please see above claim 6 for detailed claim analysis as the limitations of claim 13 are similarly rejected.
Regarding claim 14, see above rejection for claim 13. It is a method claim reciting similar subject matter as claim 7. Please see above claim 7 for detailed claim analysis as the limitations of claim 14 are similarly rejected.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIMOTHY WING HO CHOI whose telephone number is (571)270-3814. The examiner can normally be reached 9:00 AM to 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VINCENT RUDOLPH can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIMOTHY CHOI/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights