DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgement is made of the application’s status as a continuation of EP 22173109.4
Information Disclosure Statement
The information disclosure statement (IDS) was submitted on 11/11/2024. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Status
Claim(s) 1-4, 7-12, 14-15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Khadloya (US 20220076022 A1).
Claim(s) 5-6 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Khadloya (US 20220076022 A1) in view of Liang (US 20110007946 A1).
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-4, 7-12, 14-15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Khadloya (US 20220076022 A1).
Regarding claim 1, Khadloya discloses A method, (¶1 “the present invention relates to methods and systems for tracking objects in videos and/or images”) being executed by a computer or signal processor or system or apparatus, (¶4 “The object tracking system comprises of a processor,”) for processing a first video data set comprising first video data of a first video depicting a plurality of moving objects, wherein the method comprises: (¶4 “The encoded instructions when implemented by the processor, configure the object tracking system to detect one or more objects in the first frame of the video,”)
generating one or more further video data sets from the first video data set, such that each further video data set of the one or more further video data sets comprises further video data of a further video, such that exactly one moving object of the plurality of moving objects is associated with said further video data set, (¶4 “configure the object tracking system to detect one or more objects in the first frame of the video, and one or more candidate objects in the second frame of the video.”) such that each one of a plurality of video images of said further video depicts only a subarea of one of a plurality of video images of the first video, and such that said one or more video images of said further video depict the moving object, which is associated with said further video data set, and (¶37 “Exemplary input frames are shown in FIG. 5, where the one of more objects of the first frame include a first person object 502 and a first car object 504. Similarly, the one or more candidate objects of the second frame include a second person object 506 and a second car object 508.”)
analysing each further video data set of the one or more further video data sets to acquire information on the moving object being associated with said further video data set, (¶34 “Finally, at 214, the objects tracked at 210 are analysed to identify one or more events… Examples of the object related events include, but are not limited to, object enter/exit, line crossing, intrusion detection, dwell/loitering detection, fall/duress detection, and the like.”) and outputting the information on said moving object. (¶34 “When one or more events are identified, a corresponding alarm or notification is sent to the user devices 120.”)
Regarding claim 2, Khadloya discloses wherein at least two further video data sets are generated as the one or more further video data sets, (¶4 “configure the object tracking system to detect one or more objects in the first frame of the video, and one or more candidate objects in the second frame of the video.”)
wherein each moving object of at least two moving objects of the plurality of moving objects is associated with exactly one of the at least two video data sets, (¶37 “Exemplary input frames are shown in FIG. 5, where the one of more objects of the first frame include a first person object 502 and a first car object 504. Similarly, the one or more candidate objects of the second frame include a second person object 506 and a second car object 508.” Khadloya discloses the moving objects are associated with each of the video frames)
wherein the method comprises analysing each further video data set of the at least two further video data sets to acquire information on the moving object being associated with said further video data set, and (¶34 “Finally, at 214, the objects tracked at 210 are analysed to identify one or more events… Examples of the object related events include, but are not limited to, object enter/exit, line crossing, intrusion detection, dwell/loitering detection, fall/duress detection, and the like.”)
wherein the method comprises outputting the information on each moving object of the at least two moving objects. (¶34 “When one or more events are identified, a corresponding alarm or notification is sent to the user devices 120.”)
Regarding claim 3, Khadloya discloses wherein a number of further video data sets are generated as the at least two further video data sets that corresponds to a number of the plurality of moving objects being depicted in the first video. (¶37 “The object detection unit 302 (of the object tracking system 112) is configured to accept at least two input frames of a video: a first frame and a second frame. The first frame includes one or more objects. The second frame is a frame subsequent to the first frame and includes one or more candidate objects. Further, the second frame may not necessarily be a frame right next to the first frame.”)
Regarding claim 4, Khadloya discloses wherein generating each further video data set of the one or more further video data sets is conducted such that the further video of said further video data set only depicts that moving object out of the plurality of moving objects that is associated with said further video data set. (¶38 “At 402 and 404, the object detection process is estimated using a bounding box (or any other shape containing the object) of the one or more candidate objects in the second (current) frame, given its bounding box in the first (previous) frame.”)
Regarding claim 7, Khadloya discloses wherein analysing each further video data set of the one or more further video data sets comprises determining whether or not a depiction of the moving object associated with said further video data set in one or more of the plurality of video images of the plurality of further frames of said further video data set corresponds to an image pattern; or (¶33 “ Further, the classification is performed by analyzing one or more features of the tracked objects, such as… Local Binary Patterns (LBP), and the like” ¶34 “Finally, at 214, the objects tracked at 210 are analysed to identify one or more events… Examples of the object related events include, but are not limited to, object enter/exit, line crossing, intrusion detection, dwell/loitering detection, fall/duress detection, and the like.”)
wherein analysing each further video data set of the one or more further video data sets comprises determining a probability on whether the depiction of the moving object associated with said further video data set in one or more of the plurality of video images of said further video data set corresponds to the image pattern. (¶83 “a method for assigning objects across frames comprises the steps of: generating a list of at least one candidate object with a threshold-grade overlap with a predicted position of the object from a previous frame; and applying visual similarity or dissimilarity between the candidate object and object pair and rejecting the pair below a threshold-grade similarity or above a threshold-grade dissimilarity”)
Regarding claim 8, Khadloya discloses wherein analysing each further video data set of the one or more further video data sets comprises determining whether or not the moving object being associated with said further video data set exhibits a particular characteristic or a particular movement, or (¶34 “Finally, at 214, the objects tracked at 210 are analysed to identify one or more events… Examples of the object related events include, but are not limited to, object enter/exit, line crossing, intrusion detection, dwell/loitering detection, fall/duress detection, and the like.”)
wherein analysing each further video data set of the one or more further video data sets comprises determining a probability on whether the moving object being associated with said further video data set exhibits the particular characteristic or the particular movement. (¶83 “a method for assigning objects across frames comprises the steps of: generating a list of at least one candidate object with a threshold-grade overlap with a predicted position of the object from a previous frame; and applying visual similarity or dissimilarity between the candidate object and object pair and rejecting the pair below a threshold-grade similarity or above a threshold-grade dissimilarity”)
Regarding claim 9, Khadloya discloses wherein analysing each further video data set of the one or more further video data sets comprises determining whether or not a depiction of the moving object associated with said further video data set in two or more video images of the plurality of video images of said further video data set corresponds to said image pattern; (¶33 “ Further, the classification is performed by analyzing one or more features of the tracked objects, such as… Local Binary Patterns (LBP), and the like” ¶34 “Finally, at 214, the objects tracked at 210 are analysed to identify one or more events… Examples of the object related events include, but are not limited to, object enter/exit, line crossing, intrusion detection, dwell/loitering detection, fall/duress detection, and the like.”)
wherein, if a number of the two or more video images in which the depiction of said moving object corresponds to the image pattern is greater than a threshold value, it is determined that said moving object exhibits the particular characteristic, and (¶83 “a method for assigning objects across frames comprises the steps of: generating a list of at least one candidate object with a threshold-grade overlap with a predicted position of the object from a previous frame; and applying visual similarity or dissimilarity between the candidate object and object pair and rejecting the pair below a threshold-grade similarity or above a threshold-grade dissimilarity”)
wherein, if the number of the two or more video images in which the depiction of said moving object corresponds to the image pattern is smaller than or equal to the threshold value, it is determined that said moving object does not exhibit the particular characteristic. (¶83 “a method for assigning objects across frames comprises the steps of: generating a list of at least one candidate object with a threshold-grade overlap with a predicted position of the object from a previous frame; and applying visual similarity or dissimilarity between the candidate object and object pair and rejecting the pair below a threshold-grade similarity or above a threshold-grade dissimilarity”)
Regarding claim 10, Khadloya discloses wherein analysing each further video data set of the one or more further video data sets comprises determining, whether or not the moving object being associated with said further video data stream exhibits the same characteristic or the same movement in each of at least two of the further video data sets. (¶83 “a method for assigning objects across frames comprises the steps of: generating a list of at least one candidate object with a threshold-grade overlap with a predicted position of the object from a previous frame; and applying visual similarity or dissimilarity between the candidate object and object pair and rejecting the pair below a threshold-grade similarity or above a threshold-grade dissimilarity” ¶18 “The video server 102a of the real-time streaming system 102 receives a dynamic imagery or video footage from the video/image capturing devices 102b, and transmits the associated data to the video analytics engine 106 in form of input frames.” Khadloya discloses inputting a stream of images that is continuously being assigned a threshold value by the system)
Regarding claim 11, Khadloya discloses wherein the particular characteristic is one of two or more particular characteristics, or wherein the particular movement is one of two or more particular movements, (¶34 “Finally, at 214, the objects tracked at 210 are analysed to identify one or more events… Examples of the object related events include, but are not limited to, object enter/exit, line crossing, intrusion detection, dwell/loitering detection, fall/duress detection, and the like.”)
wherein analysing each further video data set of the one or more further video data sets comprises determining, for each further video data set of the one or more further video data sets, and for each of the two or more particular characteristics or for each of the two or more particular movements, whether or not the moving object being associated with said further video data set exhibits said particular characteristic or said particular movement; or (¶34 “Finally, at 214, the objects tracked at 210 are analysed to identify one or more events… Examples of the object related events include, but are not limited to, object enter/exit, line crossing, intrusion detection, dwell/loitering detection, fall/duress detection, and the like.”)
wherein analysing each further video data set of the one or more further video data sets comprises determining a probability for each further video data set of the one or more further video data sets, and for each of the two or more particular characteristics or for each of the two or more particular movements, on whether a moving object being associated with said further video data set exhibits said particular characteristic or said particular movement. (¶83 “a method for assigning objects across frames comprises the steps of: generating a list of at least one candidate object with a threshold-grade overlap with a predicted position of the object from a previous frame; and applying visual similarity or dissimilarity between the candidate object and object pair and rejecting the pair below a threshold-grade similarity or above a threshold-grade dissimilarity”)
Regarding claim 12, Khadloya discloses wherein analysing each further video data set of the one or more further video data sets comprises determining a point-in-time in the further video of said further video data set and/or in the first video when the moving object being associated with said further video data set exhibits a particular characteristic or a particular movement, or (¶34 “Finally, at 214, the objects tracked at 210 are analysed to identify one or more events… Examples of the object related events include, but are not limited to, object enter/exit, line crossing, intrusion detection, dwell/loitering detection, fall/duress detection, and the like.”)
wherein analysing each further video data set of the one or more further video data sets comprises determining a point-in-time in the further video of said further video data set and/or in the first video when the probability on whether the moving object being associated with said further video data set exhibits the particular characteristic or the particular movement is greater than a threshold probability. (¶83 “a method for assigning objects across frames comprises the steps of: generating a list of at least one candidate object with a threshold-grade overlap with a predicted position of the object from a previous frame; and applying visual similarity or dissimilarity between the candidate object and object pair and rejecting the pair below a threshold-grade similarity or above a threshold-grade dissimilarity”)
Regarding claim 14, Khadloya discloses A non-transitory digital storage medium having a computer program stored thereon to perform the method, being executed by a computer or signal processor or system or apparatus, (¶4 “The object tracking system comprises of a processor, a non-transitory storage element coupled to the processor and encoded instructions stored in the non-transitory storage element.”) for processing a first video data set comprising first video data of a first video depicting a plurality of moving objects, wherein the method comprises: (¶4 “The encoded instructions when implemented by the processor, configure the object tracking system to detect one or more objects in the first frame of the video,”)
generating one or more further video data sets from the first video data set, such that each further video data set of the one or more further video data sets comprises further video data of a further video, (¶4 “configure the object tracking system to detect one or more objects in the first frame of the video, and one or more candidate objects in the second frame of the video.”) such that exactly one moving object of the plurality of moving objects is associated with said further video data set, such that each one of a plurality of video images of said further video depicts only a subarea of one of a plurality of video images of the first video, and such that said one or more video images of said further video depict the moving object, which is associated with said further video data set, and (¶37 “Exemplary input frames are shown in FIG. 5, where the one of more objects of the first frame include a first person object 502 and a first car object 504. Similarly, the one or more candidate objects of the second frame include a second person object 506 and a second car object 508.”)
analysing each further video data set of the one or more further video data sets to acquire information on the moving object being associated with said further video data set, (¶34 “Finally, at 214, the objects tracked at 210 are analysed to identify one or more events… Examples of the object related events include, but are not limited to, object enter/exit, line crossing, intrusion detection, dwell/loitering detection, fall/duress detection, and the like.”) and outputting the information on said moving object, (¶34 “When one or more events are identified, a corresponding alarm or notification is sent to the user devices 120.”)
when said computer program is run by a computer. (¶7 “the present invention discloses a computer programmable product for tracking objects across a first frame and a second frame of a video.”)
Regarding claim 15, Khadloya discloses wherein A system for processing a first video data set comprising first video data of a first video depicting a plurality of moving objects, (¶4 “The encoded instructions when implemented by the processor, configure the object tracking system to detect one or more objects in the first frame of the video,”)
wherein the system is configured for generating one or more further video data sets from the first video data set, such that each further video data set of the one or more further video data sets comprises further video data of a further video, such that exactly one moving object of the plurality of moving objects is associated with said further video data set, (¶4 “configure the object tracking system to detect one or more objects in the first frame of the video, and one or more candidate objects in the second frame of the video.”) such that each one of a plurality of video images of said further video depicts only a subarea of one of a plurality of video images of the first video, and such that said one or more video images of said further video depict the moving object, which is associated with said further video data set, and (¶37 “Exemplary input frames are shown in FIG. 5, where the one of more objects of the first frame include a first person object 502 and a first car object 504. Similarly, the one or more candidate objects of the second frame include a second person object 506 and a second car object 508.”)
wherein the system is configured for analysing each further video data set of the one or more further video data sets to acquire information on the moving object being associated with said further video data set, (¶34 “Finally, at 214, the objects tracked at 210 are analysed to identify one or more events… Examples of the object related events include, but are not limited to, object enter/exit, line crossing, intrusion detection, dwell/loitering detection, fall/duress detection, and the like.”) and wherein the system is configured for outputting the information on said moving object, (¶34 “When one or more events are identified, a corresponding alarm or notification is sent to the user devices 120.”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 5-6 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Khadloya (US 20220076022 A1) in view of Liang (US 20110007946 A1).
Regarding claim 5, Khadloya discloses the claimed invention except for wherein the first video data set comprises a plurality of first image data sets, wherein each of the plurality of first image data sets comprises image data for one video image of the first video,
wherein generating each further video data set of the one or more further video data sets is conducted such that said further video data set comprises a plurality of further image data sets, wherein each of the plurality of further image data sets comprises image data for one video image of the further video of said further video data set, and
wherein analysing each further video data set of the one or more further video data sets is conducted by analysing the moving object being associated with said further video data set in each of one or more of the plurality of further image data sets of said further video data set.
In related art, Liang discloses the first video data set comprises a plurality of first image data sets, wherein each of the plurality of first image data sets comprises image data for one video image of the first video, (Liang: ¶16 “The digital video images are then provided to the computer where various processes are undertaken to identify and segment a predetermined object from the image. In a preferred embodiment the object is an object (e.g., a mouse) in motion with some movement from frame to frame in the video” Liang discloses a video data set consisting of image frames)
wherein generating each further video data set of the one or more further video data sets is conducted such that said further video data set comprises a plurality of further image data sets, wherein each of the plurality of further image data sets comprises image data for one video image of the further video of said further video data set, and (Liang: ¶16 “The digital video images are then provided to the computer where various processes are undertaken to identify and segment a predetermined object from the image. In a preferred embodiment the object is an object (e.g., a mouse) in motion with some movement from frame to frame in the video” Fig. 1 and ¶23 discloses generating more than one video set due to iteratively performing videotape analysis “The entire behavioral repertoire of individual mice in their home cage was categorized using successive iterations by manual videotape analysis.”)
wherein analysing each further video data set of the one or more further video data sets is conducted by analysing the moving object being associated with said further video data set in each of one or more of the plurality of further image data sets of said further video data set. (Liang: ¶16 “In a preferred embodiment the object is an object (e.g., a mouse) in motion with some movement from frame to frame in the video, and is in the foreground of the video images. In any case, the digital images may be processed to identify and segregate a desired (predetermined) object from the various frames of incoming video.”)
Therefore, it would have been obvious to for one of ordinary skill in the art before the effective filing date to incorporate the dataset containing a plurality of images disclosed by Linag into the method of tracking objects across frames disclosed by Khadloya to track an object across a greater amount of time by repeatedly analyzing more image frames.
Regarding claim 6, Khadloya discloses the claimed invention except for wherein generating each further video data set of the one or more further video data sets is conducted using the first video data set
by tracking the moving object being associated with said further video data set in the first video, for example by tracking said moving object frame by frame for each of two or more frames of the first video data set which comprise image data for two or more video images of the plurality of video images of the first video, and
by generating said further video data set using the first video data set depending on the tracking of said moving object in the first video,
for example wherein the tracking of said moving object in the first video is conducted using a tracking algorithm, for example a centroid-based object tracking algorithm.
In related art, Ling discloses generating each further video data set of the one or more further video data sets is conducted using the first video data set (Liang: ¶16 “The digital video images are then provided to the computer where various processes are undertaken to identify and segment a predetermined object from the image. In a preferred embodiment the object is an object (e.g., a mouse) in motion with some movement from frame to frame in the video” Liang discloses a video data set consisting of image frames)
by tracking the moving object being associated with said further video data set in the first video, for example by tracking said moving object frame by frame for each of two or more frames of the first video data set which comprise image data for two or more video images of the plurality of video images of the first video, and (Liang: ¶16 “In a preferred embodiment the object is an object (e.g., a mouse) in motion with some movement from frame to frame in the video, and is in the foreground of the video images. In any case, the digital images may be processed to identify and segregate a desired (predetermined) object from the various frames of incoming video.”)
by generating said further video data set using the first video data set depending on the tracking of said moving object in the first video, (Liang: ¶16 “The digital video images are then provided to the computer where various processes are undertaken to identify and segment a predetermined object from the image. In a preferred embodiment the object is an object (e.g., a mouse) in motion with some movement from frame to frame in the video” Liang discloses a video data set consisting of image frames)
for example wherein the tracking of said moving object in the first video is conducted using a tracking algorithm, for example a centroid-based object tracking algorithm. (Liang: ¶55 “For example, a mouse in a cage is detected in the foreground and segregated from the background. Then, at step 409, features such as centroid, the principal orientation angle of the object, the area (number of pixels), the eccentricity (roundness), and the aspect ratio of the object, and/or shape in terms of convex hull or b-spline, of the foreground object of interest (e.g., a mouse) are extracted.”)
Therefore, it would have been obvious to for one of ordinary skill in the art before the effective filing date to incorporate a centroid based tracking algorithm to track an object across frames disclosed by Liang into the method of tracking objects across frames disclosed by Khadloya to aid in representing the location of the target object across frames.
Regarding claim 13, Khadloya discloses the claimed invention except for wherein the method comprises receiving from a user via a user interface the information on the moving object for a further video data set of the one or more further video data sets,
wherein the method comprises machine-training an artificial intelligence algorithm, for example a neural network, using said further video data set and the information on the moving object as training data.
In related art, Liang discloses the method comprises receiving from a user via a user interface the information on the moving object for a further video data set of the one or more further video data sets, (Liang: ¶54 “As a preliminary matter, at step 415 video of the activities of a standard object and known behavior characteristics are input into the system. This information may be provided from a video storage/retrieval unit 110 in digitized video form into a standard object classified module 220. This information may then be manually categorized at step 416 to define normal and abnormal activities or behaviors by a user viewing the video images”)
wherein the method comprises machine-training an artificial intelligence algorithm, for example a neural network, using said further video data set and the information on the moving object as training data. (Liang: ¶96 “This involves building a classifier that can classify the shape using the available features. This information may be stored in, for example, a database in, for example, a data memory. In one variation of the invention a Decision Tree classifier (e.g., object shape and posture classifier 215) was implemented by training the classifier with 488 samples of digitized video of a standard, in this case, normal mouse”)
Therefore, it would have been obvious to for one of ordinary skill in the art before the effective filing date to incorporate collecting data from the user and training a machine learning algorithm using the data disclosed by Liang into method of tracking objects across frames disclosed by Khadloya to help the system identify known behavior characteristics of the target object.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Bendtson (US 11575837 B2) discloses a computer implemented method for displaying data in the form of a heatmap comprises generating and displaying a first heatmap based on a first dataset of video surveillance data, receiving a user input selecting an area of the first heatmap, generating and displaying a second heatmap based on a second dataset of video surveillance data, wherein the second dataset is a subset of the first dataset which is limited only based on the area selected by the user, and wherein the step of generating and displaying the second heatmap comprises recalibrating a colour range based on the second dataset.
Lee (US 20230316536 A1) discloses systems and methods for object tracking are described. One or more aspects of the systems and methods include receiving a video depicting an object; generating object tracking information for the object using a student network, wherein the student network is trained in a second training phase based on a teacher network using an object tracking training set and a knowledge distillation loss that is based on an output of the student network and the teacher network, and wherein the teacher network is trained in a first training phase using an object detection training set that is augmented with object tracking supervision data; and transmitting the object tracking information in response to receiving the video.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL KIM MAIDEN whose telephone number is (703)756-1264. The examiner can normally be reached Monday - Friday 7:30 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Koziol can be reached at 4089187630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL KIM MAIDEN/Examiner, Art Unit 2665
/Stephen R Koziol/Supervisory Patent Examiner, Art Unit 2665
Prosecution Insights