DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/20/206 has been entered.
Notice to the Applicant
Limitations appearing inside {} are intended to indicate the limitations not taught by said prior art(s)/combinations.
Claims 1-16 are pending in the application.
Response to Amendments
The Amendment filled 03/20/2026 in response to the Final Office Action mailed 08/04/2025 has been entered. Claims 1, 8, 10, and 16 have been amended. Rejections of claims 1-20 under 35 USC §103 have been withdrawn in light of amended claims.
Response to Arguments/Remarks
Applicant’s arguments, see Remarks, filed 03/20/2026, with respect to the rejection(s) of claim(s) 1-16 under 35 USC §103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Jones et al., (US 10824935 B2) in view of Hong, et al., (US 20240087115 A1).
Information Disclosure Statement
No Information Disclosure Statement (IDS) was filed; therefore, no applicant-submitted references were considered.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7-13, and 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over “Jones” (Jones et al. US 10824935 B2 ) in view of “Hong” (Hong et al., US 20240087115 A1).
Regarding claim 1, Jones teaches a system for video anomaly detection (“a system for video anomaly detection”; [Jones, Col 3:59-60]), comprising: a processor (processor 120; [Jones, See Fig 1]); and a memory having instructions stored thereon (memory124; [Jones, See Fig 1]) that, when executed by the processor, cause the system to:
collect a sequence of input video frames of an input video of a scene ("The system includes an input interface configured to accept an input video of a scene; a memory configured to store training video patches of a training video of the scene capturing normal activity in the scene, and store a trained neural network and a processor"; [Jones, Col 3: 59-67]);
partition the sequence of input video frames into a plurality of input video patches,
wherein each of the plurality of input video patches is a spatio-temporal patch (“FIG. 2 shows an example of partitioning 220 frames of video 210 into a set of spatio-temporal patches 230. Each spatio-temporal patch, e.g., a patch 251, is defined in space and time by a spatial dimension 250 defining a region of the spatio-temporal patch in each video frame and a temporal dimension 240 defining a number of video frames forming the spatio-temporal patch.”; [Jones, Col 5:67-68, Col6:1-9]);
process each of the plurality of input video patches with one or more classifiers (“a detector for abnormal and normal activity that is based on a classifier trained not to classify a video patch as normal or abnormal, but rather trained to classify two video patches as similar or dissimilar”; [Jones, Col 7:19-23]),
wherein each of the one or more classifiers corresponds to a deep neural network having an output layer trained to estimate one or more attributes of the plurality of input video patches from an output of a {penultimate} layer of a deep neural network associated with the one or more classifiers (“neural network takes as input two video patches 610 and 520 and outputs 690 either a distance between them or classifies them as similar or different”, [Jones, Co 9: 59-62]; “The network architecture is designed such that the first layers of the network extract features that are useful for determining whether the input video patches are similar or not and the last layers of the network use the features to compute a distance between the input video patches”; [Jones, Col 7: 53-58]; “The result of these convolutional layers is a set of feature maps that contain the important information in each video patch for subsequent comparison”; [Jones, Col 9:65- 10:1]),
{wherein the output of the penultimate layer comprises one or more feature vectors representing high-level features of the plurality of input video patches such that the high-level features are mapped to the estimated one or more attributes that are human-understandable};
compare the output of the {penultimate} layer of the deep neural network associated with the one or more classifiers generated using the plurality of input video patches with a corresponding nominal output of the {penultimate} layer of the deep neural network associated with the one or more classifiers generated using a plurality of nominal video patches from corresponding spatial regions ("compare, using the neural network, each input video patch with corresponding training video patches retrieved from the memory to determine if each input video is similar to at least one corresponding training video patch; and declare an anomaly when at least one input video patch is dissimilar to all corresponding training video patches"; Jones, [Col 3:67 – Col 4:1-7]; The network architecture is designed such that the first layers of the network extract features that are useful for determining whether the input video patches are similar or not; Jones [Col 7:44-58]),
wherein the plurality of nominal video patches are extracted from nominal video of the scene (“normal video patches extracted from spatio-temporal regions of normal video of the scene”, [Jones (Col 8: 33-34]);
detect an anomaly when the output of the {penultimate} layer of the deep neural network associated with the one or more classifiers for an input video patch from the plurality of input video patches is dissimilar to the corresponding nominal output of the {penultimate} layer of the deep neural network associated with the one or more classifiers for the plurality of nominal video patches from the corresponding spatial regions (“The method declares 330 an anomaly 335 when at least one input video patch is dissimilar to all corresponding training video patches”; [Jones, Col 8: 13-15]); and
providing an output {comprising an explanation} of a type of the detected anomaly wherein the output is provided based on the estimated one or more attributes of the plurality of input video patches that are dissimilar to estimated attributes of a closest matching nominal video patch from the plurality of nominal video patches (The neural network takes as input two video patches 610 and 520 and outputs 690 either a distance between them or classifies them as “similar” or “different”; [Jones, Col 9:59-62]; The result of these convolutional layers is a set of feature maps that contain the important information in each video patch for subsequent comparison; [Jones, Col 9:65-10:1].).
Jones teaches that features containing information for comparison are extracted from the first three layers of the network, however Jones does not explicitly disclose output of the penultimate layer. Additionally, Jones does not explicitly disclose providing an output comprising an explanation of a type of detected anomaly.
However, Hong, a similar field of endeavor of using CNN-based technology to classify images of abnormalities, teaches (classifiers extract high-level features (¶[0054]). The high-level features are specified as being output from early activation layers as well as the "decision-making pixels of the CNN penultimate layers" (¶[0055]), i.e., explanation. Furthermore, these feature maps are utilized for understanding the pixel-based contours that the algorithm visualizes (¶[0054]); “In one or more implementations, the classifier CNN of the present disclosure is trained, using two strategies that takes advantage of open-source generic datasets, for example ImageNet [6], open-source skin lesion datasets, for high-level feature extraction, but relies primarily on skin datasets of dermatoscopic and mobile-device camera images. In one or more implementations, the CNN of the present disclosure is used to produce composite skin saliency feature maps using an innovative Boundary-Attention Mapper (BAM) system to understand the pixel-based contours that the algorithm visualizes in the early activation layers as well as the decision-making pixels of the CNN penultimate layers. The result is a CNN-based Boundary Attention Mapper component.”; Hong, ¶¶[0054]-[0055]).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to use the penultimate layer of a DNN for attribute estimation as taught by Hong to the invention of Jones. The motivation for doing so would be to extract high-level features highlighting important information which correspond to any decision of interest.
Regarding claim 2, the combination of Jones and Hong teach the system of claim 1. Jones further discloses wherein each of the plurality of input video patches has a spatial dimension defining a spatial region of the spatio-temporal patch in each of the sequence of input video frames and a temporal dimension defining a number of input video frames forming the spatio-temporal patch (each spatio-temporal patch, e.g., a patch 251, is defined in space and time by a spatial dimension 250 defining a region of the spatio-temporal patch in each video frame and a temporal dimension 240 defining a number of video frames forming the spatio-temporal patch [Jones, par 14]).
Regarding claim 3, the combination of Jones and Hong teach the system of claim 1. Jones further discloses wherein the plurality of nominal video patches are generated by partitioning one or more video frames of a nominal video present in a sequence of nominal video frames (normal video patches extracted from spatio-temporal regions of normal video of the scene [Jones, par 27]),
wherein the nominal video corresponds to video of normal activities happening in the same scene as the input video (The exemplars are considered to represent normal behavior in the scene. In other words, if the input, testing video patch is similar to an exemplar, the input video patch is considered normal; Jones, par 27]).
Regarding claim 4, the combination of Jones and Hong teach the system of claim 3. Jones further discloses wherein the plurality of nominal video patches for each spatial region are a subset of every possible video patch in the sequence of nominal video frames (See Jones, par 23; the union of all video patches covers the entire video sequence)and are chosen to cover the entire set of nominal video patches (the exemplars representing a particular spatial region of normal video as a subset of video patches such that the subset “covers” all normal video patches for that spatial region [Jones, par 29]).
Regarding claim 5, the combination of Jones and Hong teach the system of claim 1. Jones further discloses wherein the spatio-temporal partitions of the input video are identical to the spatio-temporal partitions of the nominal video, wherein the identical spatio-temporal partitions are used to streamline the comparison (the spatio-temporal partitions of the input video are identical to the spatio-temporal partitions of the training video to streamline the comparison [Jones, par 15]).
Regarding claim 7, the combination of Jones and Hong teach the system of claim 1. Jones further discloses wherein the deep neural network is trained using a sequence of video frames (the neural network is trained on training examples consisting of pairs of video patches [Jones, par 35], where video patches extracted from different parts of training and/or input video frames, i.e. sequence of video frames [Jones, par 9]).
Regarding claim 8, the combination of Jones and Hong teach the system of claim 1. Jones further teaches wherein the system is configured to compare the output generated by the {penultimate} layer of the deep neural network associated with the one or more classifiers with the corresponding nominal output using one or more algorithms associated with nearest neighbor search, wherein the one or more algorithms corresponds to at least one of: brute force search, k-d trees, k-means trees, and locality sensitive hashing (given normal video that defines all normal activity for a scene along with testing video of the same scene not containing any anomalies, then any video patch of the testing video must by definition be similar to at least one video patch of the normal video from the same spatial region; Jones [Col 10:39-46]. Various nearest neighbor search algorithms known in the field could be used such as k-d trees, k-means trees, and locality sensitive hashing; [Jones, Col 11:1-3]).
Hong further teaches output generated by the penultimate layer (decision-making pixels of the CNN penultimate layers; Hong, ¶[0055]).
Regarding claim 9, the combination of Jones in view of Hong teaches the system of claim 1. Jones further disclose wherein the system is configured to calculate a distance between the one or more attributes of the input video and a closest matching attribute of a nominal video (Fig. 7 exhibits the nearest neighbor search method used by some embodiments to find the closest exemplar to a testing video patch. In FIG. 7, fv 710 is the input video patch and each x.sub.i 720 is a normal video patch. The nearest neighbor search 730 outputs the minimum distance, d, 740 between vp and the nearest x.sub.i [Jones, par 39]).
Claim 10 is similarly analyzed as analogous claim 1.
Claim 11 is similarly analyzed as analogous claim 2.
Claim 12 is similarly analyzed as analogous claim 3.
Claim 13 is similarly analyzed as analogous claim 5.
Claim 15 is similarly analyzed as analogous claim 7.
Claim 16 is similarly analyzed as analogous claim 8.
Claims 6 and 14 are rejected under 35 U.S.C. 103 as obvious over Jones in view of Hong, and further in view of Karpathy, et. al. Large-scale Video Classification with Convolutional Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 1725-1732.
Regarding claim 6, the combination of Jones and Hong teaches the system of claim 1. The combination does not explicitly disclose wherein the one or more attributes of the input video patch comprises appearance and motion attributes, wherein the appearance and motion attributes comprises at least one of: directions of motion for objects in the input video patch, speed of motion in each direction, and size of moving objects.
However, Karpathy, a similar field of endeavor of video classification with CNN, teaches wherein the one or more attributes of the input video patch comprises appearance and motion attributes, wherein the appearance and motion attributes comprises at least one of: directions of motion for objects in the input video patch, speed of motion in each direction, and size of moving objects in the input video patch (“early and direct connectivity to pixel data allows the network to precisely detect local motion direction and speed” [Karpathy, Sec 3.1, par 3: ln 7-9])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include appearance and motion attributes as taught by Karpathy to the combined invention of Jones and Hong. The motivation to do so would be to take advantage of local motion information present in the video in order to provide human-interpretable features with classification.
Claim 14 is similarly analyzed as analogous claim 6.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See PTO-829 Notice of References Cited for full citations.
Zhang Xiaolong et al. CN 113887363 A, teaches video abnormal event detection and extracts the features of the penultimate layer of the I3D model network as the feature vector of the video within the classification task.
Ramaswamy et al (2022) teaches generating explanations by deriving a simple subspace from the feature space f (typically, the penultimate layer of F) that, when added to the set of attributes, allows the model to be fully explained.
Wan et al, (2021) teaches anomaly detection in surveillance videos, concatenating features from the penultimate layer for final feature representation of video clips.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHANDHANA PEDAPATI whose telephone number is 571-272-5325. The examiner can normally be reached M-F 8:30am-6pm (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at 571-272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHANDHANA PEDAPATI/Examiner, Art Unit 2669
/JOHN B STREGE/Primary Examiner, Art Unit 2669
Prosecution Insights