DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in Korea on 4/25/2024. It is noted, however, that applicant has not filed a certified copy of the KR10-2024-0055081 application as required by 37 CFR 1.55. See document uploaded on 9/25/2025 titled “Priority Document Exchange Failure Status Report.” Contact the number listed in the form for further guidance or questions. 1-866-217-9197 (toll-free) or 571-272-4100 (local).
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 3/13/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered and attached by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
A communication module in claim 14, described on page 23 lines 21-24.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 6-9, and 11-14 are rejected under 35 U.S.C. 103 as being unpatentable over Li (U.S. Patent Pub. No. 2024/0161369) in view of Das (U.S. Patent Pub. No. 2024/0212350).
Regarding Claim 1, Li teaches a method of detecting a video anomaly and being performed by at least one processor, the method comprising:
obtaining video data comprising a plurality of frames (¶29 multiple input subject images 102 may be used in the generation of a single output image 124; teaches multiple input images, but not specifically a video;)
detecting an object included in each of the plurality of frames (¶27 Input subject image 102 may be encoded by an image encoder 104 into an image feature vector. Image encoder 104 may be a pretrained image encoder which extracts generic image features;)
extracting a multimodal feature vector including a visual feature vector, a text feature vector, and a motion feature vector for the detected object (¶27 Input subject image 102 may be encoded by an image encoder 104 into an image feature vector. Image encoder 104 may be a pretrained image encoder which extracts generic image features. Subject text 112 may be encoded by text encoder 106 into a text feature vector. The image feature vector and text feature vector may be input to multimodal encoder 108;)
generating a noise vector by injecting noise into the visual feature vector (¶75 image model 122 is a diffusion model which includes a denoising model 612 which may be a U-Net, and is trained or pre-trained according to training framework 600; ¶75 the denoising diffusion model may receive a text prompt describing image content, and start with a random noise vector as a seed vector)
generating a restoration vector with the noise removed by inputting the noise vector into a diffusion model and using the text feature vector and the motion feature vector (see Das below) as conditions; and (¶79 Denoising model ε.sub.θ 612 may be a neural network based model, which has parameters that may be learned. Input to denoising model ε.sub.θ 612 may include a noisy latent representation (e.g., noised latent representation z.sub.T 606t), and conditioning input 610 such as a text prompt describing desired content of an output image, e.g., “a hand holding a globe.” As shown, the noisy latent representation may be repeatedly and progressively fed into denoising model 612 to gradually remove noise from the latent representation vector based on the conditioning input 610)
performing anomaly detection on the video data by comparing the visual feature vector and the restoration vector (¶80 using denoising model ε.sub.θ 612 may be decoded using decoder 614 to provide an output 616 which is the denoised image; ¶61 User device 510 may be utilized by a user 540 (e.g., a driver, a system admin, etc.) to access the various features available for user device 510, which may include processes and/or applications associated with the server 530 to receive an output data anomaly report)
Li does not explicitly disclose obtaining video data comprising a plurality of frames; extracting a multimodal feature vector including a motion feature vector; generating a reconstruction error (see Li above for restoration vector) and using the motion feature vector as condition; performing anomaly detection on the video data by comparing the visual feature vector and the restoration vector.
Das is in the same field of art of image analysis. Further, Das teaches obtaining video data comprising a plurality of frames (¶27 Data processing model 106 may be configured to receive and process a plurality of multimodal data 116. The term “multimodal data” or “multimodal information” is used herein to refer to information that may be composed of a plurality of media or data types such as, but not limited to, video, audio, graphics, temperature, pressure and other sensor measurements.)
extracting a multimodal feature vector including a motion feature vector (¶28 The term “modality of multimodal data” refers to the different modes of communication that are used to create multimodal data 116. These modes can include, but are not limited to, text, images, audio, video, and even interactivity (motion). Each mode has its own unique way of communicating information, and when used together, the different modalities can create a more comprehensive description of the environment and/or events; ¶71 For each content of a plurality of content of multimodal data, machine learning system 424 may generate, using a first machine learning model (i.e., data processing model 106), a corresponding modality feature vector representative of the data (504).)
generating a reconstruction error (see Li above for restoration vector) and using the motion feature vector as condition (¶38 inference model 110 may score a degree of anomaly of said each event, according to a reconstruction error in reconstructing said each event, to obtain anomaly scores.)
performing anomaly detection on the video data by comparing the visual feature vector and the restoration vector (¶38 inference model 110 may score a degree of anomaly of said each event, according to a reconstruction error in reconstructing said each event, to obtain anomaly scores. This makes it possible to achieve very fast inferences. In an aspect, anomaly data 122 may include anomaly scores for corresponding events in event data 118.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Li by extracting motion features from a video and reconstructing the data to determine an anomaly that is taught by Das; thus, one of ordinary skilled in the art would be motivated to combine the references to efficiently obtain useful information (Das ¶3).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 6, Li in view of Das discloses the method of claim 1, wherein the generating of the noise vector by injecting the noise into the visual feature vector comprises: generating the noise vector by injecting an amount of Gaussian noise determined according to a range of a time step into the visual feature vector (Li, ¶78 The random noise ε 608 added at each iteration may be a random sample from a probability distribution such as Gaussian distribution. The amount (i.e., variance) of noise ε 608 added at each iteration may be constant, or may vary over the iterations. The amount of noise ε 608 added may depend on other factors such as image size or resolution; ¶82 Conditioning input 610 may also include a time step, which may be used to provide the model with a general estimate of how much noise remains in the image, and the time step may increment (or decrement) for each iteration.)
Regarding Claim 7, Li in view of Das discloses the method of claim 1, wherein the diffusion model includes a first diffusion model and a second diffusion model, and the generating of the restoration vector with the noise removed comprises:
a first restoration step of inputting the noise vector into the first diffusion model and removing at least some of the noise included in the noise vector by using the text feature vector as a condition; and a second restoration step of inputting a noise vector into the second diffusion model and removing at least some of the noise included in the noise vector by using the motion feature vector as a condition (Li, ¶79 Denoising model ε.sub.θ 612 may be a neural network based model, which has parameters that may be learned. Input to denoising model ε.sub.θ 612 may include a noisy latent representation (e.g., noised latent representation z.sub.T 606t), and conditioning input 610 such as a text prompt describing desired content of an output image, e.g., “a hand holding a globe.” As shown, the noisy latent representation may be repeatedly and progressively fed into denoising model 612 to gradually remove noise from the latent representation vector based on the conditioning input 610, e.g., from z′.sub.T 618t to z′.sub.0 618a; ¶80 Ideally, the progressive outputs of repeated denoising models ε.sub.θ 612 z′.sub.T 618t to z′.sub.0 618a may be an incrementally denoised version of the input latent representation z′.sub.T 618t, as conditioned by a conditioning input 610.)
Regarding Claim 8, Li in view of Das discloses the method of claim 7, wherein the generating of the restoration vector with the noise removed further comprises: generating the restoration vector with the noise removed by iteratively performing the first restoration step and the second restoration step (Li, Fig. 6; ¶79 As illustrated, denoising model ε.sub.θ 612 is iteratively used to reverse the process of noising latents (i.e., perform reverse diffusion) from z′.sub.T 618t to z′.sub.0 618a.)
Regarding Claim 9, Li in view of Das discloses the method of claim 1, wherein the performing of the anomaly detection on the video data comprises: calculating an anomaly score based on a distance between the visual feature vector and the restoration vector; and performing the anomaly detection on the video data on the basis of whether the calculated anomaly score is greater than or equal to a threshold value (Das, ¶54 The latent variable inference model 110 may determine a distance of the new observation to the latent variable model, or in other words from the latent variable approximation. Based on the determined distance, inference model 110 may determine whether or not the new observation is an outlier (anomaly). The determination of whether or not the received new observation is an outlier may, for example, comprise determining whether the distance of the new observation to the latent variable model is greater than a threshold distance. If the calculated distance of the new observation is greater than the threshold distance, inference model 110 may determine that the new observation is an outlier. In other words, according to aspects of a present disclosure, a deviation of the reconstructed data from actual data serves as a measure of the anomaly.)
The reasons for combining Li and Das are similar to that stated in the rejection of claim 1.
Regarding Claim 11, Li in view of Das discloses the method of claim 1, wherein the diffusion model comprises: an encoder (Li, Fig. 6, 604 Encoder) comprising a plurality of denoising attention blocks (DABs) (Li, Fig. 6, 612 denoising models); a bottleneck (Li, ¶83 denoising model ε.sub.θ 612 may be implemented through a structure referred to as “U-Net.”;) and a decoder (Li, Fig. 6, 614 Decoder).
Regarding Claim 12, Li in view of Das discloses the method of claim 11, wherein each denoising attention block comprises: a residual block comprising a plurality of linear layers connected by skip connection; and a transformer block comprising a self-attention layer, a cross-attention layer, and a feed- forward network (FFN) (Li, ¶83 denoising model ε.sub.θ 612 may be implemented through a structure referred to as “U-Net.” The U-Net structure may include a series of convolutional layers and pooling layers which generate progressively lower resolution multi-channel feature maps. Each pooling layer and an associated one or more convolutional layers may be considered an encoder. The convolutional and pooling layers (i.e., encoders) may be followed by a series of up-sampling layers and convolutional layers which generate progressively higher resolution multi-channel feature maps. Each up-sampling layer and an associated one or more convolutional layers may be considered a decoder. The U-Net may also include skip connections, where outputs of each encoder layer are concatenated with the corresponding decoder layer, skipping the intermediate encoder/decoder layers. Skip connections allow information about the precise location of features extracted by convolutional (encoder) layers.)
Regarding Claim 13, Li in view of Das discloses a non-transitory computer readable recording medium storing computer program to execute a method of detecting a video anomaly on a computer according to claim 1 (Li, ¶47 computing device 400 may include non-transitory, tangible, machine readable media that include executable code that when run by one or more processors (e.g., processor 410) may cause the one or more processors to perform the processes of method.)
Regarding claim 14, claim 14 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Li further teaching on: A computing device comprising: a communication module; a memory; and at least on processor connected to the memory and configured to execute at least one computer-readable program comprised in the memory (Li, ¶47 computing device 400 may include non-transitory, tangible, machine readable media that include executable code that when run by one or more processors (e.g., processor 410) may cause the one or more processors to perform the processes of method. Some common forms of machine-readable media that may include the processes of method are, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.)
Claims 2 and 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over Li (U.S. Patent Pub. No. 2024/0161369) in view of Das (U.S. Patent Pub. No. 2024/0212350) in view of Tabkhivayghan (U.S. Patent Pub. No. 2026/0024339).
Regarding Claim 2, Li in view of Das teaches the method of claim 1.
Li in view of Das does not explicitly disclose wherein the extracting of the multimodal feature vector comprises: extracting the visual feature vector for the object by providing information related to the detected object to a trained model based on Inflated 3D ConvNet (I3D) (Li and Das both use convolutional networks, but not explicitly an I3D.)
Tabkhivayghan is in the same field of art of image analysis. Further, Tabkhivayghan teaches wherein the extracting of the multimodal feature vector comprises: extracting the visual feature vector for the object by providing information related to the detected object to a trained model based on Inflated 3D ConvNet (I3D) (¶87 For higher level tasks, to help preserve privacy from a system perspective, sensitive information may be kept on a local machine by executing all high-level tasks on the local node. These tasks have access to the object, tracking, and pose data generated in the previous steps; ¶88 Embodiments may use pixel-based methods for these tasks that achieve superior accuracies than SotA posed-based methods, such as I3D, MVIT, and Stargazer, or the like)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Li in view of Das by using I3D for extracting features that is taught by Tabkhivayghan; thus, one of ordinary skilled in the art would be motivated to combine the references for rapid and equitable security, and for accurate real-time intelligent surveillance (Tabkhivayghan ¶28).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 4, Li in view of Das in view of Tabkhivayghan discloses the method of claim 1, wherein the extracting of the multimodal feature vector comprises:
extracting skeletal information corresponding to the object by providing information related to the detected object to a trained model based on High-Resolution Network (HRNet); and (Tabkhivayghan, ¶85 Image crops of the people detected in the image are also sent to a human pose estimator, where two-dimensional pose skeletons are created. Embodiments, may use HRNet for extracting 2D skeletons.)
extracting the motion feature vector representing motion of the object by using the extracted skeletal information (Tabkhivayghan, ¶87 Embodiments may use action recognition and anomaly detection as two common examples of high-level tasks that are highly relevant to intelligent surveillance. For action recognition, embodiments may use PoseConv3D.)
The reasons for combining Li and Das and Tabkhivayghan are similar to that stated in the rejection of claim 2. In addition, this same reasoning is pertinent and applicable to the rejection of claim 5 below.
Regarding Claim 5, Li in view of Das in view of Tabkhivayghan discloses the method of claim 4, wherein the extracting of the motion feature vector representing the motion of the object by using the extracted skeletal information comprises: extracting the motion feature vector by providing the extracted skeletal information to a trained model based on PoseConv3D (Tabkhivayghan, ¶87 Embodiments may use action recognition and anomaly detection as two common examples of high-level tasks that are highly relevant to intelligent surveillance. For action recognition, embodiments may use PoseConv3D.)
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Li (U.S. Patent Pub. No. 2024/0161369) in view of Das (U.S. Patent Pub. No. 2024/0212350) in view of Nag (U.S. Patent Pub. No. 2025/0239059).
Regarding Claim 3, Li in view of Das teaches the method of claim 1.
Li in view of Das does not explicitly disclose wherein the extracting of the multimodal feature vector comprises:
generating a caption for describing the object by providing information related to the detected object to a model based on Bidirectional Encoder Representations from Transformers (BERT); and extracting the text feature vector corresponding to the description of the object by providing the generated caption to a trained model based on Simple Contrastive Learning of Sentence Embeddings (SimCSE).
Nag is in the same field of art of image analysis. Further, Nag teaches wherein the extracting of the multimodal feature vector comprises:
generating a caption for describing the object by providing information related to the detected object to a model based on Bidirectional Encoder Representations from Transformers (BERT); and extracting the text feature vector corresponding to the description of the object by providing the generated caption to a trained model based on Simple Contrastive Learning of Sentence Embeddings (SimCSE) (¶79 Specifically, the referring expression segmentation system 102 utilizes the natural language model 414 to tokenize text within the digital text prompt 412, assign grammatical categories to text in the digital text prompt 412, identify relationships between words (e.g., a spatial reference), and simplify the digital text prompt 412. In particular, a natural language model 414 refers to a computer algorithm that analyzes text (e.g., a word or a grouping of words, such as a text phrase) and generates one or more corresponding embeddings in an embedding space... the natural language model 414 comprises sentence transformers as described by Jacob Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018, https://arxiv.org/abs/1810.04805, which is incorporated herein by reference in its entirety. In still further embodiments, the language embedding model 314 comprises sentence transformers described by SimCSE-BERT.sub.base and/or SimCSE-ROBERTa.sub.large as described by Gao et al. in SimCSE: Simple Contrastive Learning of Sentence Embeddings, In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republic.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Li in view of Das by using BERT and SimCSE that is taught by Nag; thus, one of ordinary skilled in the art would be motivated to combine the references for computational efficiency, computational accuracy, and operational flexibility (Nag ¶1).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Li (U.S. Patent Pub. No. 2024/0161369) in view of Das (U.S. Patent Pub. No. 2024/0212350) in view of Kanishima (U.S. Patent Pub. No. 2023/0004863).
Regarding Claim 10, Li in view of Das teaches the method of claim 9.
Li in view of Das does not explicitly disclose wherein the calculating of the anomaly score comprises: calculating the anomaly score according to the distance by using a mean square error (MSE) between the visual feature vector and the restoration vector.
Kanishima is in the same field of art of image analysis. Further, Kanishima teaches wherein the calculating of the anomaly score comprises: calculating the anomaly score according to the distance by using a mean square error (MSE) between the visual feature vector and the restoration vector (¶27 it is assumed that the model is an autoencoder. An anomaly degree S(xn), which is a reconstruction error, may be expressed by equation (1), for example, by using a mean square error between data and an output of the autoencoder.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Li in view of Das by using mean square error that is taught by Kanishima; thus, one of ordinary skilled in the art would be motivated to combine the references for accurate anomaly detection (Kanishima ¶62).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUSTIN BILODEAU whose telephone number is (571)272-1032. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUSTIN BILODEAU/Examiner, Art Unit 2664
