DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 8-13, 16 and 20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by USPN 2022/0222940 to Rawat et al.
With regard to claim 1, Rawat discloses a computer-implemented method for processing video data, the method comprising:
obtaining, by a computing system comprising one or more computing devices, video data comprising a plurality of video frames (paragraphs [0013] and [0041], untrimmed video comprising multiple frames is input into the system);
generating, by the computing system, a plurality of spatiotemporal representations from the video data, wherein the plurality of spatiotemporal representations represent respective spatiotemporal information respectively contained within a plurality of video tubelets of the video data, the plurality of video tubelets respectively comprising a length and a width and spanning two or more video frames of the plurality of video frames (paragraphs [0013]-[0015] and [0041], spatio-temporal action tubelets are identified using bounding boxes over a series of video frames to identify objects/actions. The tubelets comprise a length and width defined by the dimensions of the identified bounding boxes and the number of frames); and
processing, by the computing system, the plurality of spatiotemporal representations with a machine-learned model to generate an output from the machine-learned model (paragraphs [0005] and [0041], A classification network used to output multi-label recognition results from the generated tubelets).
With regard to claim 2, Rawat discloses the computer-implemented method of claim 1, wherein the output comprises a video data modification output (paragraph [0051], the tubelet determination and classification is output for the videos to identify actions/objects in the series of video frames).
With regard to claim 3, Rawat discloses the computer-implemented method of claim 1, wherein the machine-learned model comprises a video transformer encoder, the video transformer encoder comprising a factorized encoder, the factorized encoder comprising a spatial transformer encoder and a temporal transformer encoder (paragraph [0044]: [0044] “To perform the localization of each video clip, the method uses an encoder-decoder structure (such as a 3D convolution-based encoder, e.g. I3D [3]) which extracts class-agnostic action features (such as spatio-temporal features that are required for activity localization) and generates segmentation masks for each clip. The decoder uses these features to segment regions from the original input which contain activities.”).
With regard to claim 8, Rawat discloses the computer-implemented method of claim 1, wherein generating the plurality of spatiotemporal representations comprises:
projecting, by the computing system, the plurality of video tubelets to a plurality of tensor representations of the plurality of video tubelets (paragraphs [0050]-[0051], Rawat discloses a tensor representation in the vector set as described); and
merging, by the computing system, the plurality of tensor representations along at least one dimension to produce the plurality of spatiotemporal representations (paragraphs [0050]-[0052], As series of tubelets are recognized and categorized, they can be merged to represent as action defined by multiple sequential tubelets, thus merging the representation defining the recognized action).
With regard to claim 9, Rawat discloses the computer-implemented method of claim 1, wherein the plurality of spatiotemporal representations comprise a plurality of embedding representations (paragraphs [0044]-[0045], The encoder-decoder structure is used to generate segmentation masks for to segment regions which are interpreted as embedded representations used for identifying content of the image in the tubelet representations).
With regard to claim 10, Rawat discloses the computer-implemented method of claim 1, wherein the width of at least one of the tubelets is less than a width of the video frames (The tubelets are identified from bounding boxes across multiple frames, the width of the bounding boxes being less than a width of the video).
With regard to claim 11, Rawat discloses the computer-implemented method of claim 1, wherein the plurality of spatiotemporal representations are single-dimensional (paragraph [0043], The tubelets are represented by a set of single dimensional elements which together describe the tubelet).
With regard to claim 12, Rawat discloses the computer-implemented method of claim 1, wherein the plurality of video tubelets are nonoverlapping (paragraph [0041], the plurality of tubelets can be merged. Since the tubelets occur in series in video frames, it is interpreted that they are not overlapping).
With regard to claim 13, Rawat discloses the computer-implemented method of claim 1, wherein positional embeddings are added to the plurality of spatiotemporal representations and input to the machine-learned model (paragraphs [0041], the tubelets are passed to a classification network where the tubelets are recognized and classified in terms of their respective depicted actions. The localization network determines which pixels and their positions correspond to the recognized actions).
With regard to claim 16, the discussion of claim 1 applies. Rawat discloses a computing system configured for classifying video data with improved accuracy, the computing system comprising:
one or more processors (paragraphs [0016] and [0042]-[0043]); and
one or more memory devices (paragraphs [0016] and [0042]-[0043]) storing:
a machine-learned model (paragraphs [0005] and [0041], the classification network is a neural network machine-learning model); and
one or more operations that, when implemented by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
obtaining video data comprising a plurality of video frames (paragraphs [0013] and [0041], untrimmed video comprising multiple frames is input into the system);
extracting a plurality of spatiotemporal representations from the video data, the plurality of spatiotemporal representations comprising a representation of spatiotemporal information in the video data (paragraphs [0013]-[0015] and [0041], spatio-temporal action tubelets are identified using bounding boxes over a series of video frames to identify objects/actions. The tubelets comprise a length and width defined by the dimensions of the identified bounding boxes and the number of frames); and
processing the plurality of spatiotemporal representations with the machine-learned model to generate a model output (paragraphs [0005] and [0041], A classification network used to output multi-label recognition results from the generated tubelets).
With regard to claim 20, Rawat discloses one or more non-transitory computer-readable media that collectively store computer-executable instructions that, when executed by a computing system, cause the computing system to perform operations (paragraph [0071]), the operations comprising:
obtaining, by the computing system, video data comprising a plurality of video frames (paragraphs [0013] and [0041], untrimmed video comprising multiple frames is input into the system);
generating, by the computing system, a plurality of spatiotemporal representations from the video data, wherein the plurality of spatiotemporal representations represent respective spatiotemporal information respectively contained within a plurality of video tubelets of the video data, the plurality of video tubelets respectively comprising a length and a width and spanning two or more video frames of the plurality of video frames (paragraphs [0013]-[0015] and [0041], spatio-temporal action tubelets are identified using bounding boxes over a series of video frames to identify objects/actions. The tubelets comprise a length and width defined by the dimensions of the identified bounding boxes and the number of frames);
processing, by the computing system, the plurality of spatiotemporal representations with a machine-learned model to generate an output from the machine-learned model (paragraphs [0005] and [0041], A classification network used to output multi-label recognition results from the generated tubelets).; and
training, by the computing system, the machine-learned model based on a loss function that evaluates the output (paragraph [0046], the classification network is trained according to calculated loss functions).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of USPN 2022/0222940 to Rawat et al and publication titled “Social Fabric: Tunelet Compositions for Video Relation Detection” to Chen et al.
With regard to claim 14, Rawat discloses the computer-implemented method of claim 1, but does not explicitly disclose wherein the transformer encoder model comprises at least one normalization layer.
Chen discloses a tubelet composition determination similar to that of Rawat and further teaches a normalization layer (Section 3, third paragraph: “On top of the features, we apply layer normalization [3], followed by a linear layer to obtain embedded representation Ri ⊂ R ∈ RN× D. In this D-dimensional embedding space, we learn a set C ∈ RK×D consisting of K primitives. The idea behind our encoding is to describe a tubelet pair entirely as a weighted combination of these primitives.”). Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the normalization taught by Chen in order to represent the tubelet accurately with weighted factors.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of USPN 2022/0222940 to Rawat et al and publication titled “Spatio-temporal Tubelet Feature Aggregation and Object Linkin in Videos” to Cores at al.
With regard to claim 15, Rawat discloses the computer-implemented method of claim 1, but does not explicitly disclose wherein the transformer encoder model comprises at least one multi-layer perceptron layer.
Cores teaches a spatio-temporal tubelet generating system similar to that of Rawat and further teaches a multi-layer perceptron with two fully connected layers (Fig. 1). Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use a perceptron with multiple layers as taught by Cores in order to aid in the classification output of the classifier network of Rawat.
Allowable Subject Matter
Claims 4-7 and 17-19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Claims 4 and 6 contain allowable subject matter. Claims 5 and 7 depend from claims 4 and 6 respectively.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WESLEY J TUCKER whose telephone number is (571)272-7427. The examiner can normally be reached 9AM-5PM Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JOHN VILLECCO can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WESLEY J TUCKER/Primary Examiner, Art Unit 2661
Prosecution Insights