DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office Action is responsive to Applicant’s response received on June 01, 2026. Claims 1-20 are pending.
Claim Rejections - 35 USC § 112
The previous 112(b) rejections have been withdrawn in light of Applicant’s amendment.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 4, 11-14 and 19 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wang et al. (“Deformable Video Transformer”).
Regarding claim 1, Wang et al. discloses a computer system for performing video processing tasks with improved computational efficiency, the computer system comprising:
one or more processors (processor of implied computer); and
one or more non-transitory computer-readable media (memory of implied computer) that collectively store:
a machine-learned model (“We begin by studying how the accuracy varies as we modify the hyperparameters and the design choices in our DVT. Due to the high computational cost involved by these numerous ablations, here we limit these evaluations to the MViT architecture on the K400 and SSv2 benchmarks. The model on K400 is trained from random initialization (scratch). Since SSv2 is a much smaller dataset, we report results on it by finetuning the model pretrained on K400.” at section 4.3, line 1) comprising:
a three-dimensional video kernel configured to be applied to a plurality of data samples from a set of video data to respectively generate a plurality of video tokens (“In D-MS-A, to generate multi-scale feature maps we adopt a 3D convolutional layer with kernel size 3 × 3 × 3, initial spatial stride of 8 (halved at each stage)” at section 4.1, last sentence), wherein each data sample comprises at least a portion of multiple image frames included in the set of video data (“To address this problem, we subdivide the input clip of T frames into B non-overlapping sub-clips, each containing T′ = T/B frames.” at page 5, left column, second full paragraph, line 5); and
a visual transformer (“In this subsection we introduce our Deformable Video Transformer, which uses motion cues to identify a sparse set of space-time location to attend for each query” at section 3.2, line 1) configured to process the plurality of video tokens to generate a model output (“The multi-scale feature maps are obtained by attaching a learnable 3D convolutional layer to each block. Then, the token update equation is given by: \hat {\bf z}^ts = {\bf z}^ts + \sum {f=1}^{F} \sum {n=1}^{N'} \alpha {s^{(f)}(n)}^{t} {\bf v}{s^{(f)}(n)}^{(f),t} (15) where, as in the case of deformable space-time attention, the 2D coordinates p(s(f)(n)) of the n-th patch to attend at scale f are obtained by adding an estimated offset to the 2D location of the query at that scale, p(s(f))” at page 5, right column, line 7); and
instructions that, when executed by the one or more processors, cause the computer system to perform operations, the operations comprising:
processing the set of video data with the machine-learned model to generate the model output (“Unless otherwise noted, we use input clips of size 16 × 224×224, sampled with temporal stride of 4. The clip-level classification is obtained by feeding the average-pooled features from the last layer to a fully connected layer. Videolevel classification is performed by averaging the clip-level predictions computed from uniformly-spaced clips sampled from the video” at section 4.1, paragraph 2, line 1);
wherein processing the set of video data with the machine-learned model comprises sparsely applying the three-dimensional video kernel to the set of video data (“In this subsection we introduce our Deformable Video Transformer, which uses motion cues to identify a sparse set of space-time location to attend for each query” at section 3.2, line 1).
Regarding claim 4, Wang et al. discloses a system wherein sparsely applying the video kernel to the set of video data comprises directly applying the video kernel to pixel values included in the set of video data (“In D-MS-A, to generate multi-scale feature maps we adopt a 3D convolutional layer with kernel size 3 × 3 × 3, initial spatial stride of 8 (halved at each stage)” at section 4.1, last sentence; “The multi-scale feature maps are obtained by attaching a learnable 3D convolutional layer to each block” at page 5, right column, line 7).
Regarding claim 7, Wang et al. discloses a system wherein sparsely applying the video kernel to the set of video data comprises applying the video kernel starting at a predefined offset point that differs from an origin point of the set of video data (“In D-MS-A, to generate multi-scale feature maps we adopt a 3D convolutional layer with kernel size 3 × 3 × 3, initial spatial stride of 8 (halved at each stage)” at section 4.1, last sentence; the stride therefore samples frames offset from the beginning frame).
Regarding claim 11, Wang et al. discloses a system wherein the machine-learned model comprises a pre-trained vision encoder that has been fine-tuned using a set of video training data (“Since SSv2 is a much smaller dataset, we report results on it by finetuning the model pretrained on K400” at section 4.3, line 7).
Regarding claim 12, Wang et al. discloses a system wherein the model output comprises a video classification output (“Unless otherwise noted, we use input clips of size 16 × 224×224, sampled with temporal stride of 4. The clip-level classification is obtained by feeding the average-pooled features from the last layer to a fully connected layer. Videolevel classification is performed by averaging the clip-level predictions computed from uniformly-spaced clips sampled from the video” at section 4.1, paragraph 2, line 1).
Regarding claim 13, Bain et al. discloses a computer-implemented method, the method comprising:
obtaining, by a computing system comprising one or more computing devices, a set of video data and a video label (“We evaluate our DVT on four standard video classification benchmarks: Kinetics-400 [20] (K400), Something-Something-V2 [15] (SSv2), EPIC-KITCHENS-100 [6] (EK100), and Diving-48 [25] (D48)” at section 4.2);
processing, by the computing system, the set of video data with a machine-learned model to generate the model output (“We begin by studying how the accuracy varies as we modify the hyperparameters and the design choices in our DVT. Due to the high computational cost involved by these numerous ablations, here we limit these evaluations to the MViT architecture on the K400 and SSv2 benchmarks. The model on K400 is trained from random initialization (scratch). Since SSv2 is a much smaller dataset, we report results on it by finetuning the model pretrained on K400.” at section 4.3, line 1), wherein processing the set of video data with the machine-learned model comprises:
sparsely applying (“In this subsection we introduce our Deformable Video Transformer, which uses motion cues to identify a sparse set of space-time location to attend for each query” at section 3.2, line 1), by the computing system, a three-dimensional video kernel (“In D-MS-A, to generate multi-scale feature maps we adopt a 3D convolutional layer with kernel size 3 × 3 × 3, initial spatial stride of 8 (halved at each stage)” at section 4.1, last sentence) of the machine-learned model to the set of video data to generate a plurality of video tokens (“The multi-scale feature maps are obtained by attaching a learnable 3D convolutional layer to each block. Then, the token update equation is given by: \hat {\bf z}^ts = {\bf z}^ts + \sum {f=1}^{F} \sum {n=1}^{N'} \alpha {s^{(f)}(n)}^{t} {\bf v}{s^{(f)}(n)}^{(f),t} (15) where, as in the case of deformable space-time attention, the 2D coordinates p(s(f)(n)) of the n-th patch to attend at scale f are obtained by adding an estimated offset to the 2D location of the query at that scale, p(s(f))” at page 5, right column, line 7), the three-dimensional video kernel having a temporal dimension size of greater than one (“Unless otherwise noted, we use input clips of size 16 × 224×224, sampled with temporal stride of 4. The clip-level classification is obtained by feeding the average-pooled features from the last layer to a fully connected layer. Videolevel classification is performed by averaging the clip-level predictions computed from uniformly-spaced clips sampled from the video” at section 4.1, paragraph 2, line 1); and
processing, by the computing system, the plurality of video tokens with a visual transformer of the machine-learned model to generate the model output (“The multi-scale feature maps are obtained by attaching a learnable 3D convolutional layer to each block. Then, the token update equation is given by: \hat {\bf z}^ts = {\bf z}^ts + \sum {f=1}^{F} \sum {n=1}^{N'} \alpha {s^{(f)}(n)}^{t} {\bf v}{s^{(f)}(n)}^{(f),t} (15) where, as in the case of deformable space-time attention, the 2D coordinates p(s(f)(n)) of the n-th patch to attend at scale f are obtained by adding an estimated offset to the 2D location of the query at that scale, p(s(f))” at page 5, right column, line 7);
evaluating, by the computing system, a loss function that generates a loss value based on the model output and the video label (the finetuning for training uses a loss term to determine how to adjust the model’s weight parameters); and
modifying, by the computing system, one or more values of one or more parameters of the machine-learned model based on the loss function (parameters of the learner are adjusted during minimization of the training loss).
Regarding claim 14, Wang et al. discloses a method wherein modifying, by the computing system, the one or more values of the one or more parameters of the machine-learned model based on the loss function comprises updating parameter values of the video kernel based on the loss function (parameters of the learner are adjusted during minimization of the training loss).
Regarding claim 19, Wang et al. discloses a one or more non-transitory computer-readable media (memory of implied computer) that collectively store:
a machine-learned model (“We begin by studying how the accuracy varies as we modify the hyperparameters and the design choices in our DVT. Due to the high computational cost involved by these numerous ablations, here we limit these evaluations to the MViT architecture on the K400 and SSv2 benchmarks. The model on K400 is trained from random initialization (scratch). Since SSv2 is a much smaller dataset, we report results on it by finetuning the model pretrained on K400.” at section 4.3, line 1) comprising:
a three-dimensional video kernel configured to be applied to a plurality of data samples from a set of video data to respectively generate a plurality of video tokens (“In D-MS-A, to generate multi-scale feature maps we adopt a 3D convolutional layer with kernel size 3 × 3 × 3, initial spatial stride of 8 (halved at each stage)” at section 4.1, last sentence), wherein each data sample comprises at least a portion of multiple image frames included in the set of video data (“To address this problem, we subdivide the input clip of T frames into B non-overlapping sub-clips, each containing T′ = T/B frames.” at page 5, left column, second full paragraph, line 5); and
a visual transformer (“In this subsection we introduce our Deformable Video Transformer, which uses motion cues to identify a sparse set of space-time location to attend for each query” at section 3.2, line 1) configured to process the plurality of video tokens to generate a model output (“The multi-scale feature maps are obtained by attaching a learnable 3D convolutional layer to each block. Then, the token update equation is given by: \hat {\bf z}^ts = {\bf z}^ts + \sum {f=1}^{F} \sum {n=1}^{N'} \alpha {s^{(f)}(n)}^{t} {\bf v}{s^{(f)}(n)}^{(f),t} (15) where, as in the case of deformable space-time attention, the 2D coordinates p(s(f)(n)) of the n-th patch to attend at scale f are obtained by adding an estimated offset to the 2D location of the query at that scale, p(s(f))” at page 5, right column, line 7); and
instructions that, when executed by one or more processors, cause a computer system to perform operations, the operations comprising:
processing the set of video data with the machine-learned model to generate the model output (“Unless otherwise noted, we use input clips of size 16 × 224×224, sampled with temporal stride of 4. The clip-level classification is obtained by feeding the average-pooled features from the last layer to a fully connected layer. Videolevel classification is performed by averaging the clip-level predictions computed from uniformly-spaced clips sampled from the video” at section 4.1, paragraph 2, line 1);
wherein processing the set of video data with the machine-learned model comprises sparsely applying the three-dimensional video kernel to the set of video data (“In this subsection we introduce our Deformable Video Transformer, which uses motion cues to identify a sparse set of space-time location to attend for each query” at section 3.2, line 1).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 9 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Wang et al. and Yang et al. (“Transformers Meet Visual Learning Understanding: A Comprehensive Review”).
Wang et al. discloses a system as described in claim 1 above.
Wang et al. does not explicitly disclose generating a plurality of fixed sine positional embeddings respectively for the plurality of video tokens, wherein the fixed sine positional embedding for each token indicates a center of the video kernel relative to the set of video data.
Yang et al. teaches a system in the same field of endeavor of wherein processing the set of video data with the machine-learned model further comprises generating a plurality of fixed sine positional embeddings respectively for the plurality of video tokens, wherein the fixed sine positional embedding for each token indicates a center of the video kernel relative to the set of video data (“In Transformer, sine and cosine functions are mainly used for position encoding. The specific coding method is formulated as Eq. 5. PE(pos; 2i) = sin(pos=100002i=dmodel ); PE(pos; 2i + 1) = cos(pos=100002i=dmodel ); (5) where pos represents the position, and i means the dimension. Each dimension of the position code corresponds to a sine curve.” At section IIIC).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the position encoding as taught by Yang et al. for the embedding of Wang et al. as a way to represent the patch locations.
Claim(s) 10 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Wang et al. and Liu et al. (“TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval”).
Wang et al. discloses a system as described in claim 1 above.
Wang et al. does not explicitly disclose that each of the plurality of data samples comprises data for only a subset of a number of channels in a channel dimension of the set of video data, and wherein at least one of the plurality of tokens is generated by concatenation along a channel dimension for two temporally-displaced data samples.
Liu et al. teaches a system in the same field of endeavor of transformer based classification, wherein each of the plurality of data samples comprises data for only a subset of a number of channels in a channel dimension of the set of video data, and wherein at least one of the plurality of tokens is generated by concatenation along a channel dimension for two temporally-displaced data samples (“In this work, we propose the token selection transformer by inserting a token selection module, which aims to select informative tokens per frame, especially those tokens containing salient semantics of objects, for video feature aggregation. As shown in Fig.4, top-K informative tokens are selected via the trainable token selection module every frame. The input of the token selection module is a sequence of tokens of each frame I = {pcls, p0, p1, . . . , pn−1} ∈ R(N+1)×C. We first apply an MLP over I for channel dimension reduction and output I′ = {p′cls, p′0, p′1, . . . , p′n−1} ∈ R(N+1)×C2 . We then use p′cls as a global frame feature and concatenate it with each local token p′i, ˆpi = [p′cls, p′i] , 0 ≤ i < N. We finally feed all the concatenated token features to another MLP followed by a Softmax layer to predict the importance scores” at section 3.2, paragraph 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to incorporate a token selection as taught by Liu et al. in the system of Wang et al. to avoid token redundancy and preserving the most relevant information (see Liu et al. at section 3.2).
Claim(s) 15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Wang et al. and Bao et al. (“BEIT: BERT Pre-Training of Image Transformers”).
Regarding claim 15, Wang et al. discloses a system as described in claim 1 above.
Wang et al. does not explicitly disclose importing the video kernel to a larger pre-trained image transformer.
Bao et al. teaches a method in the same field of endeavor of transformer based image classification, comprising importing the video kernel to a larger pre-trained image transformer (“Overview of BEIT pre-training. Before pre-training, we learn an “image tokenizer” via autoencoding-style reconstruction, where an image is tokenized into discrete visual tokens according to the learned vocabulary. During pre-training, each image has two views, i.e., image patches, and visual tokens. We randomly mask some proportion of image patches (gray patches in the figure) and replace them with a special mask embedding [M]. Then the patches are fed to a backbone vision Transformer” at Figure 1 description).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to incorporate the kernels of Wang et al. into a pre-trained transformer as taught by Bao et al. to speed up the overall training of the system.
Regarding claim 16, the Wang et al. and Bao et al. combination discloses a method wherein modifying, by the computing system, the one or more values of the one or more parameters of the machine-learned model based on the loss function comprises finetuning one or more layers of a pre-trained image transformer while holding one or more other layers of the pre-trained image transformer fixed (“After pre-training BEIT, we append a task layer upon the Transformer, and fine-tune the parameters on downstream tasks, like BERT” Bao et al. at section 2.6, line 1).
Response to Arguments
Summary of Remarks (@ response page labeled 8): “As discussed during the interview, Bain does not teach or suggest "a three-dimensional video kernel configured to be applied to a plurality of data samples from a set of video data to respectively generate a plurality of video tokens, wherein each data sample comprises at least a portion of multiple image frames included in the set of video data." or "wherein processing the set of video data with the machine-learned model comprises sparsely applying the three- dimensional video kernel to the set of video data."”
Examiner’s Response: This argument is moot in view of the newly cited Wang et al. reference.
Allowable Subject Matter
Claims 2, 3, 5, 6, 8, 17, 18 and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the prior art does not disclose applying the video kernel with a spatial stride greater than the spatial dimension size of the video kernel to achieve spatial sparseness as required by claim 2; applying the video kernel with a temporal stride greater than the temporal dimension size of the video kernel to achieve temporal sparseness as required by claim 3; one or more image kernels configured to be applied to an individual image frame of the set of video data to generate a plurality of image tokens from the individual image frame as required by claim 5; a second kernel configured to be applied to a second set of data samples from the set of video data wherein at least one of the second set of data samples is overlapping with at least one of the plurality of data samples to which the video kernel is applied as required by claim 8; one or more image kernels configured to be applied to an individual image frame of the set of video data to generate a plurality of image tokens from the individual image frame, and wherein the machine-learned model comprises a single visual transformer configured to jointly process both the plurality of video tokens and the plurality of image tokens to generate the model output as required by claim 17; applying the video kernel with a spatial stride greater than the spatial dimension size of the video kernel to achieve spatial sparseness; or applying the video kernel with a temporal stride greater than the temporal dimension size of the video kernel to achieve temporal sparseness as required by claim 20.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/Primary Examiner, Art Unit 2672
Prosecution Insights