DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities: I/O interface 1020 in paragraph 0100 is not mentioned in the drawings.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 8, 12, 14, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Assael (20230306258) in view of Feng (20240152726).
Regarding claim 1, Assael teaches a method of training a machine learning model, comprising:
obtaining a training set including a training video; “obtaining one or more sequences of target video frames” (0005).
initializing a video generation model; “training a video data generation neural network having a plurality of video generation network parameters, the method comprising: generating one or more sequences of training video frames using the video data generation neural network” (0005).
Assael also teaches:
“Training the video data generation neural network using a video data embedding neural network configured to generate an embedding of a video frame” (Assael, 0005).
Assael doesn’t explicitly teach, but Feng teaches:
sampling a subnet architecture from an architecture search space; “the largest super kernel per layer need be constructed to encode all candidate choices in a search space 404, allowing different candidate choices in each layer to share super kernel weights” (0059).
identifying a subset of weights of the video generation model based on the sampled subnet architecture; “The method further includes identifying a subset of kernel encodings from the range of kernel encodings, for each layer of the super network” (0005) and “the NAS problem simplifies to finding which subset of kernel weights to use in each layer, as shown in FIG. 7” (0070).
and training, based on the training video, a subnet of the video generation model to generate synthetic video data, wherein the subnet includes the subset of the weights of the video generation model. “Under the defined search space, an over-parameterized super kernel is constructed for each operation in the graph, where different candidates are a subset of the shared weights” (0070).
Feng and Assael are combinable because they are in the same field of endeavor of image or video recognition. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine system of Feng and search space of Assael in order to allow a video data generation neural network to effectively be trained using a supervised training objective computed based on using a video data embedding network to process video data generated by the video data generation neural network and a set of target video data (Assael, 0020).
Regarding claim 2, the method of claim 1, Assael teaches wherein: the training set includes an input prompt corresponding to the training video, wherein the subnet is trained based on the input prompt. the system can receive an input from a user specifying which data that is already maintained by the system should be used for training the neural network (Assael, 0033).
Regarding claim 3, the method of claim 1, Feng teaches wherein obtaining weights from a pre-trained image generation model. “The weights of the DCN 200 may then be adjusted so the output 222 of the DCN 200 is more closely aligned with the target output” (Feng, 0041).
Regarding claim 8, the method of claim 1, Feng teaches wherein training the subnet comprises:
computing a diffusion loss based on an output of the video generation model and the training video; “the convolutional layers 504 may be parameterized by k.sub.h×k.sub.w, i, n, representing the spatial filter size (height and width), input channels, and output channels, respectively. If the baseline architecture uses a square kernel, let k.sub.h=k.sub.w=k” (Feng, 0064).
and updating the subset of the weights based on the diffusion loss. “The DCN 350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362, 364 of the DCN 350 are weights (not shown) that are to be updated” (Feng, 0052).
Regarding claim 12, Assael teaches wherein training the plurality of subnets comprises: computing a moving average of a weight of the video generation model across the plurality of training iterations. “In cases where each target video frame in the sequence of target video frames is a ground truth output of a corresponding training video frame in the sequence of training video frames, the system can determine this similarity based on repeatedly comparing every pair of training and ground truth video frames. That is, the system can compute a pair-wise similarity between each pair of training and ground truth video frames and thereafter combines, e.g., by computing a weighted or unweighted sum or average of, the pair-wise similarities.” (Assael, 0060).
Regarding claim 14, Feng teaches wherein: the subnet architecture is sampled based on a super-position algorithm. “To adjust the weights, a learning algorithm may compute a gradient vector for the weights. The gradient may indicate an amount that an error would increase or decrease if the weight were adjusted. At the top layer, the gradient may correspond directly to the value of a weight connecting an activated neuron in the penultimate layer and a neuron in the output layer. In lower layers, the gradient may depend on the value of the weights and on the computed error gradients of the higher layers. The weights may then be adjusted to reduce the error. This manner of adjusting the weights may be referred to as “back propagation” as it involves a “backward pass” through the neural network” (Feng, 0042) and “aspects of the present disclosure relate to a common one-shot differentiable NAS algorithm with super kernel weight sharing for convolutional, recurrent, MLP, and transformer architectures” (Feng, 0061).
Regarding claim 15, claim 15 is similar in scope to claim 1, 2, 7. Therefore, it is rejected under the same ground.
Claim(s) 4-7, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Assael (20230306258) and Feng (20240152726) as applied to claim 1, 5, 15 and further in view of Ryoo (20220189154).
Regarding claim 4, the method of claim 1, Ryoo teaches wherein: selecting a number of channels. “Each possible block architecture may be configured to generate a block output having a number of channels from a set of possible channel dimensionalities” (Ryoo, 0057).
Regarding claim 5, the method of claim 1, Ryoo teaches wherein selecting one or more blocks within a layer of the video generation model. “The system 200 may initialize the architecture with a predefined number of blocks” (Ryoo, 0056).
Regarding claim 6, Ryoo teaches wherein: the one or more blocks are selected from a set including a residual block, a temporal attention block, a spatial attention block, and a cross-attention block. “The neural network layers in each block may be arranged into “residual modules” (0046) and “each block comprises one or more residual modules” (Ryoo, 0011).
Regarding claim 7, the method of claim 1, Ryoo teaches wherein sampling the subnet architecture comprises: selecting a video resolution. “The architecture selection system may select: the number of blocks in each level of the architecture, the temporal resolution of each block, the number of channels in the output generated by each block, and which blocks receive inputs from which other blocks” (Ryoo, 0049).
Regarding claim 16, the method of claim 15, Ryoo teaches wherein selecting the subnet comprises: selecting the subnet comprises: selecting a subset of channels and subset of blocks of the video generation model. “Each possible block architecture may be configured to generate a block output having a number of channels from a set of possible channel dimensionalities” (0057) and “the system 200 may initialize the architecture with a predefined number of blocks” (0056).
Feng, Assael, and Ryoo are combinable because they are in the same field of endeavor of image or video recognition. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine system of Feng and search space of Assael as modified for residual block of Ryoo in order to enable a neural network having the architecture to effectively perform a machine learning task, e.g., a video processing task (Ryoo, 0029).
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Assael (20230306258) and Feng (20240152726) as applied to claim 1 above, and further in view of Liu (“AutoFreeze”).
Regarding claim 9, the method of claim 1, Liu teaches wherein training the subnet comprises: freezing one or more weights of the video generation model other than the subset of the weights corresponding to the subnet. “During ne-tuning, AutoFreeze adaptively determines layers which can be frozen. Once layers are frozen, the backward computation for those layers can be avoided” (Pg. 1, fig. 2).
Feng, Assael, and Liu are combinable because they are in the same field of endeavor of image or video recognition. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine system of Feng and search space of Assael as modified for freeze of Liu in order to optimize for efficiency without affecting model accuracy.
Claim(s) 10, 11, 13, 17, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Assael (20230306258) and Feng (20240152726) as applied to claim 1, and further in view of Dai (US12596908B2).
Regarding claim 10, the method of claim 1, Dai teaches further comprising:
iteratively selecting a plurality of subnets based on the architecture search space; “Iteratively sampling a search space, wherein the sampling comprises: generating a set of candidate architectures within the search space” (Dai, 12) and “search manager 112 has selection and sampling logic to intake architecture library 114, which holds a set of candidate architectures, and select candidates within search space 116. In some examples, architecture library 114 holds a set of NN models with 10-20 and/or CNN models. Initially, search space 116 starts relatively complete, and is refined during the progressing iterations” (Dai, 20).
and training the plurality of subnets during a plurality of training iterations, respectively. “Search space 116 starts relatively complete, and is refined during the progressing iterations” (Dai, 20).
Regarding claim 11, the method of claim 10, Dai teaches wherein selecting the plurality of subnets comprises: progressively expanding the architecture search space. “Expanding a border of search space” (Dai, 20).
Regarding claim 13, the method of claim 1, Dai teaches wherein: the subnet architecture is sampled based on a dynamic cost algorithm. “Refining a first search space to a second search space comprises refining the first search space to at least one intervening search space smaller than the first search space and larger than the second search space. For example, search space 317 may be the first search space, search space 337 may be the second search space, and search space 327 may be the intervening search space. In some examples, operation 614 includes refining a first search space to a second search space smaller than the first search space, in which search space 317 is the first search space and search space 327 is the second search space. In some examples, a greater number of iterations is used (see FIG. 5)” (Dai, 39).
Regarding claim 17, the method of claim 15, Dai teaches wherein: the video generation model comprises a plurality of individually trained subnets including the selected subnet. “Search manager 112 has selection and sampling logic to intake architecture library 114, which holds a set of candidate architectures, and select candidates within search space 116” (Dai, col. 3, line 66).
Regarding to claim 18, claim 18 is similar in scope to claim 15 and 17 except for additional limitations that Assael and Feng and further in view of Dai discloses: an apparatus comprising:
at least one processor;
at least one memory storing instructions executable by the at least one processor;
and the apparatus further comprising a video generation model comprising parameters stored in the at least one memory, wherein the video generation model includes a plurality of individually trained subnets trained to generate synthetic video data based on an input prompt and a target video resolution. “The apparatus has a memory and one or more processors coupled to the memory” (Feng, 0006).
Assael, Feng, and Dai are combinable because they are in the same field of endeavor of image or video recognition. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine system of Feng and search space of Assael as modified for plurality of subnets of Dai in order to reduce this time by sampling a search space (architecture search space) and modelling performance of candidate architectures with a predictor (Dai, 1).
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Assael (20230306258), Feng (20240152726), and Dai as applied in claim 18 and further in view of Mohan (US20250056036A1).
Regarding claim 19, the apparatus of claim 18, Mohan teaches further comprising: a layer of the video generation model comprises a residual block “the prediction residuals are then transformed (225) and quantized (230). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the non-transformed residual signal. The encoder can bypass both transform and quantization, i.e., the residual is coded directly without the application of the transform or quantization processes.” (Mohan, 0085), a spatial attention block “Spatial context may be preserved” (Mohan, 0140), a temporal attention block “FIG. 7 illustrates an example random access type temporal structure for efficient video coding.” (0017), and a cross-attention block “Interframe attention blocks” (0120).
Assael, Feng, Dai, and Mohan are combinable because they are in the same field of endeavor of image or video recognition. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine system of Feng and search space of Assael as modified for blocks of Mohan and plurality of subnets of Dai in order to reduce the storage and/or transmission bandwidth needed for such signals (Mohan, 0002).
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Assael (20230306258), Feng (20240152726), and Dai as applied in claim 18 and further in view of Ryoo.
Regarding claim 20, the apparatus of claim 18, Ryoo teaches wherein: the video generation model comprises a base diffusion model and a super-resolution model. “The neural network can be trained for super resolution (in the spatial and/or temporal domain) using a training set comprising down-sampled videos and corresponding higher-resolution ground-truth videos, with a loss function that compares output of the neural network to a higher-resolution ground-truth video corresponding to the down-sampled video input to the neural network, e.g. an L1 or L2 loss.” (Ryoo, 0062).
Feng, Assael, Dai, and Ryoo are combinable because they are in the same field of endeavor of image or video recognition. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine system of Feng and search space of Assael, and plurality of subnets of Dai as modified for residual block of Ryoo in order to enable a neural network having the architecture to effectively perform a machine learning task, e.g., a video processing task (Ryoo, 0029).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIANA SZE whose telephone number is (571)272-9916. The examiner can normally be reached Monday-Thursday 6am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.S./Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights