DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, 5-11 and 13-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Minnen (U.S. PG-PUB NO. 2020/0027247) in view of Houssby (U.S. PG-PUB NO. 2025/0005797).
-Regarding claim 1, Minnen discloses a method for visual data processing ([0059]), comprising: determining, by using a first model during a conversion between visual data and a bitstream of the visual data, a probability distribution for entropy coding associated with the bitstream (the output of the entropy model neural network 112 includes distribution parameters that define each code symbol probability distribution of the conditional entropy model, [0072]; bit string, [0076]); and performing the conversion based on the probability distribution (code symbol probability distribution, [0105]-[0111]), wherein determining the probability distribution comprises: generating intermediate information based on a quantized latent representation of the visual data by using the first model (the compression system 100 quantizes the latent representation 116 of the input data using a quantizer Q 118 to generate an ordered collection of code symbols 120 (ŷ), [0067]); and generating the probability distribution based on the intermediate information and an output of an autoregressive model and an output of a hyperprior model (context neural network 114 is configured to autoregressively process the code symbols 120 representing the input data (i.e., in accordance with an ordering of the code symbols) to generate a respective “context output” 130 (Φ) for each code symbol, [0073]; entropy model neural network 112 can process the context outputs 130 generated by the context neural network 114 (i.e., in addition to the hyper-decoder output 128) to generate the conditional entropy model, [0074]).
Minnen is silent to teaching that with an attention mechanism. However, the claimed limitation is well known in the art as evidenced by Houlsby.
In the same field of endeavor, Houlsby teaches with an attention mechanism (self-attention neural network layer is configured to receive as input a sequence of layer input elements and to apply an attention mechanism over the sequence of layer input elements to generate a sequence of layer outputs elements, [0023]; instead of segmenting the image 102 itself into the image patches 112a-n, the image patch generation system 110 can segment the feature map of the image 102 generated by the embedding neural network of the image patch embedding system 120, [0041]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Minnen with the teaching of Houlsby in order to provide capture dependencies across distant latent positions.
-Regarding claim 2, the combination further discloses the first model comprises a transformer model or a transformer context model (Houlsby, elf-attention based neural network 200 includes a sequence of M network blocks 210a-m, M≥1, [0073]).
-Regarding claim 5, the combination further discloses the probability distribution is generated by using a further model different from the first model (Minnen, the entropy model neural network 112 is configured to process the hyper-decoder output 128 to generate the conditional entropy model, [0071]).
-Regarding claim 6, the combination further discloses information on whether the probability distribution is determined by using a combination of the first model with an autoregressive model and a hyperprior model is indicated in the bitstream, or wherein information on whether the probability distribution is determined by using a combination of the first model with an autoregressive model and a hyperprior model is determined by a decoder, or wherein the probability distribution is determined by using a combination of the first model with a hyperprior model, or wherein the probability distribution is determined by using a combination of the first model with an autoregressive model, or wherein an input of the first model comprises a quantized latent representation of the visual data (Minnen, compression system 100 quantizes the latent representation 116 of the input data using a quantizer Q 118 to generate an ordered collection of code symbols 120 (ŷ), [0067]), or wherein information on whether the first model is used to replace an autoregressive model is indicated in the bitstream, or information on whether the first model is used to replace a hyperprior model is indicated in the bitstream, or information on whether the first model is used to replace the autoregressive model and the hyperprior model is indicated in the bitstream, or wherein information on whether the first model is used to replace an autoregressive model is determined by a decoder, or information on whether the first model is used to replace a hyperprior model is determined by a decoder, or information on whether the first model is used to replace the autoregressive model and the hyperprior model is determined by a decoder, or wherein the probability distribution is determined by using a plurality of the first models.
-Regarding claim 7, the combination further discloses determining the probability distribution comprises: determining a context parameter based on a quantized latent representation of the visual data by using the first model; and generating the probability distribution based on the context parameter (Minnen, context output 130 for a code symbol can be understood as causal contextual information that can be used by the entropy model neural network 112 to generate a more accurate code symbol probability distribution for the code symbol, [0073]).
-Regarding claim 8, the combination further discloses determining the context parameter comprises: performing an attention calculation based on the quantized latent representation; and generating the context parameter based on a result of the attention calculation by using at least one subnetwork (Houlsby, the self-attention neural network layer 220 is configured to apply an attention mechanism over the block input elements at the input positions (i.e., the other block input positions and, in some implementations, itself) using one or more queries derived from the block input element at the particular input position to generate a respective output for the particular position, [0076]; feed-forward layers 230 is configured to, for each input position of the input sequence 202, receive an input element at the position and apply a sequence of transformations to the input element at the position to generate an output element for the position, [0085]).
-Regarding claim 9, the combination further discloses performing the attention calculation comprises: generating a relationship between coded elements and a current element of a quantized latent representation of the visual data; and generating the result based on the relationship (Houlsby, a first attention computation in which a query generated from the particular block input element attends to a set of keys generated from respective block input elements, [0080]).
-Regarding claim 10, the combination further discloses the relationship is determined in an autoregressive way (Minnen, context neural network 114 is configured to autoregressively process the code symbols 120 representing the input data (i.e., in accordance with an ordering of the code symbols) to generate a respective “context output” 130 (Φ) for each code symbol, [0073]).
-Regarding claim 11, the combination further discloses a query, a key and a value for performing the attention calculation is determined based on the quantized latent representation (Houlsby, a first attention computation in which a query generated from the particular block input element attends to a set of keys generated from respective block input elements, [0080]; the self-attention neural network layer 220 is configured to apply an attention mechanism over the block input elements at the input positions, [0076]).
-Regarding claim 13, the combination further discloses the relationship is represented by a relation matrix, and a mask is applied on the relation matrix for generating the result (Houlsby, a first attention computation in which a query generated from the particular block input element attends to a set of keys generated from respective block input elements, [0080]; Minnen, a masked convolutional layer can generate a context output for a current code symbol by dynamically zeroing components of the convolution kernel of the layer that operate on (i) the current code symbol and (ii) code symbols that follow the current code symbol, [0104]), or wherein a key and a value for performing the attention calculation is determined based on the quantized latent representation, and a query for performing the attention calculation is determined based on additional information.
-Regarding claim 14, the combination further discloses the conversion is performed by using a second model with the attention mechanism (Houlsby, neural network system 100 can then provide the embedding of the image 102 to a downstream system for storage or further processing, e.g., by one or more other neural networks, [0061]).
-Regarding claim 15, the combination further discloses the second model comprises a transformer model (Houlsby, self-attention based neural network 200 includes a sequence of M network blocks 210a-m, M≥1, [0073]), or wherein a prediction associated with the visual data for performing the conversion is determined by using the second model, or wherein motion information associated with the visual data for performing the conversion is determined by using the second model, or wherein the visual data comprises a plurality of frames, and an input of the second model comprises coded frames of the plurality of frames.
-Regarding claim 16, the combination further discloses the visual data comprises at least one of: an image, a picture of a video, or a video (Minnen, image, [0066]).
-Regarding claim 17, the combination further discloses the conversion includes encoding the visual data into the bitstream (Minnen, bit string, [0076]).
-Regarding claim 18, Minnen discloses an apparatus for visual data processing ([0059]) comprising a processor (processor, [0143]) and a non-transitory memory with instructions thereon (memory, [0142]), wherein the instructions upon execution by the processor, cause the processor to perform acts comprising: determining, by using a first model during a conversion between visual data and a bitstream of the visual data, a probability distribution for entropy coding associated with the bitstream (the output of the entropy model neural network 112 includes distribution parameters that define each code symbol probability distribution of the conditional entropy model, [0072]; bit string, [0076]); and performing the conversion based on the probability distribution (code symbol probability distribution, [0105]-[0111]), wherein determining the probability distribution comprises: generating intermediate information based on a quantized latent representation of the visual data by using the first model (the compression system 100 quantizes the latent representation 116 of the input data using a quantizer Q 118 to generate an ordered collection of code symbols 120 (ŷ), [0067]); and generating the probability distribution based on the intermediate information and an output of an autoregressive model and an output of a hyperprior mode (context neural network 114 is configured to autoregressively process the code symbols 120 representing the input data (i.e., in accordance with an ordering of the code symbols) to generate a respective “context output” 130 (Φ) for each code symbol, [0073]; entropy model neural network 112 can process the context outputs 130 generated by the context neural network 114 (i.e., in addition to the hyper-decoder output 128) to generate the conditional entropy model, [0074]).
Minnen is silent to teaching that with an attention mechanism. However, the claimed limitation is well known in the art as evidenced by Houlsby.
In the same field of endeavor, Houlsby teaches with an attention mechanism (self-attention neural network layer is configured to receive as input a sequence of layer input elements and to apply an attention mechanism over the sequence of layer input elements to generate a sequence of layer outputs elements, [0023]; instead of segmenting the image 102 itself into the image patches 112a-n, the image patch generation system 110 can segment the feature map of the image 102 generated by the embedding neural network of the image patch embedding system 120, [0041]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Minnen with the teaching of Houlsby in order to provide capture dependencies across distant latent positions.
-Regarding claim 19, Minnen discloses a non-transitory computer-readable storage medium storing instructions (memory, [0142]) that cause a processor to perform acts (processor, [0143]) comprising: determining, by using a first model during a conversion between visual data and a bitstream of the visual data, a probability distribution for entropy coding associated with the bitstream (the output of the entropy model neural network 112 includes distribution parameters that define each code symbol probability distribution of the conditional entropy model, [0072]; bit string, [0076]); and performing the conversion based on the probability distribution (code symbol probability distribution, [0105]-[0111]), wherein determining the probability distribution comprises: generating intermediate information based on a quantized latent representation of the visual data by using the first model (the compression system 100 quantizes the latent representation 116 of the input data using a quantizer Q 118 to generate an ordered collection of code symbols 120 (ŷ), [0067]); and generating the probability distribution based on the intermediate information and an output of an autoregressive model and an output of a hyperprior mode (context neural network 114 is configured to autoregressively process the code symbols 120 representing the input data (i.e., in accordance with an ordering of the code symbols) to generate a respective “context output” 130 (Φ) for each code symbol, [0073]; entropy model neural network 112 can process the context outputs 130 generated by the context neural network 114 (i.e., in addition to the hyper-decoder output 128) to generate the conditional entropy model, [0074]).
Minnen is silent to teaching that with an attention mechanism. However, the claimed limitation is well known in the art as evidenced by Houlsby.
In the same field of endeavor, Houlsby teaches with an attention mechanism (self-attention neural network layer is configured to receive as input a sequence of layer input elements and to apply an attention mechanism over the sequence of layer input elements to generate a sequence of layer outputs elements, [0023]; instead of segmenting the image 102 itself into the image patches 112a-n, the image patch generation system 110 can segment the feature map of the image 102 generated by the embedding neural network of the image patch embedding system 120, [0041]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Minnen with the teaching of Houlsby in order to provide capture dependencies across distant latent positions.
-Regarding claim 20, Minnen discloses non-transitory computer-readable recording medium (memory, [0142]) storing a bitstream of visual data ([0059]) which is generated by a method performed by an apparatus for visual data processing, wherein the method comprises: determining, by using a first model, a probability distribution for entropy coding associated with the bitstream (the output of the entropy model neural network 112 includes distribution parameters that define each code symbol probability distribution of the conditional entropy model, [0072]; bit string, [0076]); and generating the bitstream based on the probability distribution ([0059]), wherein determining the probability distribution comprises: generating intermediate information based on a quantized latent representation of the visual data by using the first model (the compression system 100 quantizes the latent representation 116 of the input data using a quantizer Q 118 to generate an ordered collection of code symbols 120 (ŷ), [0067]); and generating the probability distribution based on the intermediate information and an output of an autoregressive model and an output of a hyperprior mode(context neural network 114 is configured to autoregressively process the code symbols 120 representing the input data (i.e., in accordance with an ordering of the code symbols) to generate a respective “context output” 130 (Φ) for each code symbol, [0073]; entropy model neural network 112 can process the context outputs 130 generated by the context neural network 114 (i.e., in addition to the hyper-decoder output 128) to generate the conditional entropy model, [0074]).
Minnen is silent to teaching that with an attention mechanism. However, the claimed limitation is well known in the art as evidenced by Houlsby.
In the same field of endeavor, Houlsby teaches with an attention mechanism (self-attention neural network layer is configured to receive as input a sequence of layer input elements and to apply an attention mechanism over the sequence of layer input elements to generate a sequence of layer outputs elements, [0023]; instead of segmenting the image 102 itself into the image patches 112a-n, the image patch generation system 110 can segment the feature map of the image 102 generated by the embedding neural network of the image patch embedding system 120, [0041]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Minnen with the teaching of Houlsby in order to provide capture dependencies across distant latent positions.
-Regarding claim 21, the combination further discloses the conversion includes decoding the visual data from the bitstream (Minnen, decompression system 200, [0080]).
Allowable Subject Matter
Claim 12 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 2, and 5-21 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PING Y HSIEH whose telephone number is (571)270-3011. The examiner can normally be reached Monday-Friday, 9am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PING Y HSIEH/ Primary Examiner, Art Unit 2664