NON-FINAL REJECTION, FIRST DETAILED ACTION
Status of Prosecution
The present application, 18/414,270 filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The application was filed on Jan. 16, 2024 in the Office
Claims 1-20 are pending and all are rejected. Claims 1, 8 and 15 are independent.
Status of the Claims
Claims 1-2, 7-9, 14-16 are rejected under 35 USC. § 103 as being unpatentable over non-patent literature, Xu et al., “Q-DETR: An Efficient Low-Bit Quantized Detection Transformer,” published in 2023 (“Q-DETR”).
Claims 3, 6, 10, 13, 17 and 20 are rejected under 35 USC. § 103 as being unpatentable over Q-DETR in view of Shoebyi et al., “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism,” published in 2020 (“Megatron-LM”).
Claims 4, 11 and 18 are rejected under 35 USC. § 103 as being unpatentable over Q-DETR in view of Gafni et al. (“Gafni”), United States Patent Application Publication 2024/0221235, published on July 4, 2024.
Claims 5, 12 and 19 are rejected under 35 USC. § 103 as being unpatentable over Q-DETR in view of Liu et al. (“Liu”), United States Patent Application Publication 2021/0375410, published on Dec. 2, 2021.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
A.
Claims 1-2, 7-9, 14-16 are rejected under 35 USC. § 103 as being unpatentable over non-patent literature, Xu et al., “Q-DETR: An Efficient Low-Bit Quantized Detection Transformer,” published in 2023 (“Q-DETR”).
As to Claim 1, Q-DETR teaches: A method, comprising:
accessing a transformer architecture including a set of decoders coupled to receive a set of inputs and generate a set of outputs, wherein at least one decoder includes an attention block, the attention block coupled to receive a query, a key, and a value and generate an attention output (Q-DETR: Fig. 3, the elements of a transformer architecture including decoders to receive query, key value (q, k ,v));
for one or more iterations:
obtaining a batch of training instances for a current iteration (Q-DETR: Sec. 5.1, the training takes place over several epochs in different experiments with different batch sizes),
applying parameters of the transformer architecture for the current iteration to a set of inputs obtained from the batch of training instances to generate a set of estimated outputs, the applying comprising for an attention block: obtaining a query, a key, and a value from the set of inputs (Q-DETR: Sec. 5.1, Fig. 3, the inputs are applied to generate outputs per a quantization of the q, k, v; see also equation 3), and applying a clipping operation to values of the query, the key, and the value (Q-DETR: equation 1 and sec. 3.1, here there is a quantization and symmetric weight quantization using a clipping function on the activation values of q, k, v),
determining a loss function indicating a difference between data in the batch of training instances and the set of estimated outputs (Q-DETR: Sec. 42., the loss function is calculated in equation 14), and
backpropagating terms obtained from the loss function to update the parameters of the transformer architecture (Q-DETR: Sec. 4.2, In backward propagation, the gradient updating drives the student queries toward their teacher hints. Fig. 3); and
PNG
media_image1.png
474
1048
media_image1.png
Greyscale
Q-DETR may not explicitly teach: deploying the trained transformer architecture to an inference system.
Q-DETR does however discuss the use of inference systems and Figure 3 also depicts a larger architecture of learning that could be used for passing the model on for inference (Q-DETR: Fig. 3, the inference system is seen in the teacher, student network, Introduction, “Despite this attractiveness, DETR usually has a tremendous number of parameters and float-pointing operations (FLOPs).”).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the invention to have modified Q-DETR by deploying the trained and quantized system to an inference system for use. Such a person would have been motivated to do so with a reasonable expectation of success to do so for realizing the gains of the optimized system that may now perform on devices of different compute resources (Q-DETR: Introduction, “This leads to an unacceptable memory and computation consumption during inference, and challenges deployments on devices with limited supplies of resources.”)
As to Claim 2, Q-DETR and Q-ViT teach the limitations of claim 1.
Q-DETR further teaches: wherein the clipping operation is associated with an interval or values, and applying the clipping operation comprises setting a value to a first value if the value is below a first threshold of the interval, or setting the value to a second value if the value is above a second threshold of the interval (Q-DETR: equations 1 and 2 and sec. 3.1, a quantization and symmetric weight quantization using a clipping function on the activation values of q, k, v).
PNG
media_image2.png
508
480
media_image2.png
Greyscale
As to Claim 7, Q-DETR teaches the limitations of claim 1.
Q-DETR further teaches: wherein each of the first hardware acceleration device and the second hardware acceleration device are graphics processor units (GPU's) (Q-DETR: Sec. 5.1, NVIDIA Tesla A100 GPUs were used).
As to Claim 8, it is rejected for similar reasons as claim 1.
As to Claim 9, it is rejected for similar reasons as claim 2.
As to Claim 14, it is rejected for similar reasons as claim 7.
As to Claim 15, it is rejected for similar reasons as claims 1 and 8.
As to Claim 16, it is rejected for similar reasons as claim 2.
B.
Claims 3, 6, 10, 13, 17 and 20 are rejected under 35 USC. § 103 as being unpatentable over non-patent literature, Xu et al., “Q-DETR: An Efficient Low-Bit Quantized Detection Transformer,” published in 2023 (“Q-DETR”), in view of Shoebyi et al., “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism,” published in 2020 (“Megatron-LM”).
As to Claim 3, Q-DETR teaches the limitations of claim 1.
Q-DETR may not explicitly teach: wherein the transformer architecture is coupled to receive a set of input tokens representing text as the set of inputs and generate a set of output tokens representing text as the set of outputs.
Megatron-LM teaches in general concepts related to training large transformer model techniques (Megatron-LM: Abstract). Specifically, Megatron teaches that text input is contemplated and tokens are used (Megatron: “Since the vocabulary size is on the order of tens of thousands of tokens for modern language models”).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the invention to have modified Q-DETR by utilizing the system with ability to receive text tokens and to output text as taught and suggested by Megatron-LM. Such a person would have done so with an expectation for success, because of the utility of using the system for text transformer machine learning uses.
As to Claim 6, Q-DETR teaches the limitations of claim 1.
Q-DETR further teaches: wherein applying the parameters for the attention block comprises:
generating a first output matrix on a first hardware acceleration device for a first attention head by applying the clipping operation to the values of the query, the key, and the value for the first attention head, and combining the clipped query, key, and value to generate the first output matrix (Q-DETR: equation 1 and sec. 3.1, here there is a quantization and symmetric weight quantization using a clipping function on the activation values of q, k, v;
tensor multiplying the first output matrix with a first subset of weights from an attention weight matrix to generate a first subset of attention representations (Q-DETR: equation 2 gives the quantized fully-connected layer which utilizes matrix multiplication).
Q-DETR may not explicitly teach: generating a second output matrix on a second hardware acceleration device for a second attention head by applying the clipping operation to values of a second query, a second key, and a second value for a second attention head, and combining the clipped second query, second key, and second value to generate the second output matrix;
tensor multiplying the second output matrix with a second subset of weights from the attention weight matrix to generate a second subset of attention representations; and
combining the first subset of attention representations with the second subset of attention representations.
Megatron-LM teaches in general concepts related to training large transformer model techniques (Megatron-LM: Abstract). Specifically, Megatron-LM teaches that a tensor operation may distributed across multiple devices (Megatron-LM: Sec. 2.3, the partition may be done accelerate computation or increate model size).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the invention to have modified Q-DETR by performing the tensor operations over two different devices t as taught and suggested by Megatron-LM. Such a person would have done so with an expectation for success for compute resource and efficiency considerations.
As to Claim 10, it is rejected for similar reasons as claim 3.
As to Claim 13, it is rejected for similar reasons as claim 6.
As to Claim 17, it is rejected for similar reasons as claim 3.
As to Claim 20, it is rejected for similar reasons as claim 6
C.
Claims 4, 11 and 18 are rejected under 35 USC. § 103 as being unpatentable over non-patent literature, Xu et al., “Q-DETR: An Efficient Low-Bit Quantized Detection Transformer,” published in 2023 (“Q-DETR”), in view of Gafni et al. (“Gafni”), United States Patent Application Publication 2024/0221235, published on July 4, 2024.
As to Claim 4, Q-DETR teach the limitations of claim 1.
Q-DETR may not explicitly teach: wherein the transformer architecture is coupled to receive a set of input tokens representing text as the set of inputs and generate a set of output pixels or latent pixels representing an image as the set of outputs.
Gafni teaches in general concepts related to receiving text and scene tokens for input into a machine-learning model to generate an image corresponding to the text and scene inputs (Gafni: Abstract). Specifically, Gafni teaches that text, scene and image tokens may be received by a transformer to allow for generation of an image (Gafni: Fig. 2, par. 0031, “The text encoder 220 may generate text tokens 225 from the input text 205. If there is an input scene 210, the scene encoder 230 may generate scene tokens 235 from the input scene 210. The transformer 250 may then predict image tokens 245 based on the text tokens 225 and scene tokens 235 (if there is an input scene 210). The predicted image tokens 245 may be provided to an image decoder 255. The image decoder 255 may further generate the image 260 for the inference.”)
PNG
media_image3.png
638
852
media_image3.png
Greyscale
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the invention to have modified Q-DETR by utilizing the system with ability to receive text tokens and generate an output of image pixels as taught and suggested by Gafni. Such a person would have done so with an expectation for success, to allow for wider use and application to other use cases for generative content.
As to Claim 11, it is rejected for similar reasons as claim 4.
As to Claim 18, it is rejected for similar reasons as claim 4.
D.
Claims 5, 12 and 19 are rejected under 35 USC. § 103 as being unpatentable over non-patent literature, Xu et al., “Q-DETR: An Efficient Low-Bit Quantized Detection Transformer,” published in 2023 (“Q-DETR”), in view of Liu et al. (“Liu”), United States Patent Application Publication 2021/0375410, published on Dec. 2, 2021.
As to Claim 5, Q-DETR teaches the limitations of claim 1.
Q-DETR further teaches: further comprising for another one or more iterations, training the parameters of the transformer architecture, wherein the clipping operation is not applied for the another one or more iterations.
Liu teaches in general concepts related to a board card device and a storage medium for data quantization (Liu: Abstract). Specifically, Liu teaches that a fine-tuning and training of a neural network (Liu: par. 0172). Decisions on whether to quantize and how data is to be quantized for each layer for each iteration is determined based on variation of range and other considerations (Liu: par. 0172).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the invention to have modified the Q-DETR device and methods by including computer instructions to allow selectively applying the clipping operations on different iterations and disclosed by Liu. Such a person would have been motivated to do so with a reasonable expectation of success to do so to reduce unnecessary computational expenses for clipping when it may not be needed.
As to Claim 12, it is rejected for similar reasons as claim 5.
As to Claim 19, it is rejected for similar reasons as claim 5.
Conclusion
Prior art not relied upon but made of the record as they are relevant to Applicant’s disclosure:
Li et al., “Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer” (2022).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAMES T TSAI whose telephone number is (571)270-3916. The examiner can normally be reached M-F 8-5 Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAMES T TSAI/ Primary Examiner, Art Unit 2147
Prosecution Insights