Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Note on prior art
There is no prior art rejection for claim 14The prior art of record does not teach or make obvious that the adapter latency is lower than the block’s latency. On Architectural Compression of Text-to-Image Diffusion Models by Kim et al teaches that removing the blocks speeds things up, but it does not teach that adding an adapter will have lower latency than the blocks.1
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: the “means for” in claim 30.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 30 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because applicant claims a system with “means for” performing operations. The “means”, in spec. 205, may be only “software components”. A system claim with no structural recitation is software/data per se, and it is not patent eligible subject matter. MPEP 2106.03.
Claims 1-30 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of a mathematical relationship without significantly more. The claims recite the abstract idea of generating tensors and generating an image based on the tensors. This judicial exception is not integrated into a practical application because additional elements of training a diffusion ML model and displaying an image are insignificant extra-solution activity. MPEP 2106.05(g). The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements of processors and memory are generic computer parts. MPEP 2106.05(f).
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claim 30 is rejected under 35 U.S.C. 112(a) or pre-AIA 35 U.S.C. 112, first paragraph, because the claim purports to invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, but fails to recite a combination of elements as required by that statutory provision and thus cannot rely on the specification to provide the structure, material or acts to support the claimed function. As such, the claim recites a function that has no limits and covers every conceivable means for achieving the stated function, while the specification discloses at most only those means known to the inventor. Accordingly, the disclosure is not commensurate with the scope of the claim.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 30 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. To claim a means for performing a specific computer-implemented function and then to disclose only a general purpose computer as the structure designed to perform that function amounts to pure functional claiming. In this instance, the structure corresponding to a 35 U.S.C. 112(f) claim limitation for a computer-implemented function must include the algorithm needed to transform the general purpose computer or microprocessor disclosed in the specification. Accordingly, a rejection under 35 U.S.C. 112(b) is appropriate if the specification discloses no corresponding algorithm associated with a computer or microprocessor. MPEP 2182(II)(B). Because applicant claims a means for performing operations, and then discloses generic computer parts (Spec. 205) as means for performing the operations, Applicant must disclose an algorithm for performing the claimed steps of generating Applicant’s different tensors. Applicant has not disclosed algorithms for generating tensors. Therefore, the claim is indefinite.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-11, 14-26 and 28-30 are rejected under 35 U.S.C. 103 as being unpatentable over A Closer Look at Parameter-Efficient Tuning in Diffusion Models to Xiang et al and High-Resolution Image Synthesis with Latent Diffusion Models to Rombach et al
Claims 12 and 27 are rejected under 35 U.S.C. 103 as being unpatentable over A Closer Look at Parameter-Efficient Tuning in Diffusion Models to Xiang et al, High-Resolution Image Synthesis with Latent Diffusion Models to Rombach et al and On Distillation of Guided Diffusion Models by Meng et al.
Xiang teaches claims 1, 16, 29 and 30. A processing system comprising:
one or more memories comprising processor-executable instructions; and
one or more processors configured to execute the processor-executable instructions and cause the processing system to: (Xiang fig. 3 below)
PNG
media_image1.png
524
904
media_image1.png
Greyscale
generate, during a first iteration of processing data using a denoising backbone of a diffusion machine learning model, a first latent tensor using a lower resolution block of the denoising backbone; (Xiang’s fig 3 input position is the first latent vector. It is generated at the last layer, below, from a lower-resolution previous layer/block.)
PNG
media_image2.png
524
904
media_image2.png
Greyscale
generate, during the first iteration, a first feature tensor based on processing the first latent tensor using a higher resolution block of the denoising backbone, the higher resolution block being configured to use a higher resolution than the lower resolution block; (Xiang fig. 3 output position from the residual block’s in/out blocks. The last layer/block in fig. 3 is higher resolution than the second-to-last layer before it.)
generate a second latent tensor based on processing the first latent tensor using an adapter block of the denoising backbone; and (Xiang fig. 3 output position from the adapter is the second latent tensor.)
generate, (Xiang fig. 3 shows that the output from the adapter is combined with the output from the out block in the residual high res block. That combination creates a second feature tensor using the residual high res block. O the second feature tensor is the tensor that actually leaves the residual block and goes to the transformer block.)
Xiang doesn’t teach a second iteration.
However, Rombach teaches how to generate, during a second iteration of processing the data using the denoising backbone, a second feature tensor based on processing the second latent tensor using the higher resolution block. (Rombach sec. 3.1 “the encoder E encodes x into a latent representation z = E(x), and the decoder D reconstructs the image from the latent, giving ˜x = D(z) = D(E(x))…” ˜x is the feature tensor, Rombachh fig. 3 below. The latent representation z is the latent tensor. The second latent tensor shown during the second iteration is labeled as zt-1 in Rombach fig. 3 below.)
PNG
media_image3.png
464
840
media_image3.png
Greyscale
Rombach, the claims and Xiang are all noisy diffusion models. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to add a second iteration of processing because the time-conditional UNet fixes the forward process, so “zt can be efficiently obtained… during training.” Rombach sec. 3.2.
Xiang teaches claims 2 and 17. The processing system of claim 1, wherein, to generate the second latent tensor, the one or more processors are configured to execute the processor-executable instructions to cause the processing system to process an embedding (Xiang fig. 3 adapter output is the processed embedding called “input position”, see below.)
PNG
media_image4.png
524
904
media_image4.png
Greyscale
Xiang teaches claims 3 and 18. The processing system of claim 1, wherein to generate the second latent tensor, the one or more processors are configured to execute the processor-executable instructions to cause the processing system to process an embedding corresponding to an input to the diffusion machine learning model using the adapter block. (Xiang fig. 3 input position corresponds to an input for the diffusion model in fig. 3.)
Xiang teaches claims 4 and 19. The processing system of claim 1, wherein, to generate the second latent tensor, the one or more processors are configured to execute the processor-executable instructions to cause the processing system to process an embedding, generated by the higher resolution block, using the adapter block. (Xiang fig. 3 shows the higher resolution blocks on the right side of the model.)
Xiang teaches claims 5 and 20. The processing system of claim 1, wherein the adapter block is configured to perform an identity mapping. (Xiang sec. 3.2 “The activation functions include ReLU, Sigmoid, SiLU, and identity operator as our design choices, and the scale factors include 0.5, 1.0, 2.0, 4.0.” Identity operator is the identity mapping. And the activation function is in the adapter block, see Xiang fig. 5 below.)
PNG
media_image5.png
298
692
media_image5.png
Greyscale
Xiang teaches claims 6 and 21. The processing system of claim 1, wherein the adapter block is configured to use a set of learned parameters to generate the second latent tensor based on the first latent tensor. (Xiang abs ‘we investigate parameter-efficient tuning in large diffusion models by inserting small learnable modules (termed adapters).”)
Xiang teaches claims 7 and 22. The processing system of claim 6, wherein the adapter block is configured to perform one or more convolution operations to generate the second latent tensor. (Xiang abs ‘we investigate parameter-efficient tuning in large diffusion models by inserting small learnable modules (termed adapters).” Xiang sec. 3.2 “the residual block adapter employs 3×3 convolution layers Convdown and Convup as the down-sampling and up-sampling operators respectively.”)
Xiang teaches claims 8 and 23. The processing system of claim 6, wherein:
the adapter block comprises an encoder and a decoder, and
to generate the second latent tensor, the one or more processors are configured to execute the processor-executable instructions to cause the processing system to:
generate a compressed tensor based on processing the first latent tensor using the encoder; and
generate the second latent tensor based on processing the compressed tensor using the decoder. (The output position is the second latent tensor, Xiang fig. 3. Xiang fig. 5, below, shows the compression and decoding steps as trapezoids on either side of the activation function.)
PNG
media_image5.png
298
692
media_image5.png
Greyscale
Xiang teaches claims 9 and 24. The processing system of claim 1, wherein the one or more processors are configured to further execute the processor-executable instructions to cause the processing system to:
generate a third latent tensor based on processing the first latent tensor using the adapter block; and
generate, during a third iteration of processing the data using the diffusion machine learning model, a third feature tensor based on processing the third latent tensor using the higher resolution block. (Xiang fig. 5, below, output of the high resolution block is the third feature tensor. The input of the high res block is the third latent tensor. The third feature tensor is based on the second latent tensor, because the input to the low res block is the second latent tensor.)
PNG
media_image6.png
320
708
media_image6.png
Greyscale
Xiang teaches claims 10 and 25. The processing system of claim 1, wherein the one or more processors are configured to further execute the processor-executable instructions to cause the processing system to:
generate a third latent tensor based on processing the second latent tensor using the adapter block; and
generate, during a third iteration of processing the data using the diffusion machine learning model, a third feature tensor based on processing the third latent tensor using the higher resolution block. (Xiang fig. 5, below, output of the high resolution block is the third feature tensor. The input of the high res block is the third latent tensor. The third feature tensor is based on the second latent tensor, because the input to the low res block is the second latent tensor.)
PNG
media_image7.png
320
708
media_image7.png
Greyscale
Xiang teaches claims 11 and 26. The processing system of claim 1, wherein the one or more processors are configured to further execute the processor-executable instructions to cause the processing system to:
generate, during a third iteration of processing the data using the diffusion machine learning model, a third latent tensor using the lower resolution block; and
generate, during the third iteration, a third feature tensor based on processing the third latent tensor using the higher resolution block. (Xiang fig. 5, below, output of the high resolution block is the third feature tensor. The input of the high res block is the third latent tensor.)
PNG
media_image8.png
320
708
media_image8.png
Greyscale
Xiang teaches claims 12 and 27. The processing system of claim 1, wherein:
the diffusion machine learning model was trained
the (Xiang fig. 5 shows the blocks and the mode. Xiang sec. 5.1 “ix the adapter size to 1.5M (0.17% of the UNet model), and train with 2.5k steps.”)
Xiang isn’t a teacher-student training model.
However, Meng teaches the diffusion machine learning model was trained using distillation from a teacher machine learning model, and
the teacher machine learning model uses a plurality of higher resolution blocks and a corresponding plurality of lower resolution blocks. (Meng sec. 1 “In the first stage, we introduce a single student model to match the combined output of the two diffusion models of the teacher.” Meng sec. 2 “Latent diffusion models can be considered as an alternative to cascaded diffusion approaches [5], which rely on one or more super-resolution diffusion models to scale up a low-dimensional image to the desired target resolution.”)
Meng, Xiang and the claims are all directed to diffusion models. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use a teacher to transfer learning because Meng’s knowledge distillation method “is able to generate images visually comparable to that of the original model using as few as 4 sampling steps…” Meng abs.
Xiang teaches claim 14. The processing system of claim 1, wherein the second latent tensor is not generated based on the first feature tensor. (There is no feedback loop where the output from the output block in Xiang fig. 3 is fed into the adapter to get the second latent tensor, the second latent tensor is not generated based on the first feature tensor.)
Xiang teaches claims 15 and 28. The processing system of claim 1, wherein the one or more processors are configured to further execute the processor-executable instructions to cause the processing system to:
generate an image based at least in part on the first and second feature tensors; and
output the image as output from the diffusion machine learning model. (Xiang fig. 2 show images generated using the feature tensors from the block sin fig. 3, and the output is an image, “Images generated by fully fine-tuned method (DreamBooth [32]) and our parameter-efficient tuning method with the best setting (Ours) on personalization tasks.”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Austin Hicks whose telephone number is (571)270-3377. The examiner can normally be reached Monday - Thursday 8-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUSTIN HICKS/Primary Examiner, Art Unit 2142
1 Kim fig. 2 “the denoising U-Net is the main processing bottleneck.” Kim sec. 1 “We compress SDMs by removing architectural blocks from the U-Net, achieving up to 51% reduction in model size and 43% improvement in latency on CPU and GPU.”
Prosecution Insights