Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The action is in response to claims dated 5/14/2024
Claims pending in the case: 1-20
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Regarding claims 20, these claims are for A computer-readable storage medium storing instructions to perform various functions. However, the computer-readable storage medium could be interpreted as signals. Therefore, the claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter.
All claims dependent on the rejected claim, are also rejected due to their direct or indirect dependencies.
Examiner suggests using the term “non-transitory” to overcome this rejection.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 6, 8, 15-17, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang (CN 120085875 - Please refer to the attached documents for claim mapping) in view of Zhang (US 20210056357).
Regarding Claim 1, Wang teaches, an apparatus for finetuning one or more neural networks, the apparatus comprising: a memory; and a processor coupled to the memory (Wang: Pg. 2 [7]) and configured to:
process, using a first trained neural network, data specific to a user to obtain … activation data representing the data, the first trained neural network comprising a plurality of neural network layers (Wang: Pg. 4 [1-4], Pg. 12 [8]: first model based on user data);
process, using a second trained neural network, the … activation data to generate an output representing the data, the second trained neural network comprising a subset of neural network layers from the plurality of neural network layers of the first trained neural network (Wang: Pg. 4 [1-4], Pg. 13 [1-2]: second model by compressing first model; output of first model to guide and train smaller model);
determine a loss based on the output; and update parameters of the second trained neural network based on the loss (Wang: Pg. 9 [3]: update model parameters as per loss function);
However Wang does not specifically teach, intermediate activation data,
Zhang teaches, intermediate activation data (Zhang: [104, 142]: early exit (intermediate activation) to generate output);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Wang and Zhang because the combination would enable using intermediate activation data as a model output. One of ordinary skill in the art would have been motivated to combine the teachings because the combination would reduce computational consumption of large models (see Zhang [4,7]).
Regarding Claim 2, Wang and Zhang teach the limitations as claimed in claim 1 and, wherein the second trained neural network is generated based on removing at least one neural network layer from the plurality of neural network layers of the first trained neural network (Wang: Pg. 4 [3-4], Pg. 17 [3-4]: second model by compressing first model which may be clipping or removing layers).
Regarding Claim 6, Wang and Zhang teach the limitations as claimed in claim 1 and, wherein the processor is configured to:
load the first trained neural network into the processor to process the data, wherein the second trained neural network is not loaded into the processor when processing the data (Wang: Pg. 4 [1-4], Pg. 12 [8]: first model based on user data); and
load the second trained neural network into the processor to process the intermediate activation data, determine the loss, and update the parameters of the second trained neural network (Wang: Pg. 4 [1-4], Pg. 13 [1-2]: second model by compressing first model; output of first model to guide and train smaller model), wherein the first trained neural network is not loaded into the processor when processing the intermediate activation data (Wang: Pg. 4 [2-3]: each model loaded separately) (Zhang: [104, 142]: early exit (intermediate activation) to generate output), determining the loss, and updating the parameters of the second trained neural network (Wang: Pg. 9 [3]: update model parameters as per loss function).
Regarding Claim 8, Wang and Zhang teach the limitations as claimed in claim 1 and, wherein the processor comprises a graphics processing unit, a neural processing unit, a neural signal processor, or a digital signal processor (Wang: Pg. 2 [7]: processor) (Zhang: [25]: processor).
Regarding Claim(s) 15-17, this/these claim(s) is/are similar in scope as claim(s) 1-2, 6 respectively. Therefore, this/these claim(s) is/are rejected under the same rationale.
Regarding Claim(s) 20, this/these claim(s) is/are similar in scope as claim(s) 1. Therefore, this/these claim(s) is/are rejected under the same rationale.
Claim(s) 3-5, 7, 9-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang (CN 120085875 - Please refer to the attached documents for claim mapping) in view of Zhang (US 20210056357) in view of (Kim - BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion).
Regarding Claim 3, Wang and Zhang teach the limitations as claimed in claim 2 and, wherein the second trained neural network is generated further based on removing … from the subset of neural network layers (Wang: Pg. 4 [3-4], Pg. 17 [3-4]: second model with simpler structure);
Although Wang and Zhang do not specifically teach, removing a plurality of parameters, Wang in Pg. 17 [4] recites that “the first model can also be compressed in other ways”. Pruning to reduce network by removing parameters is well known in the art;
Nonetheless, Kim teaches, removing a plurality of parameters (Kim: Pg. 1 abstract, section 3.2, Pg. 2 col 1 [2]: pruning to obtain reduced model);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Wang, Zhang and Kim because the combination would enable pruning to reduce model size in a diffusion model. One of ordinary skill in the art would have been motivated to combine the teachings because the combination introduces an efficient network architecture and reduces computation resources and latency in diffusion models (see Kim Abstract).
Regarding Claim 4, Wang, Zhang and Kim teach the limitations as claimed in claim 2 and, wherein the at least one neural network layer is removed based on a task (Kim: Pg. 3 Fig. 5, col 2 [1], Pg. 4 col 1 section 3.1.4: removed based on importance).
Regarding Claim 5, Wang, Zhang and Kim teach the limitations as claimed in claim 4 and, wherein the task comprises an image generation task (Kim: Pg. 1 col 2 [1], Pg. 4 col 1 section 3.2: image generation) (Wang: pg. 8 [9]: image recognition) (Zhang: [17, 54]: image processing).
Regarding Claim 7, Wang and Zhang teach the limitations as claimed in claim 1 and, Kim further teaches, wherein the first trained neural network comprises a diffusion neural network (Kim Pg. 1 col 2: diffusion model);
The same motivation to combine as stated above applies.
Regarding Claim 9, Wang and Zhang teach the limitations as claimed in claim 1 and, Kim further teaches, wherein the data specific to the user comprises an item of media content and a text input associated with the item of media content (Kim: Pg. 2 col 1-2 section 2: diffusion model may be text conditional diffusion model generating image).
The same motivation to combine as stated above applies.
Regarding Claim 10, Wang and Zhang teach the limitations as claimed in claim 1 and, Kim further teaches, wherein the output comprises an image of a particular object, and wherein the text input comprises a text prompt to generate the image of the particular object (Kim: Pg. 2 col 1-2 section 2: diffusion model may be text conditional diffusion model generating image).
The same motivation to combine as stated above applies.
Claim(s) 11, 13-14, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang (CN 120085875 - Please refer to the attached documents for claim mapping), Zhang (US 20210056357) in view of (Hu - LoRA: Low-Rank Adaptation of Large Language Models).
Regarding Claim 11, Wang and Zhang teach the limitations as claimed in claim 1 but not, wherein the processor is configured to: transfer updated parameters of the second trained neural network to the first trained neural network to generate a finetuned first neural network; and perform inference on input data using the finetuned first neural network.
Hu teaches, transfer updated parameters of the second trained neural network to the first trained neural network to generate a finetuned first neural network; and perform inference on input data using the finetuned first neural network (Hu: Pg. 2 [3] LoRA advantages allows merging the updated matrices with the original weights during deployment; Pg. 3 section 4 [4]: learned weights merged with the main weights during inference);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Wang, Zhang and Hu because the combination would enable using a low-Rank adaptation approach to train the small models and merge the updates to the original in the field of image generation. One of ordinary skill in the art would have been motivated to combine the teachings because the combination “makes training more efficient and lowers the hardware barrier” (see Hu Abstract and Pg. 2 [3]).
Regarding Claim 13, Wang, Zhang and Hu teach the limitations as claimed in claim 11 and, wherein the processor is configured to: maintain, based on a use case parameter, at least one of the finetuned first neural network or the second trained neural network in the memory (Wang: Pg. 2 [6]: model based on use case preference parameter).
Regarding Claim 14, Wang, Zhang and Hu teach the limitations as claimed in claim 13 and, wherein the use case parameter comprises at least one of a memory requirement for an inference task or a parameter associated with the inference task being a personalized task or a general task (Wang: Pg. 2 [6]: model based on use case preference parameter of hardware resource).
Regarding Claim(s) 18, this/these claim(s) is/are similar in scope as claim(s) 11. Therefore, this/these claim(s) is/are rejected under the same rationale.
Claim(s) 12, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang (CN 120085875 - Please refer to the attached documents for claim mapping), Zhang (US 20210056357) and (Hu - LoRA: Low-Rank Adaptation of Large Language Models) in view of (Kim - BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion).
Regarding Claim 12, Wang, Zhang and Hu teach the limitations as claimed in claim 11 but not, wherein the processor is configured to obtain the input data based on user input from the user, the input data comprising a text prompt to generate an image comprising a particular object;
Hu teaches, wherein the processor is configured to obtain the input data based on user input from the user, the input data comprising a text prompt to generate an image comprising a particular object (Kim: Pg. 4 section 3.2 [1], Pg. 6 Fig. 7: image generation from prompt);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Wang, Zhang, Hu and Kim because the combination would enable image generation from user prompts. One of ordinary skill in the art would have been motivated to combine the teachings because the combination makes the system more user friendly by incorporating the practice of content generation based on user prompt common in the art.
Regarding Claim(s) 19, this/these claim(s) is/are similar in scope as claim(s) 12. Therefore, this/these claim(s) is/are rejected under the same rationale.
Examiner’s suggestion
For compact prosecution, examiner suggests adding specifics from specification paragraphs [65-66] to the independent claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure listed below in the attached 892.
Rombach et. al. (High-Resolution Image Synthesis with Latent Diffusion Models) which teaches introducing cross-attention layers into the model architecture of diffusion models. The art discuss perceptual compression models using low-dimensional latent space and time-conditional UNet.
Zhang et. al (Adding Conditional Control to Text-to-Image Diffusion Models) teaches model compression using a control net (Fig. 2).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MANDRITA BRAHMACHARI whose telephone number is (571)272-9735. The examiner can normally be reached Monday to Friday, 11 am to 8 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 571 272 4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Mandrita Brahmachari/Primary Examiner, Art Unit 2144
