DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 9 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
In regards to claim 9, claim 9, dependent on claim 7, recites the limitations "the encoder blocks" and “the middle block” in lines 1-2. There is insufficient antecedent basis for this limitation in the claim. In the interest of compact prosecution, Examiner is viewing claim 9 to be dependent on claim 8, instead of the stated claim 7, as claim 8 initializes “encoder blocks” and “middle block”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 7, 12-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shin (US 2024/0419755 A1) in view of Park et al. (US 2021/0358177 A1).
In regards to claim 1, Shin teaches a computer-implemented method for post-training a machine learning system, wherein the machine learning system is configured to generate digital images under a specification of a spatial image structure, wherein the machine learning system is a generative probabilistic text-to-image diffusion model (e.g. [0035]: the interface-guided diffusion model 152 is a text-to-image ML model capable of receiving one or more prompts 101 that include the user-generated text 128 and the input condition data 116), wherein blocks of the diffusion model with parameters of the blocks are duplicated into a locked copy and a trainable copy (e.g. [0036]: the interface-guided diffusion model 152 may include a locked neural network block 153 and a trainable neural network block 155, where the weights of the locked neural network block 153 are copied and transferred to the trainable neural network block 155), the method comprising the following steps:
receiving training data, which each contain specifications of a spatial image structure, by the machine learning system (e.g. [0037]: trainable neural network block 155 is trained using training input condition data (e.g. training input condition data 216 of Fig.2) to learn the input condition(s) (e.g. the UI layout information 118, display screen information 120, limited color scheme indicated by the color vision deficiency setting 124, etc.) and the locked neural network block 153 may preserve the weights of the neural network; see also [0031]: UI layout information 118 may include information (e.g. metadata, UI edge map data, and/or image data) that indicates a size and/or location of a UI element 112 in the interface 108; Examiner’s note: as such, input condition data, which may be UI layout information (spatial image structure), can be training data);
adding further parameters to the trainable copy of at least one duplicated block of the diffusion model (e.g. further in [0037],Fig.1D: convolution layer 154 may apply a convolution with the learned parameters to the input condition data 116; then, the input condition data 116 is combined with the user-generated text 128 and inputted into the trainable neural network block 155; Examiner’s note: where the added convolution layer results in the added further parameters);
wherein:
the further parameters are added to the trainable copy of a duplicated block by inserting at least one additional layer parameterized with the further parameters (e.g. as above, [0037]: convolution layer 154 may apply a convolution with the learned parameters to the input condition data 116; then, the input condition data 116 is combined with the user-generated text 128 and inputted into the trainable neural network block 155), and/or the further parameters are added by decomposing at least one weight matrix to be adjusted in the post-training, in a trainable copy of a duplicated block, into a sum of a pre-trained weight matrix and a further summand added in the post-training, wherein the further summand is given by the matrix product of two further matrices, wherein the two further matrices are parameterized with the further parameters, wherein the parameters of the pre-trained weight matrix have been adjusted in a pre-training and are retained in the post-training, wherein ranks of the two further matrices are each lower than a rank of the pre-trained weight matrix;
generating an image for each training datum by the machine learning system (e.g. further in [0037]: the convolution layer 158 may apply a convolution to the output of the trainable neural network block 155, thereby producing the UI-compatible output image 110);
but does not explicitly teach the method, comprising:
ascertaining a spatial image structure of an image generated by the machine learning system, by a second machine learning system for ascertaining a spatial image structure of a digital image; and
adjusting the further parameters by using a loss function which measures a similarity between the ascertained spatial image structure of the image generated by the machine learning system and the spatial image structure specified by the training datum associated with the generated image.
However, Park teaches a method, comprising:
ascertaining a spatial image structure of an image generated by the machine learning system, by a second machine learning system for ascertaining a spatial image structure of a digital image (e.g. [0062]: to learn the parameters that allow the global and spatial autoencoder 112 to generate the reconstructed digital image 304 as an accurate representation of the input digital image 302, the deep image manipulation system 102 utilizes one or more loss functions; indeed, with each iteration of analyzing a new input digital image to generate a reconstructed digital image as part of the parameter learning, the deep image manipulation system 102 utilizes loss functions to modify internal parameters of the encoder neural network 206 and/or the generator neural network 216; more specifically, the deep image manipulation system 102 utilizes loss functions to evaluate a performance of the global and spatial autoencoder 112 by determining an error or a measure of loss associated with generating a reconstructed digital image (e.g. the reconstructed digital image 304) from an input digital image (e.g. the input digital image 302); the deep image manipulation system 102 further modifies various weights or other internal parameters of the global and spatial autoencoder 112 based on the error or measure of loss utilizing gradient-based back propagation); and
adjusting the further parameters by using a loss function which measures a similarity between the ascertained spatial image structure of the image generated by the machine learning system and the spatial image structure specified by the training datum associated with the generated image (e.g. as above, [0062]: the deep image manipulation system 102 utilizes loss functions to modify internal parameters of the encoder neural network 206 and/or the generator neural network 216; the deep image manipulation system 102 further modifies various weights or other internal parameters of the global and spatial autoencoder 112 based on the error or measure of loss utilizing gradient-based back propagation).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings/combination of Shin to modify using loss functions, in the same conventional manner as taught by Park as both deal with generating digital images using machine learning. The motivation to combine the two would be that it would allow the determination of performance for the resulting image, and modify parameters based on the determination.
In regards to device claim 12 and medium claim 13, claim(s) 12-13 recite(s) limitations that is/are similar in scope to the limitations recited in claim 1. Therefore, claim(s) 12-13 is/are subject to rejections under the same rationale as applied hereinabove for claim 1.
In regards to claim 7, the combination of Shin and Park teaches a method, wherein each training datum further includes text, which includes multiple words and describes a quality and/or a content of the image to be generated by the machine learning system (e.g. Shin as above, [0035]: the interface-guided diffusion model 152 is a text-to-image ML model capable of receiving one or more prompts 101 that include the user-generated text 128 and the input condition data 116; see also [0027]: user-generated text 128 may include a natural language description 130 about the generation of a UI-compatible output image 110; [0029]: a user may enter, via the query interface 126, user-generated text 128 (e.g. “create an image of trees having wet leaves”) about an image to be generated).
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Shin and Park as applied to claim 1 above, and further in view of Buscema (US 2006/0230006 A1).
In regards to claim 2, the combination of Shin and Park teaches the method of claim 1, but does not explicitly teach the method, wherein the further parameters are adjusted using a distribution-based evolutionary algorithm.
However, Buscema teaches a method, wherein the further parameters are adjusted using a distribution-based evolutionary algorithm (e.g. [0034]: an evolutionary algorithm can be provided which combines the different models of distribution of the records of the complete data set in a training subset and a testing subset; each model of distribution is represented by a corresponding prediction algorithm which has been trained and tested using the training and testing data set from that distribution model, and scored according to a fitness score calculated; [0091]: evolutionary algorithm can provide, for example, for the formation of a "child" generation of prediction algorithms 221, based on a new distribution of records onto the training and testing set, such distribution being obtained by merging or mutating the distribution of records of the parent algorithms; Examiner’s note: this shows that parameters are iteratively adjusted until optimum score is reached).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings/combination of Shin and Park to use evolutionary algorithm, in the same conventional manner as taught by Buscema as both deal with machine learning/training. The motivation to combine the two would be that using the evolutionary algorithm would allow the optimization of the model.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Shin and Park as applied to claim 1 above, and further in view of Shanbhag et al. (US 2021/0406681 A1).
In regards to claim 3, the combination of Shin and Park teaches the method of claim 1, but does not explicitly teach the method, wherein the loss function is given by a non-differentiable metric.
However, Shanbhag teaches a method, wherein the loss function is given by a non-differentiable metric (e.g. [0033]: the loss function DL model 108 can comprise a model trained to predict or otherwise generate a non-differentiable loss function metric or value).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings/combination of Shin and Park to generate non-differentiable loss function metrics, in the same conventional manner as taught by Shanbhag as both deal with use of loss functions in machine learning. The motivation to combine the two would be that it would allow the generation of non-differentiable loss function metrics, as it would circumvent use of loss function metrics that are computationally complex and/or otherwise difficult to implement (see [0033]).
Claim(s) 8-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Shin and Park as applied to claim 1 above, and further in view of Yu et al. (US 2024/0386623 A1).
In regards to claim 8, the combination of Shin and Park teaches the method of claim 1, but does not explicitly teach the method, wherein the diffusion model has a U-Net structure with multiple encoder blocks, multiple decoder blocks, and a middle block.
However, Yu teaches a method, wherein the diffusion model has a U-Net structure with multiple encoder blocks, multiple decoder blocks, and a middle block (e.g. [0019]: a framework comprising a fixed denoising diffusion model (DDM) modulated by a single trainable DDM, to generate images for a number of tasks; trainable DDM may start as a copy of the fixed DDM, but with trainable parameters; [0040],Fig.3 a fixed diffusion model comprising a U-Net encoder 310, U-Net middle 312, and U-Net decoder 314 is controlled by trainable diffusion model comprising a U-Net encoder 320 and U-Net middle 322; the U-Net encoders and decoders may include multiple layers and produce internal representations (e.g. feature maps) which are passed to subsequent layers).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings/combination of Shin and Park to use U-Net structure, in the same conventional manner as taught by Yu as both deal with using diffusion model to generate images. The motivation to combine the two would be that it would allow diffusion model to be structured as a U-Net, to generate progressively lower and higher resolution feature maps (see [0031]).
In regards to claim 9, the combination of Shin, Park and Yu teaches a method, wherein the encoder blocks and the middle block of the diffusion model are each duplicated into a trainable copy and a locked copy (e.g. Yu as above, [0019]: a framework comprising a fixed (locked) denoising diffusion model (DDM) modulated by a single trainable DDM, to generate images for a number of tasks; trainable DDM may start as a copy of the fixed DDM, but with trainable parameters; [0040],Fig.3 a fixed diffusion model comprising a U-Net encoder 310, U-Net middle 312, and U-Net decoder 314 is controlled by trainable diffusion model comprising a U-Net encoder 320 and U-Net middle 322), and wherein the trainable copy is in each case connected to the locked copy of a block by a convolutional layer (e.g. Yu, [0042],Fig.3: the output of the last layer of U-Net encoder 320 is input to U-Net middle 322; skip connections from the one or more layers of U-Net encoder 320 are modulated by modulated zero convolution 326; the output of the U-Net middle 322 is modulated by modulated zero convolution 324; Examiner’s note: Fig.3 shows connection of each encoder/middle of trainable copy to fixed/locked copy via convolutional layer).
In addition, the same rationale/motivation of claim 8 is used for claim 9.
Allowable Subject Matter
Claim(s) 4-6, 10-11 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. To note, claim 11 is included as it depends on claim 10.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JED-JUSTIN IMPERIAL whose telephone number is (571)270-5807. The examiner can normally be reached Monday to Friday, 9am - 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JED-JUSTIN IMPERIAL/Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights