DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 6-11, 14-16, and 20 are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Kansy (US 20230377214 A1).
Regarding to claim 1, Kansy discloses a processor-implemented method ([0023]: a CPU configured to operate in conjunction with a GPU; Fig. 1; [0027]: memory 116 includes various software programs that are executed by processor(s) 102 and application data associated with said software programs; Fig. 1; [0028]: training engine 122 trains a diffusion model for use in a sequence of diffusion iterations that iteratively remove noise from an input image until an image depicting a given identity without noise is produced; Fig. 1; [0029]: execution engine 124 uses a trained diffusion model to perform a sequence of reverse diffusion iterations that produces an output image depicting a face based on a given identity representation) comprising:
obtaining input data that corresponds to noise (Fig. 3; [0037]: training engine 122 generates and obtains a noisy image 252 based on a clean image 202 and a combined embedding 242; a pure noise image; [0043]: training engine 122 generates a noisy image 252 for use as input to a diffusion model 254;
PNG
media_image1.png
178
210
media_image1.png
Greyscale
);
iteratively updating the input data based on a diffusion model and a condition model (Fig. 3; [0055]: uses identity conditioning module 204, i.e. a condition model, to convert the input identity image 302 to an identity embedding 236A; use attribute conditioning module 224, i.e. a condition model, to convert the input identity image 302;
PNG
media_image2.png
156
510
media_image2.png
Greyscale
; Fig. 3; [0056]: execution engine 124 converts the predicted noise 256 to a current de-noised image 308; Fig. 3; [0057]: in an iteration, execution engine 124 provides the pure noise image 244 to the diffusion model 254 as input; a noise-to-image converter 306 converts the predicted noise 256 to a current de-noised image 308; Fig. 3; [0059], if the termination condition in 310 is not satisfied, then execution engine 124 performs another de-noising iteration 380, in second and subsequent iterations, by providing the current de-noised image 308 and the combined embedding 242 to the diffusion model 254 as input;
PNG
media_image3.png
516
770
media_image3.png
Greyscale
; iteratively update noise image based on diffusion model 254 and combined embedding data generated by identity conditioning module 204 and attribute conditioning module 224 as illustrated in Fig. 3); and
outputting image data that meets a sampling condition based on the iteratively updated
data (Fig. 3; [0059]: at condition block 310, the execution engine 124 determines whether a threshold termination condition, i.e. a sampling condition, is satisfied; the termination condition, i.e. a sampling condition, is satisfied if the number of iterations 380 executed is greater than or equal to a threshold number of inference de-noising operations; Fig. 3; [0062]: generate a de-noised output image 312, which is the current de-noised image 308 generated in the last de-noising iteration;
PNG
media_image4.png
238
338
media_image4.png
Greyscale
).
Regarding to claim 2. Kansy discloses the method of claim 1, wherein the iterative updating of the input data (same as rejected in claim 1) comprises:
obtaining first data corresponding to the diffusion model and second data corresponding to the condition model (Kansy; Fig. 3; [0059], if the termination condition in 310 is not satisfied, then execution engine 124 performs another de-noising iteration 380, in second and subsequent iterations, by providing the current de-noised image 308 and the combined embedding 242 to the diffusion model 254 as input; obtain the current de-noised image 308 and the combined embedding 242, i.e. first data and second data), wherein the first data and the second data are output in a previous iteration (Kansy; Fig. 3; [0059]: the current de-noised image 308 and the combined embedding 242 are an output of previous iteration as illustrated in Fig. 3;
PNG
media_image5.png
404
610
media_image5.png
Greyscale
); and
updating the obtained first data and the obtained second data in parallel based on the
diffusion model and the condition model (Kansy; [0023]: a graphics processing unit, i.e. GPU, executes instructions in parallel; Fig. 3; [0055]: execution engine 124 uses identity conditioning module 204 to convert the input identity image 302 to an identity embedding 236A; execution engine 124 also uses attribute conditioning module 224 to convert the input identity image 302 and one or more attribute values 222 and/or 223 to an attribute embedding 236B; execution engine 124 provides the identity embedding 236A and the attribute embedding 236B to an embedding adder 238, which combines the identity embedding 236A and attribute embedding 236B with an iteration identifier embedding to produce a combined embedding 242; Fig. 3; [0056]: the execution engine 124 uses a diffusion model 254 to generate predicted noise 256; execution engine 124 converts the predicted noise 256 to a current de-noised image 308 in parallel).
Regarding to claim 6, Kansy discloses the method of claim 1, wherein the condition model comprises a function for a task of the output image data (Kansy; [0053]: the attribute conditioning module 224 generates an attribute embedding 236B based on a given clean image 202 and/or one or more given attribute values 222; Fig. 3; [0055]: execution engine 124 uses identity conditioning module 204 to convert the input identity image 302 to an identity embedding 236A; execution engine 124 also uses attribute conditioning module 224 to convert the input identity image 302 and one or more attribute values 222 and/or 223 to an attribute embedding 236B; [0075]: execution engine 124 converts, using a neural network and based on the combined embedding, a first input image that includes first noise into a first predicted image depicting one or more facial features that include one or more facial identity features and one or more attribute-based facial features).
Regarding to claim 7, Kansy discloses the method of claim 1, wherein the iterative updating of the input data comprises reducing a step of updating the input data to 0 in a level unit of "1" or higher (Kansy; Fig. 3; [0045]: each iteration number t is between 1 and a total number of iterations T; when t is 1, reduce a step of updating to 0; Fig. 3; [0059]: the execution engine 124 determines at condition block 310 whether to execute another inference de-noising iteration 380 after generating the current de-noised image 308;
PNG
media_image6.png
196
716
media_image6.png
Greyscale
; if step 310 is No, then, do not implement second iteration).
Regarding to claim 8, Kansy discloses the method of claim 1, further comprising:
performing a second update, which iteratively updates the image data based on the
diffusion model (Kansy; Fig. 2A; [0032]: a diffusion model 254 iteratively updates and generates a predicted noise 256;
PNG
media_image7.png
162
714
media_image7.png
Greyscale
; Fig. 3; [0076]: a forward sequence 245 includes a first input image 644A followed by a second input image 644B); and
outputting translated data that meets a translation condition by iterating a first update
based on a result of the second update, the first update being the iterative updating of the input data (Kansy; [0050]: training engine 122 and/or execution engine 124 rotates, translates, crops, scales, and/or otherwise transform or process the image so that the eyes of the largest face lie on a predefined horizontal line and have a predefined ocular distance; Fig. 3; [0057]: in an iteration, execution engine 124 provides the pure noise image 244 to the diffusion model 254 as input; a noise-to-image converter 306 converts the predicted noise 256 to a current de-noised image 308; Fig. 3; [0059], if the termination condition in 310 is not satisfied, then execution engine 124 performs another de-noising iteration 380, in second and subsequent iterations, by providing the current de-noised image 308 and the combined embedding 242 to the diffusion model 254 as input; Fig. 6; [0078]: training engine 122 performs training de-noising iterations 280 to predict an amount of noise on which the diffusion model 254 is trained;
PNG
media_image8.png
568
754
media_image8.png
Greyscale
).
Regarding to claim 9, Kansy discloses a processor-implemented method ([0023]: a CPU configured to operate in conjunction with a GPU; Fig. 1; [0027]: memory 116 includes various software programs that are executed by processor(s) 102 and application data associated with said software programs; Fig. 1; [0028]: training engine 122 trains a diffusion model for use in a sequence of diffusion iterations that iteratively remove noise from an input image until an image depicting a given identity without noise is produced; Fig. 1; [0029]: execution engine 124 uses a trained diffusion model to perform a sequence of reverse diffusion iterations that produces an output image depicting a face based on a given identity representation) comprising:
obtaining image data (Fig. 3; [0037]: training engine 122 generates a noisy image 252 based on a clean image 202 and a combined embedding 242; a pure noise image; [0043]: training engine 122 generates a noisy image 252 for use as input to a diffusion model 254);
performing a first update, which iteratively updates the image data based on a diffusion
model and a condition model (Fig. 3; [0055]: uses identity conditioning module 204, i.e. a condition model, to convert the input identity image 302 to an identity embedding 236A; use attribute conditioning module 224, i.e. a condition model, to convert the input identity image 302;
PNG
media_image2.png
156
510
media_image2.png
Greyscale
; Fig. 3; [0056]: execution engine 124 converts the predicted noise 256 to a current de-noised image 308; Fig. 3; [0057]: in an iteration, execution engine 124 provides the pure noise image 244 to the diffusion model 254 as input; a noise-to-image converter 306 converts the predicted noise 256 to a current de-noised image 308; Fig. 3; [0059], if the termination condition in 310 is not satisfied, then execution engine 124 performs another de-noising iteration 380, in second and subsequent iterations, by providing the current de-noised image 308 and the combined embedding 242 to the diffusion model 254 as input;
PNG
media_image3.png
516
770
media_image3.png
Greyscale
; iteratively update noise image based on diffusion model 254 and combined embedding data generated by identity conditioning module 204 and attribute conditioning module 224 as illustrated in Fig. 3);
performing a second update, which iteratively updates the image data based on the
diffusion model (Kansy; Fig. 2A; [0032]: a diffusion model 254 iteratively updates and generates a predicted noise 256;
PNG
media_image7.png
162
714
media_image7.png
Greyscale
; Fig. 3; [0076]: a forward sequence 245 includes a first input image 644A followed by a second input image 644B); and
outputting translated data that meets a translation condition by iterating the first update based on a result of the second update (Kansy; [0050]: training engine 122 and/or execution engine 124 rotates, translates, crops, scales, and/or otherwise transform or process the image so that the eyes of the largest face lie on a predefined horizontal line and have a predefined ocular distance; Fig. 3; [0057]: in an iteration, execution engine 124 provides the pure noise image 244 to the diffusion model 254 as input; a noise-to-image converter 306 converts the predicted noise 256 to a current de-noised image 308; Fig. 3; [0059], if the termination condition in 310 is not satisfied, then execution engine 124 performs another de-noising iteration 380, in second and subsequent iterations, by providing the current de-noised image 308 and the combined embedding 242 to the diffusion model 254 as input; Fig. 6; [0078]: training engine 122 performs training de-noising iterations 280 to predict an amount of noise on which the diffusion model 254 is trained;
PNG
media_image8.png
568
754
media_image8.png
Greyscale
).
Regarding to claim 10, Kansy discloses the method of claim 9, wherein the condition model comprises a function that indicates a translation condition according to a semantic feature of the image data ([0033]: the facial features includes one or more facial identity features and/or one or more attribute-based facial features; each of the facial features correspond to a respective facial identity feature of the clean image 202, and are based at least on the identity embedding 236A; facial identity features strongly correlated with identity include facial hair, hair, hair color, and gender, i.e. semantic features; [0050]: training engine 122 and/or execution engine 124 rotates, translates, crops, scales, and/or otherwise transform or process the image so that the eyes of the largest face lie on a predefined horizontal line and have a predefined ocular distance; Fig. 6; [0079]: the last predicted image 606 includes meaningful content, i.e. semantic feature).
Regarding to claim 11, Kansy discloses the method of claim 9, wherein the performing of the first update (same as rejected in claim 9) comprises:
The rest claim limitations are similar to claim limitations recited in claim 2. Therefore, same rational used to reject claim 2 is also used to reject claim 11.
Regarding to claim 14, Kansy discloses a non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the operating method ([0023]: a CPU configured to operate in conjunction with a GPU; Fig. 1; [0026]: CD-ROM, DVD-ROM, Blu-Ray, HD-DVD, or other magnetic, optical, or solid state storage devices; Fig. 1; [0027]: memory 116 includes a random access memory (RAM) module, a flash memory unit, or any other type of memory unit or combination thereof; memory 116 includes various software programs that are executed by processor(s) 102 and application data associated with said software programs; Fig. 1; [0028]: training engine 122 trains a diffusion model for use in a sequence of diffusion iterations that iteratively remove noise from an input image until an image depicting a given identity without noise is produced; Fig. 1; [0029]: execution engine 124 uses a trained diffusion model to perform a sequence of reverse diffusion iterations that produces an output image depicting a face based on a given identity representation).
The rest claim limitations are similar to claim limitations recited in claim 9. Therefore, same rational used to reject claim 9 is also used to reject claim 14.
Regarding to claim 15, Kansy discloses an apparatus ([0023]: a CPU configured to operate in conjunction with a GPU; Fig. 1; [0026]: CD-ROM, DVD-ROM, Blu-Ray, HD-DVD, or other magnetic, optical, or solid state storage devices; Fig. 1; [0027]: memory 116 includes a random access memory (RAM) module, a flash memory unit, or any other type of memory unit or combination thereof; memory 116 includes various software programs that are executed by processor(s) 102 and application data associated with said software programs; Fig. 1; [0028]: training engine 122 trains a diffusion model for use in a sequence of diffusion iterations that iteratively remove noise from an input image until an image depicting a given identity without noise is produced; Fig. 1; [0029]: execution engine 124 uses a trained diffusion model to perform a sequence of reverse diffusion iterations that produces an output image depicting a face based on a given identity representation) comprising:
one or more processors configured to ([0023]: a CPU configured to operate in conjunction with a GPU):
the rest claim limitations are similar to claim limitations recited in claim 1. Therefore, same rational used to reject claim 1 is also used to reject claim 15.
Regarding to claim 16, Kansy discloses the apparatus of claim 15, wherein, for the iterative updating of the input data, the one or more processors are configured to (same as rejected in claim 15):
The rest claim limitations are similar to claim limitations recited in claim 2. Therefore, same rational used to reject claim 2 is also used to reject claim 16.
Regarding to claim 20, Kansy discloses an apparatus ([0023]: a CPU configured to operate in conjunction with a GPU; Fig. 1; [0026]: CD-ROM, DVD-ROM, Blu-Ray, HD-DVD, or other magnetic, optical, or solid state storage devices; Fig. 1; [0027]: memory 116 includes a random access memory (RAM) module, a flash memory unit, or any other type of memory unit or combination thereof; memory 116 includes various software programs that are executed by processor(s) 102 and application data associated with said software programs; Fig. 1; [0028]: training engine 122 trains a diffusion model for use in a sequence of diffusion iterations that iteratively remove noise from an input image until an image depicting a given identity without noise is produced; Fig. 1; [0029]: execution engine 124 uses a trained diffusion model to perform a sequence of reverse diffusion iterations that produces an output image depicting a face based on a given identity representation) comprising:
one or more processors configured to ([0023]: a CPU configured to operate in conjunction with a GPU):
the rest claim limitations are similar to claim limitations recited in claim 9. Therefore, same rational used to reject claim 9 is also used to reject claim 20.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4, 12, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Kansy (US 20230377214 A1) and in view of Pronovost (US 20240212360 A1).
Regarding to claim 4, Kansy discloses the method of claim 2, wherein the obtaining of the first data corresponding to the diffusion model and the second data corresponding to the condition model (same as rejected in claim 1) comprises
Kansy fails to explicitly disclose determining the second data based on the first data.
In same field of endeavor, Pronovost teaches determining the second data based on the first data ([0052]: remove noise from the input data using the condition data 308; [0125]: receiving, by the generative model and from a diffusion model, condition data representing image data or text data associated with the object or the environment; the generative model receives condition data generated from a diffusion model; the machine learned model 214 can receive the condition data 308 comprising text data describing an intersection type, a number of objects in the environment; [0126]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kansy to include determining the second data based on the first data as taught by Pronovost. The motivation for doing so would have been to learn from a training data set and to improve accuracy of an output; to remove noise from the input data using the condition data 308; to denoise the input data as taught by Pronovost in paragraphs [0026], [0052], and [0157].
Regarding to claim 12, Kansy discloses the method of claim 11,
The rest claim limitations are similar to claim limitations recited in claim 4. Therefore, same rational used to reject claim 4 is also used to reject claim 12.
Regarding to claim 18, Kansy discloses the apparatus of claim 16,
The rest claim limitations are similar to claim limitations recited in claim 4. Therefore, same rational used to reject claim 4 is also used to reject claim 18.
Claims 5, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kansy (US 20230377214 A1) and in view of Kumar (US 20250217929 A1).
Regarding to claim 5, Kansy discloses the method of claim 1, wherein the iterative updating of the input data comprises iteratively updating the input data based on a technique that processes the diffusion model and the condition model in parallel (Kansy; [0023]: a graphics processing unit, i.e. GPU, executes instructions in parallel; Fig. 3; [0055]: Execution engine 124 uses identity conditioning module 204 to convert the input identity image 302 to an identity embedding 236A; execution engine 124 also uses attribute conditioning module 224 to convert the input identity image 302 and one or more attribute values 222 and/or 223 to an attribute embedding 236B; execution engine 124 provides the identity embedding 236A and the attribute embedding 236B to an embedding adder 238, which combines the identity embedding 236A and attribute embedding 236B with an iteration identifier embedding to produce a combined embedding 242; Fig. 3; [0056]: the execution engine 124 uses a diffusion model 254 to generate predicted noise 256; execution engine 124 converts the predicted noise 256 to a current de-noised image 308 in parallel).
Kansy fails to explicitly disclose a technique is alternating direction method of multipliers (ADMM).
In same field of endeavor, i.e., Two-dimensional image generation, Kumar teaches a technique is alternating direction method of multipliers (ADMM) ([0017]: reconstruct a High Resolution (HR) image z of target modality from a Low Resolution (LR) image x of target modality with the guidance of HR image y from another modality; [0028]: train the plurality of sparse coefficients; update the first set of sparse coefficients using Alternating Direction Method of Multipliers (ADMM) technique; [0029]: the ADMM iterations).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kansy to include a technique is alternating direction method of multipliers (ADMM) as taught by Kumar. The motivation for doing would have been to train the sparse coefficients by keeping dictionaries fixed and then to train the dictionaries by keeping sparse coefficients fixed; to update the first set of sparse coefficients using Alternating Direction Method of Multipliers (ADMM) technique as taught by Kumar in paragraphs [0017], [0019] and [0028].
Regarding to claim 13, Kansy discloses the method of claim 11, wherein the performing of the first update (same as rejected in claim 9) comprises
The rest claim limitations are similar to claim limitations recited in claim 5. Therefore, same rational used to reject claim 5 is also used to reject claim 13.
Regarding to claim 19, Kansy discloses the apparatus of claim 15, wherein, for the iterative updating of the input data, the one or more processors are configured to (same as rejected in claim 15).
The rest claim limitations are similar to claim limitations recited in claim 5. Therefore, same rational used to reject claim 5 is also used to reject claim 19.
Allowable Subject Matter
Claims 3 and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hai Tao Sun whose telephone number is (571)272-5630. The examiner can normally be reached 9:00AM-6:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 5712727642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAI TAO SUN/Primary Examiner, Art Unit 2616
Prosecution Insights