Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-5, 8, and 17-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Mishra (US 20220067519 A1).
Regarding claim 1, Mishra teaches a method comprising:
receiving input comprising an image of a face and a target value of an attribute of the face to be modified (par. 0035: “A generator comprised of three hourglass networks, which are encoder-decoder pairs, starts with an input facial image and a target attributes vector. The input facial image is an RGB image that includes unpaired attributes. The target attributes vector includes expressions and lighting conditions.”);
encoding the image using an encoder of an image generation neural network to obtain an image embedding (par. 0036: “The facial image is processed through a first encoder-decoder pair and a second encoder-decoder pair. The encoder-decoders pairs, or hourglass networks, comprise a generator network. The first encoder-decoder pair decomposes a first image attribute subspace and the second encoder-decoder pair decomposes a second image attribute subspace. An attribute subspace can include a facial lighting subspace and a facial expression subspace. The first encoder-decoder pair outputs a first image transformation mask based on the first image attribute subspace and the second encoder-decoder pair outputs a second image transformation mask based on the second image attribute subspace.” NOTE: in order to decode the images, they must first be embedded.); and
generating a modified image of the face having the target value of the attribute based on the image embedding using a decoder of the image generation neural network (par. 0055: “A generator 410 includes an encoder-decoder pair, where the encoder-decoder pair comprises an hourglass network. The generator obtains an input image 412, a target expression, and target lighting. In embodiments, the image data comprises RGB image data. The generator processes the input image to synthesize a target image 414.”), wherein the image generation neural network is trained using a plurality of training images depicting a single subject generated by a training image generation neural network, and wherein the plurality of training images includes a first synthetic image and a second synthetic image, wherein the first synthetic image depicts a subject having a first value of a first training attribute and a second value of a second training attribute different from the first training attribute, and wherein the second synthetic image depicts the same subject having a third value of the first training attribute and a fourth value of the second training attribute (par. 0032: “Facial images are processed using the concatenated transformation masks to synthesize facial images suitable for training a machine learning network. The generator neural network or hourglass network within the GAN is trained to provide synthetic facial images, where the synthetic facial images include altered lighting, altered lighting direction, and altered facial expression relative to an original, or “starter,” image.” NOTE: The “first training attribute” listed here is lighting; the second is facial expression.).
Regarding claim 2, Mishra teaches the method of claim 1, further comprising:
generating an edit vector that indicates the target value of the attribute, wherein the modified image is generated based on the edit vector (par. 0035: “A generator comprised of three hourglass networks, which are encoder-decoder pairs, starts with an input facial image and a target attributes vector. The input facial image is an RGB image that includes unpaired attributes. The target attributes vector includes expressions and lighting conditions.”).
Regarding claim 3, Mishra teaches the method of claim 2, wherein: the edit vector indicates target values for a plurality of attributes of the face (par. 0035, as above in claim 2 rejection).
Regarding claim 4, Mishra teaches the method of claim 1, further comprising:
generating a noise vector, wherein the modified image is generated based on the noise vector (par. 0032: “The generating of synthetic images and the discriminating of the synthetic images can comprise a generative adversarial network (GAN) technique.” NOTE: it is well-known in the art that GANs make use of noise vectors for image generation.).
Regarding claim 5, Mishra teaches the method of claim 1, further comprising:
providing an intermediate image embedding from the encoder as input to an intermediate layer of the decoder (par. 0077: “The network includes a collection of intermediate layers 1120. The multilayered analysis engine can include a convolutional neural network. Thus, the intermediate layers can include a convolutional layer 1122. The convolutional layer 1122 can include multiple sublayers, including hidden layers, within it. The output of the convolutional layer 1122 feeds into a pooling layer 1124. The pooling layer 1124 performs a data reduction, which makes the overall computation more efficient. Thus, the pooling layer reduces the spatial size of the image representation to reduce the number of parameters and computation in the network.”).
Regarding claim 8, Mishra teaches the method of claim 1, further comprising:
caching the image embedding (par. 0099: “A programmable apparatus which executes any of the above-mentioned computer program products or computer-implemented methods may include one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors, programmable devices, programmable gate arrays, programmable array logic, memory devices, application specific integrated circuits, or the like. Each may be suitably employed or configured to process computer program instructions, execute computer logic, store computer data, and so on.”;
receiving a subsequent input including an additional target value for an additional attribute to be modified (par. 0075: “The combiner 208 may generate additional generated images for additional scales, including a third combined image based on the third scale representation of the input image 106 and the third scale representation of the generated image 139.”); and
generating a subsequent modified image based on the cached image embedding and the additional target value (par. 0075, as above).
Regarding claim 17, Mishra teaches an apparatus comprising:
at least one processor (par. 0046: “The flow 200, or portions thereof, can be implemented using one or more computers, processors, personal electronic devices, and so on.”);
at least one memory including instructions executable by the at least one processor (par. 0092: “The example video data collection machine 1420 shown comprises one or more processors 1424 coupled to a memory 1426 which can store and retrieve instructions, a display 1422, a camera 1428, and a microphone 1430.”); and
an image generation neural network configured to perform the method of claim 1 (as above in claim 1 rejection).
Regarding claim 18, Mishra teaches the apparatus of claim 17, wherein:
the image generation neural network includes an encoder and a decoder, and wherein an intermediate layer of the encoder provides input to an intermediate layer of the decoder (par. 0031: “In the disclosed materials, machine learning is based on encoder-decoder models which are used to form generators for synthesizing facial images. The encoder-decoder pairs, or hourglass networks, are used to process facial images in order to downsample into lower dimensional attribute space and then to upsample to the desired attributes.”).
Regarding claim 19, Mishra teaches the apparatus of claim 17, further comprising:
a training image generation neural network configured to generate the plurality of training images (par. 0011: “In embodiments, the training data, which can include synthetic facial images, is augmented with generated synthetic images. As the neural network is being trained, additional synthetic data such as synthetic facial images can be generated. The synthetic data can be created, filtered, supplemented, modified, and so on. The synthetic data that can be generated to augment the training dataset can be received from a generative adversarial network, or GAN. The results of training the neural network with the training dataset and the augmented (synthetic facial image) training dataset can be used to train further neural networks.”).
Regarding claim 20, Mishra teaches the apparatus of claim 19, wherein:
the training image generation neural network comprises a global discriminator and a region- specific discriminator (par. 0032: “The training of the generator neural network to generate synthetic facial images can be designed to avoid detection by the discriminator neural network. That is, a “good” synthetic facial image is indistinguishable by the discriminator from a real facial image.”).
Claim(s) 9-12 and 14-16 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Kalarot (US 20220122308 A1).
Regarding claim 9, Kalarot teaches a method comprising:
identifying a subject and a plurality of face attributes including a first attribute and a second attribute (par. 0072: “For each of the input image 106 and the generated image 139, the landmark detector 202 identifies landmarks within the image. For example, in the case of images of human faces, the landmark detector 202 identifier identifies 66 key points on the face. The landmarks may, for example, be identified using a trained neural network.”);
creating a training set by generating a plurality of training images depicting a single subject using a training image generation neural network (par. 0069: “The generator 132 is pre-trained to generate data that is similar to a training set. Depending on the type of image to be edited by the image editing system 102, the generator may be trained to generate an image of a human face, a landscape, a dog, a cat, a shoe, and so forth.”), wherein the plurality of training images includes a first synthetic image and a second synthetic image (par. 0083: “The training data 142 may include real images, synthetic images (e.g., as generated by the GAN), and/or latent space representations of the real and synthetic images.”), wherein the first synthetic image depicts the subject having a first value of the first attribute and a second value of the second attribute, and wherein the second synthetic image depicts the same subject having a third value of the first attribute and a fourth value of the second attribute, wherein the plurality of training images are generated by adjusting a latent vector in a plurality of directions corresponding to a plurality of attributes, respectively (par. 0036: “The encoder neural network may, for example, be a feedforward encoder network trained to compress an image into a vector representation.”); and
training, using the training set, an image generation neural network to modify face images based on a target value of the first attribute and the second attribute (par. 0042: “The image editing system provides the edited latent space representation as input to a trained generator neural network implemented by the computing system (e.g., the generator of the GAN, as described above) to generate a generated edited image.”).
Regarding claim 10, Kalarot teaches the method of claim 9, futher comprising:
generating a latent vector for the training image generation neural network (par. 0031: “When editing an image using a Generative Adversarial Network (GAN), an input image is provided as input to a machine learning model that generates a latent space representation, an N-dimensional vector representing a point in a latent space of a GAN.”);
generating a first modified latent vector based on the latent vector and the first value of the first attribute, wherein the first synthetic image is generated based on the first modified latent vector (par. 0031: “The latent space representation is edited (e.g., by vector addition or nonlinear modifications to the latent space representation). The edited latent space representation is then provided as input to the GAN, which generates an edited image reflecting the edits applied to the latent space representation.”); and
generating a second modified latent vector based on the latent vector and the third value of the first attribute, wherein the second synthetic image is generated based on the second modified latent vector (par. 0036: “In one example, the machine learning model is an encoder neural network. The encoder neural network may, for example, be a feedforward encoder network trained to compress an image into a vector representation. Alternatively, or additionally, generating the latent space representation may include applying an optimizer model to minimize a loss between an image generated using an initial latent space representation and the original output image. Once the latent space representation is generated, edits may be applied to the latent space representation based on user-configured parameters (e.g., make a dog depicted in an image a different breed, change the gender of a person depicted in an image, etc.).”).
Regarding claim 11, Kalarot teaches the method of claim 10, futher comprising:
generating a third modified latent vector based on the latent vector and a fifth value of the first attribute (par. 0036, as above in claim 10 rejection); and
generating a third synthetic image based on the third modified latent vector (par. 0036, as above in claim 10 rejection).
Regarding claim 12, Kalarot teaches the method of claim 10, further comprising:
identifying a modification basis vector corresponding to the first attribute (par. 0090: “The projection subsystem can modify the latent space representation by applying linear or nonlinear modifications to the latent space representation of the input image. As an example, based on the topology of the W.sub.p space, a vector is discovered which, when added to the latent space representation, modifies the latent space representation such that the ultimate output image has a targeted modification (e.g., the edit parameters are reflected in the output image).”); and
multiplying the modification basis vector by the first value of the first attribute to obtain a latent modification vector, wherein the first modified latent vector is based on the latent modification vector (par. 0090, as above).
Regarding claim 14, Kalarot teaches the method of claim 9, wherein:
the plurality of training images includes additional synthetic images generated based on a plurality of additional attributes (par. 0083, as above in claim 9 rejection).
Regarding claim 15, Kalarot teaches the method of claim 9, further comprising:
training the training image generation neural network based on a global discriminator and a region-specific discriminator (par. 0082: “The training subsystem 140 includes a discriminator 146. The discriminator 146 is part of the GAN 138 including the generator 132, and evaluates the output of the generator 132 to train the generator 132.”).
Regarding claim 16, Kalarot teaches the method of claim 9, further comprising:
generating a modified image based on the first value of the first attribute using the image generation neural network (par. 0036: “The image editing system can generate the latent space representation using one or more machine learning models trained to generate a latent space representation of an input image.”); and
comparing the modified image to the first synthetic image, wherein the image generation neural network is trained based on the comparison (par. 0082: “The discriminator 146 compares images produced by the generator 132 to real images, and the generator 132 works to “trick” the discriminator into determining that a generated image is actually a real image.”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 6-7 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mishra (US 20220067519 A1) as applied to claim 1 above, and further in view of Kalarot (US 20220122308 A1).
Regarding claim 6, Mishra teaches the method of claim 1, but fails to teach wherein the modified image preserves a texture of the image that is unrelated to the attribute.
Kalarot teaches wherein the modified image preserves a texture of the image that is unrelated to the attribute (par. 0039: “The first combined image is given by α.sub.1ow times the first scale representation of the input image plus (1−α.sub.low) times the first scale representation of the generated image. The α.sub.low value can be tuned to preserve a relatively high ratio of features from the generated image (e.g., to reflect applied global lighting edits). The image editing system generates a second combined image based on the second scale representation of the input image, the second scale representation of the generated image, and a second value different from the first value. The image editing system may implement a different value for high-level features to ensure preservation of fine textural detail such as skin pores and hair textures from the input image.”).
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to combine the image generation techniques of Mishra and Kalarot, as both are in the same field of endeavor of facial attribute editing. Preserving the identity of the individual being modified is obviously desirable to anyone familiar in the art, and preserving facial textures is crucial to this objective.
Regarding claim 7, Mishra teaches the method of claim 1, but fails to teach wherein the modified image preserves an identity of the face.
Kalarot teaches wherein the modified image preserves an identity of the face (par. 0039, as above in claim 6 rejection).
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to combine the image generation techniques of Mishra and Kalarot, as both are in the same field of endeavor of facial attribute editing. Preserving the identity of the individual being modified is obviously desirable to anyone familiar in the art.
Regarding claim 13, Kalarot teaches the method of claim 9, but fails to teach wherein the first value of the first attribute comprises a positive value and the third value of the first attribute comprises a negative value.
Mishra teaches wherein the first value of the first attribute comprises a positive value and the third value of the first attribute comprises a negative value (par. 0033: “Neural network training is based on techniques such as applying “known good” data to the neural network in order to adjust one or more weights or biases, to add or remove layers, etc., within the neural network.”; par. 0082: “When positive outcomes are encountered, the positive outcomes can be reinforced by changing weighting values within the model, algorithm, etc. Positive outcomes may result in increased weighting values. Negative outcomes can also be considered, where weighting values may be reduced or otherwise adjusted.”).
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to combine the image generation techniques of Mishra and Kalarot, as both are in the same field of endeavor of facial attribute editing. Positive and negative values for attributes in training data are well-known and commonly used in the art, and anyone familiar in the art would be able to use positive or negative values to encourage or discourage certain elements in an AI-generated image.
Response to Arguments
Applicant’s arguments, see Remarks, filed 05/20/2026, with respect to the rejection(s) of claim(s) 1-20 under Theobald (US 20210097730 A1) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Mishra (US 20220067519 A1) and Kalarot (US 20220122308 A1).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN A BARHAM whose telephone number is (571)272-4338. The examiner can normally be reached Mon-Fri, 8:30am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu, can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN ALLEN BARHAM/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights