DETAILED ACTION
Notice of Pre-AIA or AIA Status.
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. Claims 1-16, 18-19, and 21-22 filed and preliminary amended on 07/30/2024 are pending and being examined. Claims 1, 18, and 19 are independent form.
Priority
3. Acknowledgment is made of applicant's claim for PCT priority under 35 U.S.C. 371, where the benefit of foreign priority was further claimed.
35 USC § 101—Positive Statement
4. Independent claim 19 is drawn to a “computer-readable storage medium (CRM)” storing a computer program. In light of the specification, see paragraph [0028], wherein “a computer-readable storage medium is clearly defined excluding “a computer-readable signal medium”. Thus, independent claim 19 and its dependent claims are patent eligible.
Claim Rejections - 35 USC § 103
5. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
6. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
7. Claims 1-16, 18-19, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Xiao et al (CN111489287, hereinafter “Xiao”) in view of Zhang et al CN111815504, hereinafter “Zhang”). A machine translated English version (CN111489287-Eng) of document CN111489287 and English version (CN111815504-Eng) of document CN111815504 are provided by the examiner with this office action.
Regarding claim 1, Xiao discloses an image processing method (the face image conversion method for generating a beautiful face image via the trained face image conversion model (such as a generative adversarial network (GAN)); see Abstract, fig.2 and fig.3) comprising:
acquiring a target facial image to be processed of a target object (see 202 of fig.2 and pg.6, lines 11-12: “obtaining the first image; The first image comprises face information of the object to be converted”; see the first face images in the first row of fig.3); and
inputting the target facial image to be processed to a pre-trained target facial processing
model to obtain a facial processing target image with a target facial effect, wherein the target facial processing model is trained (see 206 of fig.2 and pg.7, lines 39-46: “based on the second image output by the trained image conversion model, obtaining the target image corresponding to the first image”; see the second face images in the second row of fig.3)
As explained above, the mere distance is, Xiao does not explicitly disclose: the face image conversion model in the method in Xiao (which is called the target facial processing model in the claim) is trained by a plurality of reference facial images to be processed to construct a preliminary to-be-processed sample set and a plurality of facial processing reference images with the target facial effect to construct a preliminary processing effect set. However, in the same field of endeavor, Zhang teaches:
acquiring a plurality of reference facial images to be processed to construct a preliminary to-be-processed sample set, and acquiring a plurality of facial processing reference images with the target facial effect to construct a preliminary processing effect set; determining a sample facial image to be processed and a facial processing sample image corresponding to the sample facial image to be processed according to the reference facial images to be processed in the preliminary to-be-processed sample set and the facial processing reference images in the preliminary processing effect set; and
training an initial facial processing model according to the sample facial image to be
processed and the facial processing sample image corresponding to the sample facial image to be processed to obtain the target facial processing model (see fig.5 and pg.12, line 16—pg.13, line 37. Specially, see pg.12, lines 16-23; “provides a human face image beautifying method based on deep learning, FIG. 5 is a human face image beautifying method according to an alternative embodiment of the present application of the schematic diagram, as shown in FIG. 5, [...]”; see pg.13, lines 24-30: “in order to achieve the face global beautifying effect, when generating the training data set, the high quality face image with high definition and high score is selected as the output image,”). In other words, to generate a beautiful face image by a human face image beautifying method, Zhang teaches training “a human face image beautifying method” based on a deep learning (i.e., “StyleGAN”) network which is trained by “the training data set” including the input face images and the corresponding output face images with the high quality scores; wherein a face image quality score indicate the beauty effect of a face image (see pg.8, lines 24-34). As such, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was made to incorporate the teachings of Zhang into the teachings of Xiao and train a face image conversion model based on a training data set including input face images and the corresponding output face images with the processing effect (i.e., with the high quality scores). Suggestion or motivation for doing so would have been to generate a beautiful face image by a trained deep neural network as taught by Zhang, cf., fig.2, fig.5, and Abstract. Therefore, the combination of Xiao and Zhang suggests or teaches all the limitations recited in claim 1, and the claim is unpatentable over Xiao in view of Zhang.
Regarding claim 2, 21, the combination of Xiao and Zhang discloses, wherein determining the sample facial image to be processed and the facial processing sample image corresponding to the sample facial image to be processed according to the reference facial images to be processed in the preliminary to-be-processed sample set and the facial processing reference images in the preliminary processing effect set comprises:
training a pre-built first initial image generation model according to the reference facial
images to be processed in the preliminary to-be-processed sample set to obtain an image to be processed generation model; training a pre-built second initial image generation model according to the facial processing reference images in the preliminary processing effect set to obtain a sample effect image generation model; and
generating the sample facial image to be processed and the facial processing sample
image corresponding to the sample facial image to be processed according to the image to be processed generation model and the sample effect image generation model, wherein the first initial image generation model and the second initial image generation model are style-based generative adversarial networks (Xiao: the trained face image conversion model includes the first generative adversarial network including the generator Gs-t which generates the face image t based on its input face image s and discriminator Dt which generates the new face image s’ with the new style based on its input face image t; see a generative adversarial network shown by fig.4; see fig.4 and pg.9, lines 18-43. Zhang: wherein “the network of StyleGAN” is trained by “the training data set” including the input face images and the corresponding output face images with the high quality/beautifying scores to generate a beautifying face image; see fig.5 and pg.12, line 16—pg.13, line 30).
Regarding claim 3, 22, the combination of Xiao and Zhang discloses, wherein generating the sample facial image to be processed and the facial processing sample image corresponding to the sample facial image to be processed according to the image to be processed generation model and the sample effect image generation model comprises:
determining a target image conversion model according to the reference facial images to be processed in the preliminary to-be-processed sample set and the image to be processed generation model, wherein the target image conversion model is used for converting an image input to the target image conversion model into a target image vector; and generating the sample facial image to be processed according to the image to be processed generation model, and generating the facial processing sample image corresponding to the sample facial image to be processed according to the sample facial image to be processed, the target image conversion model, and the sample effect image generation model (Xiao: the trained face image conversion model includes the first generative adversarial network including the generator Gs-t which generates the face image t based on its input face image s and discriminator Dt which generates the new face image s’ with the new style based on its input face image t, and therefore the face image conversion model can generates the new reference face image s’ with the new style based on its input face image t; see a generative adversarial network shown by fig.4; see fig.4 and pg.9, lines 18-43. Zhang: wherein “the network of StyleGAN” is trained by “the training data set” including the input face images and the corresponding output face images with the high quality scores to generate a beautifying face image; see fig.5 and pg.12, line 16—pg.13, line 30).
Regarding claim 4, the combination of Xiao and Zhang discloses the method according to claim 3, wherein determining the target image conversion model according to the reference facial images to be processed in the preliminary to-be-processed sample set and the image to be processed generation model comprises:
inputting the reference facial images to be processed in the preliminary to-be-processed
sample set to an initial image conversion model to obtain model conversion vectors;
inputting the model conversion vectors to the image to be processed generation model to obtain model-generated images corresponding to the model conversion vectors; and
performing parameter adjustment on the initial image conversion model according to a
loss between the model-generated images and the reference facial images to be processed which are input to the initial image conversion model and correspond to the model-generated images, so as to obtain the target image conversion model (Xiao, minimizing “loss function” for training the face image conversion model, see fig.13 and pg.19, line 9—pg.21, line 2).
Regarding claim 5, the combination of Xiao and Zhang discloses the method according to claim 3, wherein generating the sample facial image to be processed according to the image to be processed generation model to be processed, and generating the facial processing sample image corresponding to the sample facial image to be processed according to the sample facial image to be processed, the target image conversion model, and the sample effect image generation model comprises:
inputting the reference facial images to be processed to the target image conversion
model to obtain target image vectors corresponding to the reference facial images to be
processed; inputting the target image vectors to the image to be processed generation model to obtain the sample facial image to be processed; and inputting the target image vectors to the sample effect image generation model to obtain the facial processing sample image corresponding to the sample facial image to be processed (Xiao: the trained face image conversion model includes the first generative adversarial network including the generator Gs-t which generates the face image t based on its input face image s and discriminator Dt which generates the new reference face image s’ with the new style based on its input face image t, and therefore the face image conversion model is gradually and iteratively trained by both an original input face image s with its original style and its generated face image s’ with the new style to achieve “the circulation consistency” of the model; see a generative adversarial network shown by fig.4; see fig.4 and pg.9, lines 18-43.).
Regarding claims 6-9, wherein the claimed method further recites “color correction processing”, “facial deformation correction processing”, or “facial makeup restoration processing”. However, each of the techniques is well known and widely used in the field of processing face images. It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was made to know that color correction processing, facial deformation correction processing, and/or facial makeup restoration processing would be able to improve the performance of detecting facial regions in a facial image. Suggestion or motivation for doing so would have been to improve the performance of detecting facial regions in a facial image and thereby improve the performance of generating a beautiful face image via the trained face image conversion model. Therefore, each of the claims is unpatentable and an obvious variation of the method in the combination of Xiao and Zhang.
Regarding claim 10, the combination of Xiao and Zhang discloses the method according to claim 1, wherein the initial facial processing model comprises a processing effect generation model (Xiao, see the generator Gs-t in fig.4) and a processing effect discrimination model (Xiao, see the discriminator Dt in fig.4); and training the initial facial processing model according to the sample facial image to be processed and the facial processing sample image corresponding to the sample facial image to be processed to obtain the target facial processing model comprises:
inputting the sample facial image to be processed to the processing effect generation
model to obtain a processing effect generation image; adjusting the processing effect generation model according to the sample facial image to be processed, the processing effect generation image, and the facial processing sample image corresponding to the sample facial image to be processed; and determining, according to a discrimination result obtained by the processing effect discrimination model for the processing effect generation image, whether to stop adjusting the processing effect generation model, and using the processing effect generation model obtained at the end of the adjustment as the target facial processing model (Xiao, see pg.12, lines 38-42: “the encoder in the first discriminator in the application of the invention is as follows: in the training process of the image conversion model, extracting the facial feature of the object to be converted in the second image and based on the facial feature, assisting the first generator Gs-t to generate a rough second image, gradually transitioning to the second image generating excellent quality.”).
Regarding claim 11, the combination of Xiao and Zhang discloses the method according to claim 10, wherein adjusting the processing effect generation model according to the sample facial image to be processed, the processing effect generation image, and the facial processing sample image corresponding to the sample facial image to be processed comprises:
determining a first facial feature loss between the sample facial image to be processed
and the processing effect generation image, and determining a second facial feature loss between the processing effect generation image and the facial processing sample image corresponding to the sample facial image to be processed; and adjusting the processing effect generation model according to the first facial feature loss
and the second facial feature loss (Xiao, minimizing “loss function” for training the face image conversion model, see fig.13 and pg.19, line 9—pg.21, line 2).
Regarding claims 12-16, wherein the claimed method further recites capturing the target facial image, adjusting an image processing degree in the target display area, determining a target weight corresponding to the processing degree adjustment operation, determining the facial processing target image corresponding to the processing degree adjustment operation, and displaying the facial processing target image. However, each of the techniques is well known and widely used techniques in the field of processing face images. It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was made to know that these techniques would be able to improve the performance of displaying a facial processing target image. Suggestion or motivation for doing so would have been to improve the performance of displaying the facial processing target image. Therefore, each of the claims is unpatentable and an obvious variation of the method in the combination of Xiao and Zhang.
Regarding claim 18, 19, each of which is an inherent variation of claim 1, thus it is interpreted and rejected for the reasons set forth in the rejection of claim 1.
Conclusion
8. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
SHCHERBININ et al, US 20200387750.
Kollias et al: “Deep Neural Network Augmentation: Generating Faces for Affect Analysis”, 2020.
9. Any inquiry concerning this communication or earlier communications from the examiner should be directed to RUIPING LI whose telephone number is (571)270-3376. The examiner can normally be reached 8:30am--5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, HENOK SHIFERAW can be reached on (571)272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit https://patentcenter.uspto.gov; https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center, and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RUIPING LI/Primary Examiner, Ph.D., Art Unit 2676
Prosecution Insights