DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 08/27/2024 and 08/27/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The drawings filed on 08/27/2024 are accepted by the examiner.
Specification
The disclosure filed on 08/27/2024 is accepted by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-10 and 12-20 are rejected under 35 U.S.C. 103 as being unpatentable over Antipov (“Face aging with conditional generative adversarial networks” May 30, 2017) in view of Korshunova ("Fast Face-Swap Using Convolutional Neural Networks," 29 November 2016).
Regarding claims 1, 17 and 19, Antipov discloses one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising (a GAN uses processors and memory) receiving an indication of an instruction to modify an initial image comprising a face, the instruction indicating to change the face from having a first image to having a second image (section 2 and 3 Figures 1 and 2 for different ages the face portion changes and depending in the changes in the face, the hair changes); and modifying a face portion and a non-face portion of the initial image, the modified face portion having the second image, the modified non face portion having effects based on the second image (section 2 and 3 Figures 1 and 2 for different ages the face portion changes and depending in the changes in the face, the hair changes).
PNG
media_image1.png
236
403
media_image1.png
Greyscale
Antipov doesn’t specifically disclose changes from a first appearance to a second appearance. Korshunova discloses changes from a first appearance to a second appearance (section 1 and 2 figure 5 “Face replacement or face swapping is relevant in many scenarios including the provision of privacy, appearance transfiguration in portraits, video compositing, and other creative applications” … “Note how our method alters the appearance of the nose, eyes, eyebrows, lips and facial wrinkles.”). Antipov and Korshunova are analogous art because they are from the same field of communications. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate in the technique disclosed by Antipov the convolutional neural network disclosed by Korshunova. The suggestion/motivation for doing so would have been face swapping in images (Korshunova abstract). See also KSR Int'l Co. v. Teleflex Inc. Case cited as 550 US (2007). In the KSR case, the Court stated that in certain circumstances what is obvious to try is also obvious, such as where "there is a design need or market pressure to solve a problem, and there are a finite number of identified, predictable solutions, a person of ordinary skill has good reason to pursue the known options within his or her technical grasp. If this leads to the anticipated success, it is likely the product not of innovation but of ordinary skill and common sense." Regarding hindsight, the Court found that "[r]igid preventive rules that deny fact finders recourse to common sense . . . are neither necessary under our case law nor consistent with it." The Court stated that "familiar items may have obvious uses beyond their primary purposes," analogizing an obvious invention to the fitting together of pieces to a puzzle. The Court in this regard further stated that the person of ordinary skill is also a person of ordinary creativity, and not "an automaton."
Regarding claim 2, Antipov and Korshunova disclose claim 1, Antipov also discloses a convolutional neural network (section 2.1 “Conditional GAN (cGAN) [13, 14] extends the GAN model allowing the generation of images with certain attributes (“conditions”).” … “The Age-cGAN model proposed in this work uses the same design for the generator G and the discriminator D as in [15]. Following [12], we inject the conditional information at the input of G and at the first convolutional layer of D. Age-cGAN is optimized using the ADAM algorithm [16] during 100 epochs. In order to encode person’s age, we have defined six age categories: 0-18, 19-29, 30-39, 40-49, 50-59 and 60+ years old”). Korshunova also discloses a convolutional neural network (title, abstract “To perform this mapping, we use convolutional neural networks trained to capture the appearance of the target identity from an unstructured collection of his/her photographs.”).
Regarding claim 3, Antipov and Korshunova disclose claim 2, Antipov also discloses the convolutional neural network is trained on a set of images including images of faces exhibiting the first appearance, and images of user faces exhibiting the second appearance (section 2.1 “They have been selected so that the training dataset (cf. Subsection 3.1) contains at least 5; 000 examples in each age category. Thus, the conditions of Age-cGAN are six-dimensional one-hot vectors.”). Korshunova also discloses convolutional neural network is trained on a set of images including images of faces exhibiting the first appearance, and images of user faces exhibiting the second appearance (title, abstract “To perform this mapping, we use convolutional neural networks trained to capture the appearance of the target identity from an unstructured collection of his/her photographs.”).
Regarding claim 4, Antipov and Korshunova disclose claim 2, Antipov also discloses the convolutional neural network is trained on a set of images including images of faces exhibiting the first appearance, and images of user faces exhibiting the second appearance (section 2.1 “They have been selected so that the training dataset (cf. Subsection 3.1) contains at least 5; 000 examples in each age category. Thus, the conditions of Age-cGAN are six-dimensional one-hot vectors.”). Korshunova also discloses a plurality of downsampling convolution layers that input into a plurality of residual block layers (figure 3 downsampling block 32 with different resolutions and residual blocks 64, 96 and upsampling)
Regarding claim 5, Antipov and Korshunova disclose claim 1, Antipov also discloses modifying, using a first convolutional neural network, the face portion of the initial image to have the second appearance; and modifying, using a second convolutional neural network, the non-face portion of the initial image (section 1-3 figure 3 “We design Age-cGAN (Age Conditional Generative Adversarial Network), the first GAN to generate high quality synthetic images within required age categories.” … ““Pixelwise” optimization better reflects superficial face details: such as the hair color in the first line and the beard in the last line.”)
Regarding claim 6, Antipov and Korshunova disclose claim 1, Antipov also discloses detecting the face in the initial image (figures 1-3 section 2.1 “They have been selected so that the training dataset (cf. Subsection 3.1) contains at least 5; 000 examples in each age category. Thus, the conditions of Age-cGAN are six-dimensional one-hot vectors.”). Korshunova also discloses detecting the face in the initial image (abstract figure 1-10)
Regarding claim 7, Antipov and Korshunova disclose claim 1, Antipov also discloses an image capturing device coupled to the one or more memories, and wherein the operations further comprise: capturing the initial image (figures 1-3 section 2.1 “They have been selected so that the training dataset (cf. Subsection 3.1) contains at least 5; 000 examples in each age category. Thus, the conditions of Age-cGAN are six-dimensional one-hot vectors.”). Korshunova also discloses an image capturing device coupled to the one or more memories, and wherein the operations further comprise: capturing the initial image (abstract figure 1-10)
Regarding claim 8, Antipov and Korshunova disclose claim 1, Korshunova also discloses separating the initial image into a cropped portion of the initial image and a non-cropped portion of the initial image, the cropped portion of the initial image comprising the face (figure 2)
Regarding claim 9, Antipov and Korshunova disclose claim 8, Antipov also discloses modifying the cropped portion using a convolutional neural network to have the second appearance and modifying the non-cropped portion by applying adjustments to the non-cropped portion, wherein the adjustments are effects selected based on the second appearance (section 1-3 figure 3 “We design Age-cGAN (Age Conditional Generative Adversarial Network), the first GAN to generate high quality synthetic images within required age categories.” … ““Pixelwise” optimization better reflects superficial face details: such as the hair color in the first line and the beard in the last line.”)
Regarding claim 10, Antipov and Korshunova disclose claim 9, Korshunova also discloses blending the modified cropped portion with the modified non-cropped portion (figure 2)
Regarding claim 12, Antipov and Korshunova disclose claim 1, Antipov also discloses the second appearance is an aged face and the effects selected comprise an effect to color hair in the non-face portion to appear gray (figure 2 and 3)
Regarding claim 13, Antipov and Korshunova disclose claim 1, Antipov also discloses the second appearance is a change to an expression of the face portion and the effects is a change to hair of the non-face portion (figure 2 and 3)
Regarding claim 14, Antipov and Korshunova disclose claim 1, Antipov also discloses a change to the face portion and the effects is a change in lighting to the non-face portion (figure 2 and 3)
Regarding claim 15, Antipov and Korshunova disclose claim 1, Antipov also discloses adjusting color values of the modified non face portion spatially close to the face portion (figure 2 and 3)
Regarding claim 16, Antipov and Korshunova disclose claim 1, Antipov also discloses the second appearance is a change to the face portion and the effects is an annotation (figure 2 and 3)
Regarding claims 18 and 20, Antipov and Korshunova disclose claims 17 and 19, Antipov also discloses the second appearance is a change to an expression of the face portion and the effects is a change to hair of the non-face portion (section 2 and 3 Figures 1 and 2 for different ages the face portion changes and depending in the changes in the face, the hair changes).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Antipov and Korshunova as applied to claim 10 above, and further in view of Kwan (“Image Pyramids and Blending” CMU 2005).
Regarding claim 11, Antipov and Korshunova discloses claim 10, Antipov and Korshunova don’t specifically disclose using Laplacian blending. Kwan discloses using Laplacian blending (page 29 Laplacian Pyramid blending). Antipov, Korshunova and Kwan are analogous art because they are from the same field of communications. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate in the technique disclosed by Antipov and Korshunova the Laplacian blending network disclosed by Kwan. The suggestion/motivation for doing so would have been to use one of a limited number of possibilities blending (see KSR above).
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b). 9
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. US 11683362 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because claims 1-20 of the present application are anticipated by claims 1-20 of U.S. Patent No. US 11683362 B2.
Present Application
US 12107914 B2
1. A system comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving an indication of an instruction to modify an initial image comprising a
face, the instruction indicating to change the face from having a first
appearance to having a second appearance; and
modifying a face portion and a non-face portion of the initial image, the modified face portion having the second appearance, the modified non face portion having effects based on the second appearance
1. A system comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving an indication of an instruction to modify an initial image comprising a user face, the instruction indicating to change the user face from having a first appearance to having a second appearance;
modifying a user face portion of the initial image using a convolutional neural network, the modified user face portion having the second appearance;
and modifying a non-user face portion of the initial image by applying adjustments to the non-user face portion, wherein the adjustments are effects selected based on the convolutional neural network used to modified the user face portion
2. The system of claim 1, wherein the modifying is performed using a convolutional neural network
1. …..modifying a user face portion of the initial image using a convolutional neural network, …and modifying a non-user face portion of the initial image by applying adjustments to the non-user face portion, wherein the adjustments are effects selected based on the convolutional neural network used to modified the user face portion
3. The system of claim 2, wherein the convolutional neural network is trained on a set of images including images of faces exhibiting the first appearance, and images of user faces exhibiting the second appearance
10. The system of claim 1, wherein the convolutional neural network is trained on a set of images including images of user faces exhibiting the first appearance, and images of user faces exhibiting the second appearance
4. The system of claim 2, wherein the convolutional neural network comprises a plurality of downsampling convolution layers that input into a plurality of residual block layers
12. The system of claim 1, wherein the convolutional neural network comprises a plurality of downsampling convolution layers that input into a plurality of residual block layers
5. The system of claim 1, wherein the modifying further comprises: modifying, using a first convolutional neural network, the face portion of the initial image to have the second appearance; and
modifying, using a second convolutional neural network, the non-face portion of the initial image
6. The system of claim 4, wherein the modifying the user face portion further comprises: modifying the cropped portion using the convolutional neural network, the modified cropped portion displaying the user face having the second appearance
modifying the non-cropped portion by applying adjustments to the non-cropped portion, wherein the adjustments are effects selected based on the convolutional neural network used to modified to the cropped portion.
6. The system of claim 1, wherein the operations further comprise: detecting the face in the initial image
2. The system of claim 1, wherein the operations further comprise: detecting the user face in the initial image.
7. The system of claim 1 further comprising: an image capturing device coupled to the one or more memories, and wherein the operations further comprise: capturing the initial image
3. The system of claim 1 further comprising: an image capturing device coupled to the one or more memories, and wherein the operations further comprise: capturing the initial image
8. The system of claim 1, wherein the operations further comprise: separating the initial image into a cropped portion of the initial image and a non-cropped portion of the initial image, the cropped portion of the initial image comprising the face
4. The system of claim 1, wherein the operations further comprise: separating the initial image into a cropped portion of the initial image and a non-cropped portion of the initial image, the cropped portion of the initial image comprising the user face
9. The system of claim 8, wherein the modifying the face portion further comprises: modifying the cropped portion using a convolutional neural network to have the second appearance; and
modifying the non-cropped portion by applying adjustments to the non-cropped portion, wherein the adjustments are effects selected based on the second appearance
6. The system of claim 4, wherein the modifying the user face portion further comprises: modifying the cropped portion using the convolutional neural network, the modified cropped portion displaying the user face having the second appearance
modifying the non-cropped portion by applying adjustments to the non-cropped portion, wherein the adjustments are effects selected based on the convolutional neural network used to modified to the cropped portion.
10. The system of claim 9, wherein the operations further comprise: generating a result image by blending the modified cropped portion with the modified non-cropped portion
10. The system of claim 1, wherein the convolutional neural network is trained on a set of images including images of user faces exhibiting the first appearance, and images of user faces exhibiting the second appearance
11. The system of claim 10 wherein the result image is generated by blending the modified cropped portion with the modified non-cropped portion and the initial image using Laplacian blending
9. The system of claim 8 wherein the result image is generated by blending the modified cropped portion with the modified non-cropped portion and the initial image using Laplacian blending.
12. The system of claim 1, wherein the second appearance is an aged face and the effects selected comprise an effect to color hair in the non-face portion to appear gray
15. The system of claim 1, wherein the second appearance is an aged face and the effects selected comprise an effect to color hair in the non-user face portion to appear gray
13. The system of claim 1, wherein the second appearance is a change to an expression of the face portion and the effects is a change to hair of the non-face portion
15. The system of claim 1, wherein the second appearance is an aged face and the effects selected comprise an effect to color hair in the non user face portion to appear gray
14. The system of claim 1, wherein the second appearance is a change to the face portion and the effects is a change in lighting to the non-face portion
14. The system of claim 1, wherein the operations further comprise:
selecting the convolutional neural network from a plurality of convolution neural networks, wherein the convolution neural network was trained to change user faces to the second appearance
15. The system of claim 1, wherein the operations further comprise: adjusting color values of the modified non face portion spatially close to the face portion
17. The system of claim 1, further comprising: adjusting
color values of the modified non user face portion spatially close to the face portion.
16. The system of claim 1, wherein the second appearance is a change to the face portion and the effects is an annotation
17. The system of claim 1, further comprising: adjusting color values of the modified non user face portion spatially close to the face portion.
17. A method comprising: receiving an indication of an instruction to modify an initial image comprising a face, the instruction indicating to change the
face from having a first appearance to having a second appearance; and
modifying a face portion and a non-face portion of the initial image, the modified face portion having the second appearance, the modified non face portion having effects based on the second appearance
19. A method comprising: receiving an indication of an instruction to modify an initial image comprising a user face, the instruction indicating to change the user face from having a first appearance to having a second appearance;
modifying a user face portion of the initial image using a convolutional neural network, the modified user face portion having the second appearance; and modifying a non-user face portion of the initial image by applying adjustments to the non-user face portion, wherein the adjustments are effects selected based on the convolutional neural network used to modified the user face portion
18. The method of claim 17, wherein the second appearance is a change to an expression of the face portion and the effects is a change to hair of the non-face portion
15. The system of claim 1, wherein the second appearance is an aged face and the effects selected comprise an effect to color hair in the non-user face portion to appear gray.
19. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: receiving an indication of an instruction to modify an initial image comprising a face, the instruction indicating to change the face from having a first appearance to having a second appearance; and
modifying a face portion and a non-face portion of the initial image, the modified face portion having the second appearance, the modified non face portion having effects based on the second appearance
20. A computer-readable storage medium embodying instructions that, when executed by a device, cause the device to perform operations comprising: receiving an indication of an instruction to modify an initial image comprising a user face, the instruction indicating to change the user face from having a first appearance to having a second appearance;
modifying a user face portion of the initial image using a convolutional neural network, the modified user face portion having the second appearance; and modifying a non-user face portion of the initial image by applying adjustments to the non-user face portion, wherein the adjustments are effects selected based on the convolutional neural network used to modified the user face portion
20. The non-transitory computer-readable storage medium of claim 19, wherein the second appearance is a change to an expression of the face portion and the effects is a change to hair of the non-face portion
15. The system of claim 1, wherein the second appearance is an aged face and the effects selected comprise an effect to color hair in the non-user face portion to appear gray
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. US 11683362 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because claims 1-20 of the present application are anticipated by claims 1-20 of U.S. Patent No. US 11683362 B2. See Non-Final Rejection with Receipt Date 02/15/2024 of application 18195813.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. US 10891723 B1. Although the claims at issue are not identical, they are not patentably distinct from each other because claims 1-20 of the present application are anticipated by claims 1-20 of U.S. Patent No. US 10891723 B1. See Non-Final Rejection with Receipt Date 02/15/2024 of application 18195813.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Bukar,” Facial Age Synthesis Using Sparse Partial Least Squares (The Case of Ben Needham)” J Forensic Sci, September 2017, Vol. 62, No. 5
Fu, “Age Synthesis and Estimation via Faces: A Survey” IEEE transactions on pattern analysis and machine intelligence, vol. 32, No. 11, November 2010
Larsen, “Autoencoding beyond pixels using a learned similarity metric,” Proceedings of International Conference on Machine Learning, New York, USA, 2016.
Perarnau, “Invertible conditional GANs for image editing,” in Proceedings of advances in Neural Information Processing Systems Workshops, Barcelona, Spain, 2016.
Bookbinder (US 4276570 A) discloses method and apparatus for producing an image of a person's face at a different age.
Chu (US 9384384 B1) discloses adjusting faces displayed in images.
Park (US 10552977 B1) discloses fast face-morphing using neural networks.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JUAN A TORRES whose telephone number is (571) 272-3119. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kenneth N Vanderpuye can be reached on (571) 272-3078. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JUAN A TORRES/Primary Examiner, Art Unit 2634
Prosecution Insights