DETAILED ACTION
Response to Arguments
Applicant's arguments filed with respect to claims 1-6 and 9-20 have been fully considered but are moot in view of the new ground(s) of rejection. The rejections are necessitated due to claim amendments.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Theobald et al. (Pub. No. US 2021/0097730) in view of Park et al. (Pub. No. US 2021/0358177).
Regarding claim 1, Theobald teaches computer-implemented method for generating modified images from input images using a trained machine learning, ML, model, [Para. 25 and abstract] the method comprising: obtaining an image depicting at least one human face [Para. 4, 6, and 73 “obtaining an input image that depicts a face of a subject”]; and using the trained ML model for: determining, for the obtained image, visual features (reference shape description) of one human face in the image [Para. 4, 6, 74, and abstract “determining a reference shape description based on the input image”]; generating, using the determined visual features, at least one representation in vector space (shape space) which encodes a specific attribute (facial expression difference; pose difference) of the human face in the image [Para. 45 “the shape estimation model 230 is performing a transformation within shape space from a reference shape, which is in this example is given by the reference shape description 228, to a target shape”; Para. 4, 6, and 75 “determining a target shape description based on the reference shape description, a facial expression difference, and a pose difference”]; modifying/transforming, in vector space (, one or more of the at least one generated representation [Para. 45 “performing a transformation within shape space from a reference shape, which is in this example is given by the reference shape description 228, to a target shape”]; and generating, using the or each modified generated representation, a modified image [Para. 17 “generate an image that looks like a realistic image of a human face, incorporates a face shape”; claim 1 and corresponding description “generating an output image based on the input image and the rendered target shape image using an image”].
However, Theobald doesn’t explicitly teach the rest of claim limitations.
Park teaches generating, a modified image (modified digital image; hybrid digital image) by training, using a training dataset and the modified one or more generated (global and spatial autoencoder; encoder neural network; generator neural network) representations, the trained ML model and generating the modified image (modified digital image; hybrid digital image) using trained ML model, the training dataset comprising a plurality of images depicting human faces (digital images that depict a particular attribute, e.g. a smiling face) [Para. 32 “the deep image manipulation system can extract latent codes from one or more digital images that depict a particular attribute (e.g., a smiling face or a snowy landscape”; Para. 103; para. 63 “training image distribution”; Para. 26 “the deep image manipulation system can generate a hybrid digital image that includes spatial features of a first digital image and global features of a second digital image”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald to teach the claim limitation, feature as taught by Park; because the modification enables the system improves the realism and control of image edits by using a global and spatial autoencoder that disentangles and recombines separate latent codes to generate high-quality modified digital images.
Claim 20 is rejected for the same reason as claim 1 above. Furthermore, Theobald teaches a display and at least one processor coupled to memory [fig. 7 and corresponding description].
Claims 1, 5, 6, 13, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Theobald et al. (Pub. No. US 2021/0097730) in view of KALAROT et al. (Pub. No. US 2022/0391611).
Regarding claim 1, Theobald teaches computer-implemented method for generating modified images from input images using a trained machine learning, ML, model, [Para. 25 and abstract] the method comprising: obtaining an image depicting at least one human face [Para. 4, 6, and 73 “obtaining an input image that depicts a face of a subject”]; and using the trained ML model for: determining, for the obtained image, visual features (reference shape description) of one human face in the image [Para. 4, 6, 74, and abstract “determining a reference shape description based on the input image”]; generating, using the determined visual features, at least one representation in vector space (shape space) which encodes a specific attribute (facial expression difference; pose difference) of the human face in the image [Para. 45 “the shape estimation model 230 is performing a transformation within shape space from a reference shape, which is in this example is given by the reference shape description 228, to a target shape”; Para. 4, 6, and 75 “determining a target shape description based on the reference shape description, a facial expression difference, and a pose difference”]; modifying/transforming, in vector space, one or more of the at least one generated representation [Para. 45 “performing a transformation within shape space from a reference shape, which is in this example is given by the reference shape description 228, to a target shape”]; and generating, using the or each modified generated representation, a modified image [Para. 17 “generate an image that looks like a realistic image of a human face, incorporates a face shape”; claim 1 and corresponding description “generating an output image based on the input image and the rendered target shape image using an image”].
However, Theobald doesn’t explicitly teach the rest of claim limitations.
KALAROT teaches generating, a modified image by training, using a training dataset (training set) and the modified one or more generated representations (modified latent vector), the trained ML model and generating the modified image using trained ML model, the training dataset comprising a plurality of images depicting human faces (image of face) [Para. 96 “a training set comprising an image of a face, a latent vector representing the image, and a target attribute vector representing a target attribute for the face, computing a modified latent vector based on the latent vector and the target attribute vector by performing a non-linear transformation of the latent vector using a mapping network.”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald to teach the claim limitation, feature as taught by KALAROT; because the modification enables the system to improve facial attribute editing by using a learned latent to latent that transforms face feature vectors to apply new attributes while better preserving the person’s identity.
Regarding claim 5, Theobald teaches wherein obtaining an image comprises obtaining an image or a single frame of a video [Para. 19 “one or more video frames are used as the input image 102”].
Regarding claim 6, Theobald teaches wherein obtaining an image comprises obtaining an image depicting a single human face [Para. 19 “The input image 102 may be a still image that shows a person, and in particular, shows a face of the person”].
Regarding claim 13, Theobald teaches receiving, from a user, information on at least one specific attribute to be modified, and how the at least one specific attribute is to be modified [Para. 22 “an absolute value may be provided as an input by a user or by another system, and the absolute value may be converted to a relative value for use in processing operations by the image generation system 100”].
Claim 20 is rejected for the same reason as claim 1 above. Furthermore, Theobald teaches a display and at least one processor coupled to memory [fig. 7 and corresponding description].
Claims 2-4 are rejected under 35 U.S.C. 103 as being unpatentable over Theobald et al. (Pub. No. US 2021/0097730) in view of in view of KALAROT et al. (Pub. No. US 2022/0391611) further in view of Xu (“A Latent Transformer for Disentangled Face Editing in Images and Videos”).
Regarding claim 2, Theobald teaches wherein the trained ML model comprises at least one transformer block [para. 25 “the image generation system 100 may be implemented using one or more neural networks.”].
However, Theobald in view of KALAROT doesn’t explicitly teach ML model comprises at least one transformer block.
Xu teaches wherein generating the at least one representation comprises: processing the determined visual features using the at least one transformer block of the ML model, wherein each of the at least one transformer block generates a representation in vector space of a specific attribute [1. Introduction: Para 5 “Our contributions can be summarized as follows. We propose a latent transformation network for facial attribute editing, achieving disentangled and controllable manipulations on real images with good identity preservation].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by Xu; because the modification enables the system improves facial image generation by enabling a machine learning system to independently control and adjust pose and expression while preserving the person’s identity in the synthesized image.
Theobald teaches operation 652 includes determining a reference shape description based on the input image [Para. 74], however, Theobald doesn’t explicitly teach the rest of claim limitation.
Xu teaches generating the at least one representation comprises: processing the determined visual features using the at least one transformer block of the ML model, wherein each of the at least one transformer block generates a representation in vector space of a specific attribute [Abstract “To tackle these limitations, we propose to edit facial attributes via the latent space of a StyleGAN generator, by training a dedicated latent transformation network and incorporating explicit disentanglement and identity preservation terms in the loss function”, “Our model achieves a disentangled, controllable, and identity preserving facial attribute editing, even in the challenging case of real (i.e., non-synthetic) images and videos”; fig. 5 and corresponding description “Given an input image, we manipulate a list of attributes sequentially”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by Xu; because the modification enables the system improves facial image generation by enabling a machine learning system to independently control and adjust pose and expression while preserving the person’s identity in the synthesized image.
Regarding claim 3, Theobald in view of KALAROT does not explicitly teach the claim limitation.
However, Xu teaches wherein the ML model comprises a plurality of transformer blocks [fig. 10 related description “we manipulate a list of attributes sequentially, where each time a single attribute is modified on top of the previous modification”; 1. Introduction: our contribution: “We propose a latent transformation network for facial attribute editing, achieving disentangled and controllable manipulations on real images with good identity preservation”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by Xu; because the modification enables the system improves facial image generation by enabling a machine learning system to independently control and adjust pose and expression while preserving the person’s identity in the synthesized image.
Theobald in view of KALAROT doesn’t explicitly teach the rest of claim limitation.
Xu teaches wherein generating at least one representation comprises: using each of the plurality of transformer blocks to generate a representation of different specific attributes of the human face in the image [fig. 5 and corresponding description “we manipulate a list of attributes sequentially, where each time a single attribute is modified from the previous latent representation”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by Xu; because the modification enables the system improves facial image generation by enabling a machine learning system to independently control and adjust pose and expression while preserving the person’s identity in the synthesized image.
Regarding claim 4. Theobald teaches performing a transformation within shape space from a reference shape [Para. 45].
However, Theobald in view of KALAROT doesn’t explicitly teach modifying one or more of the at least one generated representation comprises using a transformer-based face editing module of the ML model to modify, in vector space, at least one specific attribute of the human face.
Xu teaches modifying one or more of the at least one generated representation comprises using a transformer-based face editing module of the ML model to modify, in vector space, at least one specific attribute of the human face [section 3.1 “train a latent transformer T in the latent space to edit a single attribute of the projected image G(w)”; Abstract “Our model achieves a disentangled, controllable, and identity preserving facial attribute editing, even in the challenging case of real (i.e., non-synthetic) images and videos”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by Xu; because the modification enables the system improves facial image generation by enabling a machine learning system to independently control and adjust pose and expression while preserving the person’s identity in the synthesized image.
Claims 9-12 are rejected under 35 U.S.C. 103 as being unpatentable over Theobald et al. (Pub. No. US 2021/0097730) in view of in view of KALAROT et al. (Pub. No. US 2022/0391611) further in view of Kumar et al. (Pub. No. US 2019/0370529).
Regarding claim 9, Theobald in view of KALAROT doesn’t explicitly teach the claim limitation.
However, Kumar teaches wherein the method comprises: recognizing, using the ML model, a specific user's face as one of the at least two human faces; and processing, using the ML model, the recognized specific user's face [Para. 3 “Facial recognition processes may generally be used to identify individuals in an image.”; Para. 96 “may be used to detect one or more faces in an image captured by camera 102 on device 100.”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by Kumar; because the modification enables the system improves face detection accuracy by providing a system that can detect one or more faces in an image and output detailed properties such as location, pose, and distance.
Regarding claim 10, Theobald in view of KALAROT doesn’t explicitly teach the claim limitation.
However, Kumar teaches separately processing, using the ML model, each of the at least two human faces [Para. 96 “detect one or more faces in an image captured by camera 102”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by Kumar; because the modification enables the system improves face detection accuracy by providing a system that can detect one or more faces in an image and output detailed properties such as location, pose, and distance.
Regarding claim 11, Theobald teaches the image generation system 100 may be used in the context of still image or video editing, to revise the head pose or facial expression of a subject [Para. 79].
However, Theobald in view of KALAROT doesn’t explicitly teach the rest of claim limitation.
Kumar teaches for each of the at least two human faces: modifying, in vector space, a generated representation that represents one specific attribute [Para. 96 “Output data 260 may include a decision on a face being in the captured image along with data for values of properties of the detected face (e.g., location, pose, and/or distance from the camera)”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by Kumar; because the modification enables the system improves face detection accuracy by providing a system that can detect one or more faces in an image and output detailed properties such as location, pose, and distance.
Regarding claim 12, Theobald teaches modifying, in vector space, a generated representation that represents a different specific attribute for each of the human faces [Para. 15 and 16].
Claims 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Theobald et al. (Pub. No. US 2021/0097730) in view of in view of KALAROT et al. (Pub. No. US 2022/0391611) further in view of Fu et al. (Patent No. US 10,339,685).
Regarding claim 14, Theobald in view of KALAROT doesn’t explicitly teach the rest of claim limitation.
However, Fu teaches determining, using the ML model, at least one generated representation to be modified to improve an attractiveness score of the image of the human face [Abstract].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by Fu; because the modification enables the system improves facial beauty analysis accuracy by introducing an autoencoder based system that learns attractiveness aware features for more objective assessment.
Regarding claim 15, Theobald in view of KALAROT doesn’t explicitly teach the rest of claim limitation.
However, Fu teaches wherein the attractiveness score is improved based on learned image preferences during training of the ML model [Abstract “extract attractiveness-aware features to perform an assessment of facial beauty”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by Fu; because the modification enables the system improves facial beauty analysis accuracy by introducing an autoencoder based system that learns attractiveness aware features for more objective assessment.
Regarding claim 16, Theobald in view of KALAROT doesn’t explicitly teach the rest of claim limitation.
However, Fu teaches the attractiveness score is improved based on learning a user's image preferences [Abstract “An autoencoder-based framework is provided to extract attractiveness-aware features to perform an assessment of facial beauty”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by Fu; because the modification enables the system improves facial beauty analysis accuracy by introducing an autoencoder based system that learns attractiveness aware features for more objective assessment.
Regarding claim 17, Theobald in view of KALAROT doesn’t explicitly teach the rest of claim limitation.
However, Fu teaches receiving at least one positive sample image of a human face from the user indicative of image preferences the user likes and learning, using the ML model and the at least one received sample image, one or more features of the image of a human face indicative of image preferences [Col. 22 lines 26-30].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by Fu; because the modification enables the system improves facial beauty analysis accuracy by introducing an autoencoder based system that learns attractiveness aware features for more objective assessment.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Theobald et al. (Pub. No. US 2021/0097730) in view of in view of KALAROT et al. (Pub. No. US 2022/0391611) further in view of PERRY et al. (Pub. No. US 2020/0097767).
Regarding claim 18, Theobald in view of KALAROT doesn’t explicitly teach the rest of claim limitation.
However, PERRY teaches receiving, from a user, an input indicating that face anonymization is to be performed; wherein modifying, in vector space, at least one generated representation comprises modifying the at least one generated representation so that the generated modified image comprises an anonymized version of the human face in the obtained image depicting at least one human face [Para. 61 “the output image data may be generally identical to the input image data other than one or more selected face regions that may be modified to prevent, or at least significantly limit, ability of computerized methods or techniques in determining identity associated with the one or more modified faces in the image data”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by PERRY; because the modification enables the system improves privacy protection in digital images by providing a de-identification system.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Theobald et al. (Pub. No. US 2021/0097730) in view of in view of KALAROT et al. (Pub. No. US 2022/0391611) further in view of Rymkowski et al. (Pub. No. US 2018/0082407).
Regarding claim 19, Theobald in view of KALAROT doesn’t explicitly teach the rest of claim limitation.
However, Rymkowski teaches receiving, from a user, an input indicating a preferred style; wherein modifying, in vector space, at least one generated representation comprises modifying the at least one generated representation so that the generated modified image is in the preferred style [Para 6 “for applying an artistic style extracted from one or more source images, e.g., paintings, to one or more target images.” And “The extracted artistic style may then be stored as a plurality of layers in a neural network].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Theobald in view of KALAROT to teach the claim limitation, feature as taught by Rymkowski; because the modification enables the system improves privacy protection in digital images by providing a de-identification system.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOLOMON G BEZUAYEHU whose telephone number is (571)270-7452. The examiner can normally be reached on Monday-Friday 10 AM-8 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 888-786-0101 (IN USA OR CANADA) or 571-272-4000.
/SOLOMON G BEZUAYEHU/
Primary Examiner, Art Unit 2666