DETAILED ACTION
The United States Patent & Trademark Office appreciates the response filed for the current application that is submitted on 04/23/2026. The United States Patent & Trademark Office reviewed the following documents submitted and has made the following comments below.
Amendment
Applicant submitted amendments on 04/23/2026. The Examiner acknowledges the amendment and has reviewed the claims accordingly.
Priority
Receipt is acknowledged that application is a 371 of PCT/CN2023/113992. Applicant claims the benefit of Foreign Priority from Application No CN202211075798.7, filed 09/05/2022. Claims 1-20 have been afforded the benefit of this filing date.
Information Disclosure Statement
The IDS dated 01/18/2024 have been considered and placed in the application file.
Applicant Arguments:
In regards to the argument on Argument 1, Applicant/s state/s “Applicant respectfully submits that the cited references do not teach or suggest at least the claimed ‘wherein the training comprises determining a pixel difference between the second identity swapping image and the fake labeled image.’” (See Remarks Pg 17, paragraph 1). Therefore U.S.C 103 rejection on Claim 1 should be withdrawn.
In regards to the argument on Argument 2, Applicant/s state/s “When rejecting claim
2, the Office failed to demonstrate how using the "discriminator" to label the "output" of the "generator" teaches determining a pixel difference between the second identity swapping image
and the fake labeled image.” (See Remarks Pg 17, paragraph 3). Therefore U.S.C 103 rejection on Claim 1 should be withdrawn.
In regards to the argument on Argument 3, Applicant/s state/s “The Office failed to demonstrate how computing a "pixel difference" between inputs and outputs of the "generation
network" can be relied upon to allegedly teach the claimed ‘determining a pixel difference between the second identity swapping image and the fake labeled image’” (See Remarks Pg 18, paragraph 1). Therefore U.S.C 103 rejection on Claim 1 should be withdrawn.
In regards to the argument on Argument 4, Applicant/s state/s “In regards to the argument on Argument 3, Applicant/s state/s “Independent claims 13 and 18 recite similar features as noted above with respect to claim 1. Therefore, Applicant respectfully submits that claims 13 and 18 are patentable for similar reasons as noted above with respect to claim 1. Dependent claims 2-12, 14-17, and 19-20 each depend from one of independent claims 1, 13, and 18 described above. Accordingly, Applicant respectfully submits that claims 2-12, 14-17, and 19-20 are patentable at least due to their respective dependencies and the additional features recited therein. ‘” (See Remarks Pg 19, paragraph 1-2). Therefore U.S.C 103 rejection on Claims 2-20 should be withdrawn.
Examiner’s Responses:
In response to Argument 1, Applicant’s arguments, see Remarks, filed 04/23/2026, with respect to the rejection(s) of claim(s) 1-2, 4-8, 12-14, 16-19 under 35 U.S.C. 103 have been considered and are persuasive. Therefore, the rejection has been withdrawn due to the amendment. However, upon further consideration, a new ground(s) of rejection is made for claims 1-2, 4-8, 12-14, 16-19 under 35 U.S.C. 103 in view of Cao et al CN111401216A (using translation from epsace.net and images translated from google translate) hereafter referred to as Cao) in view of Zhu et al. CN111353546A (using translation from epsace.net and images translated from google translate) hereafter referred to as Zhu) in further view of Kim et al (J. Kim, J. Lee and B. -T. Zhang, "Smooth-Swap: A Simple Enhancement for Face-Swapping with Smoothness," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 10769-10778).
The Examiner finds that Cao teaches on the amendment claim language “the second identity swapping image” in the amended independent claims 1, 13, and 18.
Specifically, Cao teaches a second initial facial image and second target facial image corresponding to the facial image sample and the second template facial image sample, which is used in fusion model to swap the attribute features in Pg 13 ¶08 and Fig 10, 1003, and Pg 2 ¶01. Applicant argues that “the cited references do not teach or suggest at least the claimed ‘wherein the training comprises determining a pixel difference between the second identity swapping image and the fake labeled image”. However, we determine claim scope not solely on the basis of claim language, but also on giving claims their broadest reasonable construction in light of the specification as it would be interpreted by one of ordinary skill in the art. In re Am. Acad. of Sci. Tech. Ctr., 367 F.3d 1359, 1364 (Fed. Cir. 2004). See also Superguide Corp. v. DirecTV Enterprises, Inc., 358 F.3d 870, 875 (Fed. Cir. 2004) (“Though understanding the claim language may be aided by explanations contained in the written description, it is important not to import into a claim limitations that are not part of the claim.”). The Examiner interprets that under broadest reasonable interpretation “swapping” has no special definition in the claims, and therefore can be interpreted as the swapping of features between a target sample and an initial sample image, and based on the cited references this can be done iteratively resulting in a second identity swapped image. Therefore, the Examiner interprets that Cao teaches the main concept of using an identity swapping model based on multiple initial and target facial images, the additional details of the functions of the main concepts as stated above by the applicant in the amendments is taught by Zhu and Kim in the details of the rejection below. The Examiner will maintain prior art Cao and details of the rejection are below.
The Examiner finds that Zhu teaches on the amendment claim language “the fake labeled image” in the amended independent claims 1, 13, and 18.
Specifically, Zhu teaches a discriminator that labels the images including fake and non-fake images based on the attributes in Pg 1 ¶10-¶12 and Pg 10 ¶01 discloses the fake images being used to train the model. Applicant argues that “the cited references do not teach or suggest at least the claimed ‘wherein the training comprises determining a pixel difference between the second identity swapping image and the fake labeled image”. However, we determine claim scope not solely on the basis of claim language, but also on giving claims their broadest reasonable construction in light of the specification as it would be interpreted by one of ordinary skill in the art. In re Am. Acad. of Sci. Tech. Ctr., 367 F.3d 1359, 1364 (Fed. Cir. 2004). See also Superguide Corp. v. DirecTV Enterprises, Inc., 358 F.3d 870, 875 (Fed. Cir. 2004) (“Though understanding the claim language may be aided by explanations contained in the written description, it is important not to import into a claim limitations that are not part of the claim.”). The Examiner interprets that under broadest reasonable interpretation “fake” has no special definition in the claims, and therefore can be interpreted as attributes that allow the image to be labeled as fake. Therefore, the Examiner interprets that Cao and Zhu teach the main concept of using an identity swapping model based on multiple initial and target facial images with labels, the additional details of the functions of the main concepts as stated above by the applicant in the amendments is taught by Kim in the details of the rejection below. The Examiner will maintain prior art Zhu and details of the rejection are below.
In response to Arguments 2 and 3, see Remarks, filed 04/23/2026, with respect to the rejection(s) of claim(s) 1-2, 4-8, 12-14, 16-19 under 35 U.S.C. 103 have been considered and are moot in view of new ground(s) of rejection based on the amendments. A new reference, Kim et al (J. Kim, J. Lee and B. -T. Zhang, "Smooth-Swap: A Simple Enhancement for Face-Swapping with Smoothness," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 10769-10778), has been introduced which in Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change. Kim introduces a similar process as described in the amended claim where the pixel level change is used to train the model. After reviewing the amendments, the Examiner interprets that Cao et al CN111401216A (using translation from epsace.net and images translated from google translate) hereafter referred to as Cao) in view of Zhu et al. CN111353546A (using translation from epsace.net and images translated from google translate) hereafter referred to as Zhu) in further view of Kim et al (J. Kim, J. Lee and B. -T. Zhang, "Smooth-Swap: A Simple Enhancement for Face-Swapping with Smoothness," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 10769-10778) teaches on the amended claims that were presented. The details of the rejection are listed below.
In response to Argument 4, see Remarks, filed 04/23/2026, with respect to the rejection(s) of claim(s) 3, 9-11, 15, 20 under 35 U.S.C. 103 have been considered and are moot in view of new ground(s) of rejection based on the amendments. A new reference, Kim et al (J. Kim, J. Lee and B. -T. Zhang, "Smooth-Swap: A Simple Enhancement for Face-Swapping with Smoothness," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 10769-10778), has been introduced which in Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change. Kim introduces a similar process as described in the amended claim where the pixel level changed is used to train the model. After reviewing the amendments, the Examiner interprets that Cao et al CN111401216A (using translation from epsace.net and images translated from google translate) hereafter referred to as Cao) in view of Zhu et al. CN111353546A (using translation from epsace.net and images translated from google translate) hereafter referred to as Zhu) in further view of Kim et al (J. Kim, J. Lee and B. -T. Zhang, "Smooth-Swap: A Simple Enhancement for Face-Swapping with Smoothness," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 10769-10778) in further view of Berlin et al (US Patent No 11,308,957 hereafter referred to as Berlin) teaches on the amended claims and their dependent claims that were presented. The details of the rejection are listed below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 4-8, 12-14, and 16-19 are rejected under 35 U.S.C. 103 as unpatentable over Cao et al CN111401216A (using translation from epsace.net and images translated from google translate) hereafter referred to as Cao) in view of Zhu et al. CN111353546A (using translation from epsace.net and images translated from google translate) hereafter referred to as Zhu) in further view of Kim et al (J. Kim, J. Lee and B. -T. Zhang, "Smooth-Swap: A Simple Enhancement for Face-Swapping with Smoothness," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 10769-10778).
Regarding Claim 1, Cao teaches an image processing method (Cao Pg 1, ¶05 and Pg 1 ¶09 discloses an image processing method) performed by a computer device (Cao Pg 2 ¶03 and Pg 3 ¶06 discloses a computer device with memory to execute a computer program), the image processing method comprising:
comprising a first source image (Cao Pg 2 ¶04 discloses an initial facial image) and a fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) being based on identity swapping processing of the real labeled image (Cao Pg 2 ¶14 discloses the swapping of the target feature on the facial image so that the target facial image matches the facial identity characteristics of the initial facial image, and matches the attribute characteristics of the template facial image), the first source image (Cao Pg 2 ¶04 discloses an initial facial image) having a same identity attribute (Cao Pg 6 ¶04 disclose the initial facial image provides facial identity features), and the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) having a same non-identity attribute (Cao Pg 6 ¶04 disclose the template image providing the attribute features);
inputting (Cao Fig 5 discloses inputting the image into the model) the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) into an identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) and performing identity swapping processing (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) on the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) based on the first source image (Cao Pg 2 ¶04 discloses an initial facial image) to obtain a first identity swapping image of the fake template image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character);
comprising a second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image), a real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) the fake labeled image being based on identity swapping processing (Cao Pg 2 ¶14 discloses the swapping of the target feature on the facial image so that the target facial image matches the facial identity characteristics of the initial facial image, and matches the attribute characteristics of the template facial image) of the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) based on the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image), the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image) having a same identity attribute (Cao Pg 6 ¶04 disclose the initial facial image provides facial identity features), and the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) having a same non-identity attribute (Cao Pg 6 ¶04 disclose the template image providing the attribute features);
inputting (Cao Fig 5 discloses inputting the image into the model) the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) into the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) and performing identity swapping processing (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) on the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) based on the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image) to obtain a second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) of the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image); and
training (Cao Pg 2 ¶17 disclose the method for training the model) the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature)
the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character)
and the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) to generate a trained identity swapping model (Cao Pg 13 ¶08 discloses training the model) to perform identity swapping processing (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) on a target template image (Cao Pg 9 ¶04 discloses the template facial image reference) based on a target source image (Cao Pg 12 ¶05 and Pg 16 ¶02 disclose a target image)
between the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image).
Cao does not explicitly disclose obtaining a fake template sample group, a real labeled image, and the real labeled image, and the real labeled image, obtaining a fake labeled sample group, and a fake labeled image, and the fake labeled image, and the fake labeled image, based on the fake template sample group, the fake labeled sample group, and the fake labeled image.
Zhu is in the same field of image analysis to produce synthesized images or video. Further, Zhu teaches obtaining a fake template sample group (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image), a real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image), and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image), and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image) obtaining a fake labeled sample group (Zhu Pg 1 ¶10-¶12 discloses a triplet sample group that includes an image with attributes including a fake image) and a fake labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶01-¶02 and discloses labeling the second combination as a fake image), and the fake labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶01-¶02 and discloses labeling the second combination as a fake image), and the fake labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶01-¶02 and discloses labeling the second combination as a fake image) based on the fake template sample group (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image) the fake labeled sample group (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image) and the fake labeled image (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Cao by including real labeled images as part of the fake template sample group and updating the parameters of the model and assigning weights to the pixel differences as taught by Zhu, to make an invention that can more accurately differentiate between the features need to complete a successful and clear swapped image; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to improve the stability and robustness of the image processing model, the image processing model obtained by the triple sample training can ensure the synthesis of area and the original image or the original video in shape, illumination, the action is consistent, thereby improving the quality of the image and a synthetic video (Zhu Pg 14 ¶06).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Cao and Zhu in combination do not explicitly disclose wherein the training comprising determining a pixel difference.
Kim is in the same field of image analysis to produce synthesized images or video. Further, Kim teaches wherein the training comprising determining a pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Cao in view of Zhu by incorporating the pixel difference into the training of the model as taught by Kim, to make an invention that can more accurately differentiate between the features need to complete a successful and clear swapped image based on the pixel difference; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to allow for faster training of the image generator by providing more stable gradient information (Kim Pg 2, Col 1, ¶03).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 2, Cao in view of Zhu in further view of Kim teaches the image processing method according to claim 1, wherein the training comprises:
determining a pixel reconstruction loss (Cao Pg 13 ¶03-¶04 discloses a pixel reconstruction loss) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) based on a pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change ) (Cao Pg 16 ¶10 discloses the pixel difference between the first images) between the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image) and a pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change ) (Cao Pg 13 ¶08 discloses a pixel difference between the second images) between the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) and the fake labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶01-¶02 and discloses labeling the second combination as a fake image);
determining a feature reconstruction loss (Cao Pg 13 ¶04 and Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) based on a feature difference (Cao Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function) between the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image);
extracting face features (Cao Pg 15 ¶04 discloses feature points from the images including eyes, nose, mouth, eyebrows) of the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) the first source image (Cao Pg 2 ¶04 discloses an initial facial image), the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image), the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image), and the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) to determine an identity loss (Cao Pg 13 ¶04 and Pg 14 ¶04 discloses an identity loss) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature);
performing discriminative processing (Pg 13 ¶089 discloses inputting the image into a discriminant network) on the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) to obtain an adversarial loss (Cao Pg 13 ¶04 discloses a loss function of the adversarial training generation network) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature); and
performing summation (Cao Pg 13 ¶04 discloses performing weighted summation) on the pixel reconstruction loss (Cao Pg 13 ¶03-¶04 discloses a pixel reconstruction loss), the feature reconstruction loss (Cao Pg 13 ¶04 and Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function), the identity loss (Cao Pg 13 ¶04 and Pg 14 ¶04 discloses an identity loss), and the adversarial loss (Cao Pg 13 ¶04 discloses a loss function of the adversarial training generation network) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature), to obtain loss information of the identity swapping model (Cao Pg 13 ¶03 discloses obtaining loss information from the model), and updating model parameters of the identity swapping model (Zhu Fig 11 1108 Pg 20 ¶01-03 discloses updating the samples to continue training the model) based on the loss information of the identity swapping model (Cao Pg 13 ¶03 discloses obtaining loss information from the model) to train (Cao Pg 2 ¶17 disclose the method for training the model) the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature). See claim 1 for rationale, its parent claim.
Regarding Claim 4, Cao in view of Zhu in view of Kim teaches the image processing method according to claim 2, wherein the identity loss (Cao Pg 13 ¶04 and Pg 14 ¶04 discloses an identity loss) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) comprises a first identity loss (Cao Pg 13 ¶04 disclose performing an identity loss) and a second identity loss (Cao Pg 14 ¶04 discloses the identity loss also being calculated for the second image);
and the extracting comprises:
determining the first identity loss (Cao Pg 13 ¶04 and Pg 14 ¶04 discloses an identity loss) based on a similarity between face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) the first source image (Cao Pg 2 ¶04 discloses an initial facial image), and face features (Cao Pg 13 ¶02 discloses facial identity features) of the first source image (Cao Pg 2 ¶04 discloses an initial facial image) and a similarity between face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) and face features (Cao Pg 13 ¶02 discloses facial identity features) of the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image); and
determining the second identity loss (Cao Pg 14 ¶04 discloses the identity loss also being calculated for the second image) based on a similarity between the face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and face features (Cao Pg 13 ¶02 discloses facial identity features) of the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), a similarity between the face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the first source image (Cao Pg 2 ¶04 discloses an initial facial image) and the face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), a similarity between the face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) and face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), and a similarity between the face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image) and the face features of the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image). See claim 1 for rationale, its parent claim.
Regarding Claim 5, Cao in view of Zhu in view of Kim teaches the image processing method according to claim 2, wherein the performing discriminative processing (Pg 13 ¶089 discloses inputting the image into a discriminant network) on the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) comprises:
obtaining a discriminative model (Pg 13 ¶089 discloses inputting the image into a discriminant network);
inputting the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) into the discriminative model (Cao Pg 13 ¶089 discloses inputting the image into a discriminant network) and performing discriminative processing (Pg 13 ¶089 discloses inputting the image into a discriminant network) on the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) to obtain a first discriminative result (Cao Pg 12 ¶06 discloses the output being the different in pixels between first images);
inputting the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) into the discriminative model (Pg 13 ¶089 discloses inputting the image into a discriminant network) and performing discriminative processing (Pg 13 ¶089 discloses inputting the image into a discriminant network) on the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) to obtain a second discriminative result (Cao Pg 13 ¶08 discloses the output being the second target facial image samples being used as negative samples); and
determining the adversarial loss (Cao Pg 13 ¶04 discloses a loss function of the adversarial training discrimination network) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) based on the first discriminative result (Cao Pg 12 ¶06 discloses the output being the different in pixels between first images) and the second discriminative result (Cao Pg 13 ¶08 discloses the output being the second target facial image samples being used as negative samples). See claim 1 for rationale, its parent claim.
Regarding Claim 6, Cao in view of Zhu in view of Kim teaches the image processing method according to claim 2, wherein determining a pixel reconstruction loss (Cao Pg 13 ¶03-¶04 discloses a pixel reconstruction loss) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) based on the pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a lose function including pixel level change) (Cao Pg 16 ¶10 discloses the pixel difference between the first images) between the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image) and the pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a lose function including pixel level change) (Cao Pg 13 ¶08 discloses a pixel difference between the second images) between the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) and the fake labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶01-¶02 and discloses labeling the second combination as a fake image) comprises:
obtaining a first weight (Zhu Pg 14 ¶02 discloses assigning weighted for each feature vector) corresponding to the pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change) between the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image Pg 14 ¶02 discloses assigning weighted for each feature vector corresponding to the similarity of each feature vector) and a second weight (Zhu Pg 14 ¶02 discloses assigning weighted for each feature vector) corresponding to the pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change) between the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) and the fake labeled image (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image Pg 14 ¶02 discloses assigning weighted for each feature vector corresponding to the similarity of each feature vector);
performing weighted processing (Cao Pg 13 ¶4 and Pg 14 ¶05 disclose weighted summation being performed) on the pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change) between the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image) based on the first weight (Zhu Pg 14 ¶02 discloses assigning weighted for each feature vector), to obtain a first weighted pixel difference (Cao Pg 16 ¶10 discloses the pixel difference between the first images Pg 7 ¶06 discloses weight addition on characteristics for use in the network);
performing weighted processing (Cao Pg 13 ¶4 and Pg 14 ¶05 disclose weighted summation being performed) on the pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change) (Cao Pg 13 ¶08 discloses a second output image) between the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) and the fake labeled image (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image) based on the second weight (Zhu Pg 14 ¶02 discloses assigning weighted for each feature vector), to obtain a second weighted pixel difference (Cao Pg 13 ¶08 discloses a pixel difference between the second images and Pg 7 ¶06 discloses weight addition on characteristics for use in the network); and
performing summation (Cao Pg 13 ¶04 discloses performing weighted summation) on the first weighted pixel difference(Cao Pg 16 ¶10 discloses the pixel difference between the first images Pg 7 ¶06 discloses weight addition on characteristics for use in the network) and the second weighted pixel difference (Cao Pg 13 ¶08 discloses a pixel difference between the second images and Pg 7 ¶06 discloses weight addition on characteristics for use in the network), to obtain the pixel reconstruction loss (Cao Pg 13 ¶03-¶04 discloses a pixel reconstruction loss) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature). See claim 1 for rationale, its parent claim.
Regarding Claim 7, Cao in view of Zhu in view of Kim teaches the image processing method according to claim 1, wherein the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) comprises an encoding network (Cao Pg 7 ¶03 discloses a neural network which is an encoding model) and a decoding network (Cao Pg 10 ¶02 discloses a decoding model); and
performing the identity swapping processing (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) on the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) based on the first source image (Cao Pg 2 ¶04 discloses an initial facial image) comprises:
calling the encoding network (Cao Pg 7 ¶03 discloses a neural network which is an encoding model) to perform fusion encoding processing (Cao Pg 10 ¶03 discloses the feature fusion model being an encoding model) on the first source image (Cao Pg 2 ¶04 discloses an initial facial image) and the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), to obtain an encoding result (Cao Pg 8 ¶04 discloses the result of the encoding being obtaining the facial features of the first image); and
calling the decoding network (Cao Pg 10 ¶02 discloses a decoding model) to perform decoding processing (Cao Pg 11 ¶4 discloses the decoding model is used to decode the target feature to obtain an image that matches the facial identity feature of the initial facial image sample) on the encoding result (Cao Pg 8 ¶04 discloses the result of the encoding being obtaining the facial features of the first image) to obtain the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) of the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image). See claim 1 for rationale, its parent claim.
Regarding Claim 8, Cao in view of Zhu in view of Kim teaches the image processing method according to claim 7, wherein calling the encoding network (Cao Pg 7 ¶03 discloses a neural network which is an encoding model) to perform fusion encoding processing (Cao Pg 10 ¶03 discloses the feature fusion model being an encoding model) on the first source image (Cao Pg 2 ¶04 discloses an initial facial image) and the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) comprises:
performing splicing processing (Cao Pg 20 ¶04 discloses performing splicing) on the first source image (Cao Pg 2 ¶04 discloses an initial facial image) and the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), to obtain a spliced image (Cao Pg 20 ¶04 discloses splicing the image features from the image);
performing feature learning (Cao Pg 15 ¶09 and Pg 8 ¶06 discloses a recognition feature coding model and using machine learning for facial identity features) on the spliced image (Cao Pg 20 ¶04 discloses splicing the image features from the image) to obtain identity swapping features (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character);
performing face feature recognition (Cao Pg 15 ¶09 and Pg 8 ¶06 discloses a recognition feature coding model and using machine learning for facial identity features) on the first source image (Cao Pg 2 ¶04 discloses an initial facial image) to obtain face features of the first source image (Cao Pg 8 ¶2 discloses obtaining the facial identity characteristics of the initial facial image); and
performing feature fusion processing (Cao Pg 10 ¶02-¶04 discloses a feature fusion model used to obtain target features including: the facial identity feature, attribute feature) on the identity swapping features (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the face features of the first source image (Cao Pg 8 ¶2 discloses obtaining the facial identity characteristics of the initial facial image) to obtain the encoding result (Cao Pg 8 ¶04 discloses the result of the encoding being obtaining the facial features of the first image). See claim 1 for rationale, its parent claim.
Regarding Claim 12, Cao in view of Zhu in view of Kim teaches the image processing method according to claim 1, wherein training (Cao Pg 2 ¶17 disclose the method for training the model) the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature), to use a trained identity swapping model (Cao Pg 13 ¶08 discloses training the model) to perform the identity swapping processing on the (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) on a target template image (Cao Pg 9 ¶04 discloses the template facial image reference) based on a target source image (Cao Pg 12 ¶05 and Pg 16 ¶02 disclose a target image):
receiving the target source image (Cao Pg 12 ¶05 and Pg 16 ¶02 disclose a target image) and the target template image (Cao Pg 9 ¶04 discloses the template facial image reference) that are to be processed (Cao Pg 16 ¶03 discloses the image and template being processed); and
inputting the target template image (Cao Figure 5 discloses inputting the template face image into a model) into the trained identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) and performing identity swapping processing (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) on the target template image (Cao Pg 9 ¶04 discloses the template facial image reference) based on the target source image (Cao Pg 12 ¶05 and Pg 16 ¶02 disclose a target image) to obtain an identity swapping image (Cao Figure 5 discloses the output being the target facial image result) of the target template image (Cao Pg 9 ¶04 discloses the template facial image reference), wherein the target source image(Cao Pg 12 ¶05 and Pg 16 ¶02 disclose a target image) and the identity swapping image (Cao Figure 5 discloses the output being the target facial image result) of the target template image(Cao Pg 9 ¶04 discloses the template facial image reference) have a same identity attribute (Cao Pg 6 ¶04 disclose the initial facial image provides facial identity features), and the target template image (Cao Pg 9 ¶04 discloses the template facial image reference) and the identity swapping image (Cao Figure 5 discloses the output being the target facial image result) of the target template image (Cao Pg 9 ¶04 discloses the template facial image reference) have a same non-identity attribute (Cao Pg 6 ¶04 disclose the template image providing the attribute features). See claim 1 for rationale, its parent claim.
Regarding Claim 13, Cao teaches an image processing apparatus (Cao Pg 1 ¶01 and Pg 1 ¶15 discloses an image processing device), comprising:
at least one memory (Cao Pg 2 ¶03 discloses a memory) configured to store program code (Cao Pg 2 ¶03 discloses the memory storing a computer program); and
at least one processor (Cao Pg 2 ¶03 discloses a processor) configured to read the program code and operate as instructed by the program code (Cao Pg 2 ¶03 discloses a processor reads and executes the program), the program code comprising:
obtaining code (Cao Pg 2 ¶03 discloses a computer program) configured to cause at least one of the at least one processor (Cao Pg 2 ¶03 discloses a processor reads and executes the program) comprising a first source image (Cao Pg 2 ¶04 discloses an initial facial image) and a fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) being based on identity swapping processing of the real labeled image (Cao Pg 2 ¶14 discloses the swapping of the target feature on the facial image so that the target facial image matches the facial identity characteristics of the initial facial image, and matches the attribute characteristics of the template facial image), the first source image (Cao Pg 2 ¶04 discloses an initial facial image) having a same identity attribute (Cao Pg 6 ¶04 disclose the initial facial image provides facial identity features), and the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) having a same non-identity attribute (Cao Pg 6 ¶04 disclose the template image providing the attribute features); and
processing code (Cao Pg 2 ¶03 discloses a processor reads and executes the program) configured to cause at least one of the at least one processor (Cao Pg 2 ¶03 discloses a processor) to input (Cao Fig 5 discloses inputting the image into the model) the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) into an identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) and performing identity swapping processing (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) on the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) based on the first source image (Cao Pg 2 ¶04 discloses an initial facial image) to obtain a first identity swapping image of the fake template image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character), wherein the obtaining code (Cao Pg 2 ¶03 discloses a computer program) is further configured to cause at least one of the at least one processor (Cao Pg 2 ¶03 discloses a processor reads and executes the program)
comprising a second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image), a real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) the fake labeled image being based on identity swapping processing (Cao Pg 2 ¶14 discloses the swapping of the target feature on the facial image so that the target facial image matches the facial identity characteristics of the initial facial image, and matches the attribute characteristics of the template facial image) of the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) based on the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image), the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image) having a same identity attribute (Cao Pg 6 ¶04 disclose the initial facial image provides facial identity features), and the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) having a same non-identity attribute (Cao Pg 6 ¶04 disclose the template image providing the attribute features); and
the processing code (Cao Pg 2 ¶03 discloses a processor reads and executes the program) is further configured to cause at least one of the at least one processor (Cao Pg 2 ¶03 discloses a processor reads and executes the program) to:
input (Cao Fig 5 discloses inputting the image into the model) the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) into the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) and performing identity swapping processing (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) on the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) based on the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image) to obtain a second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) of the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image); and
train (Cao Pg 2 ¶17 disclose the method for training the model) the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character), and the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) to generate a trained identity swapping model (Cao Pg 13 ¶08 discloses training the model) to perform identity swapping processing (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) on a target template image (Cao Pg 9 ¶04 discloses the template facial image reference) based on a target source image (Cao Pg 12 ¶05 and Pg 16 ¶02 disclose a target image)
between the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image).
Cao does not explicitly disclose to obtain a fake template sample group, a real labeled image, and the real labeled image, and the real labeled image, to obtain a fake labeled sample group, and a fake labeled image, and the fake labeled image, and the fake labeled image, based on the fake template sample group, the fake labeled sample group, and the fake labeled image.
Zhu is in the same field of image analysis to produce synthesized images or video. Further, Zhu teaches to obtain a fake template sample group (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image) a real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image), and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image), and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image), to obtain a fake labeled sample group (Zhu Pg 1 ¶10-¶12 discloses a triplet sample group that includes an image with attributes including a fake image) and a fake labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶01-¶02 and discloses labeling the second combination as a fake image) and the fake labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶01-¶02 and discloses labeling the second combination as a fake image) and the fake labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶01-¶02 and discloses labeling the second combination as a fake image) and the fake labeled image (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Cao by including real labeled images as part of the fake template sample group and updating the parameters of the model and assigning weights to the pixel differences as taught by Zhu, to make an invention that can more accurately differentiate between the features need to complete a successful and clear swapped image; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to improve the stability and robustness of the image processing model, the image processing model obtained by the triple sample training can ensure the synthesis of area and the original image or the original video in shape, illumination, the action is consistent, thereby improving the quality of the image and a synthetic video (Zhu Pg 14 ¶06).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Cao and Zhu in combination do not explicitly disclose wherein the training is based on a pixel difference.
Kim is in the same field of image analysis to produce synthesized images or video. Further, Kim teaches wherein the training is based on a pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Cao in view of Zhu by incorporating the pixel difference into the training of the model as taught by Kim, to make an invention that can more accurately differentiate between the features need to complete a successful and clear swapped image based on the pixel difference between images; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to allow for faster training of the image generator by providing more stable gradient information (Kim Pg 2, Col 1, ¶03).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 14, Cao in view of Zhu in view of Kim teaches the image processing apparatus according to claim 13, wherein the processing code (Cao Pg 2 ¶03 discloses a processor reads and executes the program) is further configured to cause at least one of the at least one processor (Cao Pg 2 ¶03 discloses a processor reads and executes the program) to determine a pixel reconstruction loss (Cao Pg 13 ¶03-¶04 discloses a pixel reconstruction loss) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) based on a pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change ) (Cao Pg 16 ¶10 discloses the pixel difference between the first images) between the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image) and the pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change ) (Cao Pg 13 ¶08 discloses a pixel difference between the second images) between the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) and the fake labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶01-¶02 and discloses labeling the second combination as a fake image);
determine a feature reconstruction loss (Cao Pg 13 ¶04 and Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) based on a feature difference (Cao Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function) between the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image);
extract face features (Cao Pg 15 ¶04 discloses feature points from the images including eyes, nose, mouth, eyebrows) of the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) the first source image (Cao Pg 2 ¶04 discloses an initial facial image), the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image), the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image), and the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) to determine an identity loss (Cao Pg 13 ¶04 and Pg 14 ¶04 discloses an identity loss) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature);
perform discriminative processing (Pg 13 ¶089 discloses inputting the image into a discriminant network) on the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) to obtain an adversarial loss (Cao Pg 13 ¶04 discloses a loss function of the adversarial training generation network) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature); and
perform summation (Cao Pg 13 ¶04 discloses performing weighted summation) on the pixel reconstruction loss (Cao Pg 13 ¶03-¶04 discloses a pixel reconstruction loss), the feature reconstruction loss (Cao Pg 13 ¶04 and Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function), the identity loss (Cao Pg 13 ¶04 and Pg 14 ¶04 discloses an identity loss), and the adversarial loss (Cao Pg 13 ¶04 discloses a loss function of the adversarial training generation network) of the identity swapping model(Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature), to obtain loss information of the identity swapping model (Cao Pg 13 ¶03 discloses obtaining loss information from the model), and updating model parameters of the identity swapping model (Zhu Fig 11 1108 Pg 20 ¶01-03 discloses updating the samples to continue training the model) based on the loss information of the identity swapping model (Cao Pg 13 ¶03 discloses obtaining loss information from the model) to train (Cao Pg 2 ¶17 disclose the method for training the model) the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature). See claim 13 for rationale, its parent claim.
Regarding Claim 16, Cao in view of Zhu in view of Kim teaches the image processing apparatus according to claim 14, wherein the identity loss (Cao Pg 13 ¶04 and Pg 14 ¶04 discloses an identity loss) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) comprises a first identity loss (Cao Pg 13 ¶04 disclose performing an identity loss) and a second identity loss (Cao Pg 14 ¶04 discloses the identity loss also being calculated for the second image); and
the processing code (Cao Pg 2 ¶03 discloses a processor reads and executes the program) is further configured to cause at least one of the at least one processor (Cao Pg 2 ¶03 discloses a processor reads and executes the program):
determine the first identity loss (Cao Pg 13 ¶04 and Pg 14 ¶04 discloses an identity loss) based on a similarity between face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) the first source image (Cao Pg 2 ¶04 discloses an initial facial image), and face features (Cao Pg 13 ¶02 discloses facial identity features) of the first source image (Cao Pg 2 ¶04 discloses an initial facial image) and a similarity between face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) and face features (Cao Pg 13 ¶02 discloses facial identity features) of the second source image(Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image); and
determine the second identity loss (Cao Pg 14 ¶04 discloses the identity loss also being calculated for the second image) based on a similarity between the face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and face features (Cao Pg 13 ¶02 discloses facial identity features) of the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), a similarity between the face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the first source image (Cao Pg 2 ¶04 discloses an initial facial image) and the face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), a similarity between the face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) and face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), and a similarity between the face features (Cao Pg 13 ¶03 discloses the face similarity between the images) of the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image) and the face features of the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image). See claim 13 for rationale, its parent claim.
Regarding Claim 17, Cao in view of Zhu in view of Kim teaches the image processing apparatus according to claim 14, wherein the processing code (Cao Pg 2 ¶03 discloses a processor reads and executes the program) is further configured to cause at least one of the at least one processor (Cao Pg 2 ¶03 discloses a processor reads and executes the program) to:
obtain a discriminative model (Pg 13 ¶089 discloses inputting the image into a discriminant network);
input the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) into the discriminative model (Cao Pg 13 ¶089 discloses inputting the image into a discriminant network) and performing discriminative processing (Pg 13 ¶089 discloses inputting the image into a discriminant network) on the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) to obtain a first discriminative result (Cao Pg 12 ¶06 discloses the output being the different in pixels between first images);
input the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) into the discriminative model (Pg 13 ¶089 discloses inputting the image into a discriminant network) and performing discriminative processing (Pg 13 ¶089 discloses inputting the image into a discriminant network) on the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) to obtain a second discriminative result (Cao Pg 13 ¶08 discloses the output being the second target facial image samples being used as negative samples); and
determine the adversarial loss (Cao Pg 13 ¶04 discloses a loss function of the adversarial training discrimination network) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) based on the first discriminative result (Cao Pg 12 ¶06 discloses the output being the different in pixels between first images) and the second discriminative result (Cao Pg 13 ¶08 discloses the output being the second target facial image samples being used as negative samples). See claim 13 for rationale, its parent claim.
Regarding Claim 18, Cao teaches a non-transitory computer-readable storage medium (Cao Pg 21 ¶04 discloses the memory of the computer device includes a non-volatile storage medium and an internal memory, which is a form of non-transitory media) storing computer code (Cao Pg 21 ¶04 discloses the storage medium including a computer program) which, when executed by at least one processor (Cao Pg 21 ¶04 discloses the computer program is executed by the processor to realize an image processing/model training method), causes the at least one processor to at least:
comprising a first source image (Cao Pg 2 ¶04 discloses an initial facial image) and a fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) being based on identity swapping processing of the real labeled image (Cao Pg 2 ¶14 discloses the swapping of the target feature on the facial image so that the target facial image matches the facial identity characteristics of the initial facial image, and matches the attribute characteristics of the template facial image), the first source image (Cao Pg 2 ¶04 discloses an initial facial image) having a same identity attribute (Cao Pg 6 ¶04 disclose the initial facial image provides facial identity features), and the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), having a same non-identity attribute (Cao Pg 6 ¶04 disclose the template image providing the attribute features);
input (Cao Fig 5 discloses inputting the image into the model) the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) into an identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) and performing identity swapping processing (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) on the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) based on the first source image (Cao Pg 2 ¶04 discloses an initial facial image) to obtain a first identity swapping image of the fake template image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character);
comprising a second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image), a real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), the fake labeled image being based on identity swapping processing (Cao Pg 2 ¶14 discloses the swapping of the target feature on the facial image so that the target facial image matches the facial identity characteristics of the initial facial image, and matches the attribute characteristics of the template facial image) of the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) based on the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image), the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image) having a same identity attribute (Cao Pg 6 ¶04 disclose the initial facial image provides facial identity features), and the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) having a same non-identity attribute (Cao Pg 6 ¶04 disclose the template image providing the attribute features);
input (Cao Fig 5 discloses inputting the image into the model) the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) into the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) and performing identity swapping processing (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) on the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) based on the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image) to obtain a second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) of the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image); and
train (Cao Pg 2 ¶17 disclose the method for training the model) the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character),and the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) to generate a trained identity swapping model (Cao Pg 13 ¶08 discloses training the model) to perform identity swapping processing (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) on a target template image (Cao Pg 9 ¶04 discloses the template facial image reference) based on a target source image (Cao Pg 12 ¶05 and Pg 16 ¶02 disclose a target image) between the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image).
Cao does not explicitly disclose obtain a fake template sample group, a real labeled image, and the real labeled image, and the real labeled image, obtain a fake labeled sample group, and a fake labeled image, and the fake labeled image, and the fake labeled image, based on the fake template sample group, the fake labeled sample group, and the fake labeled image.
Zhu is in the same field of image analysis to produce synthesized images or video. Further, Zhu teaches obtain a fake template sample group (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image), a real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image), and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image), and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image), obtain a fake labeled sample group (Zhu Pg 1 ¶10-¶12 discloses a triplet sample group that includes an image with attributes including a fake image), and a fake labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶01-¶02 and discloses labeling the second combination as a fake image), and the fake labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶01-¶02 and discloses labeling the second combination as a fake image), and the fake labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶01-¶02 and discloses labeling the second combination as a fake image) based on the fake template sample group (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image), the fake labeled sample group (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image) and the fake labeled image (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Cao by including real labeled images as part of the fake template sample group and updating the parameters of the model and assigning weights to the pixel differences as taught by Zhu, to make an invention that can more accurately differentiate between the features need to complete a successful and clear swapped image; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to improve the stability and robustness of the image processing model, the image processing model obtained by the triple sample training can ensure the synthesis of area and the original image or the original video in shape, illumination, the action is consistent, thereby improving the quality of the image and a synthetic video (Zhu Pg 14 ¶06).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Cao and Zhu in combination do not explicitly disclose wherein the training is based on a pixel difference.
Kim is in the same field of image analysis to produce synthesized images or video. Further, Kim teaches wherein the training is based on a pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Cao in view of Zhu by incorporating the pixel difference into the training of the model as taught by Kim, to make an invention that can more accurately differentiate between the features need to complete a successful and clear swapped image based on the pixel difference between images; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to allow for faster training of the image generator by providing more stable gradient information (Kim Pg 2, Col 1, ¶03).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 19, Cao in view of Zhu in view of Kim teaches the non-transitory computer-readable storage medium (Cao Pg 21 ¶04 discloses the memory of the computer device includes a non-volatile storage medium and an internal memory, which is a form of non-transitory media) according to claim 18, wherein the train comprises:
determining a pixel reconstruction loss (Cao Pg 13 ¶03-¶04 discloses a pixel reconstruction loss) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) based on a pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change ) (Cao Pg 16 ¶10 discloses the pixel difference between the first images) between the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image) and the pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change ) (Cao Pg 13 ¶08 discloses a pixel difference between the second images) between the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) and the fake labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶01-¶02 and discloses labeling the second combination as a fake image);
determining a feature reconstruction loss (Cao Pg 13 ¶04 and Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) based on a feature difference (Cao Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function) between the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image);
extracting face features (Cao Pg 15 ¶04 discloses feature points from the images including eyes, nose, mouth, eyebrows) of the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) the first source image (Cao Pg 2 ¶04 discloses an initial facial image), the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image), the second source image (Cao Pg 11 ¶02 and Pg 13 ¶08 discloses a second initial facial image), and the real template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image) to determine an identity loss (Cao Pg 13 ¶04 and Pg 14 ¶04 discloses an identity loss) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature);
performing discriminative processing (Pg 13 ¶089 discloses inputting the image into a discriminant network) on the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) to obtain an adversarial loss (Cao Pg 13 ¶04 discloses a loss function of the adversarial training generation network) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature); and
performing summation (Cao Pg 13 ¶04 discloses performing weighted summation) on the pixel reconstruction loss (Cao Pg 13 ¶03-¶04 discloses a pixel reconstruction loss), the feature reconstruction loss (Cao Pg 13 ¶04 and Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function), the identity loss (Cao Pg 13 ¶04 and Pg 14 ¶04 discloses an identity loss), and the adversarial loss (Cao Pg 13 ¶04 discloses a loss function of the adversarial training generation network) of the identity swapping model(Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature), to obtain loss information of the identity swapping model (Cao Pg 13 ¶03 discloses obtaining loss information from the model), and updating model parameters of the identity swapping model (Zhu Fig 11 1108 Pg 20 ¶01-03 discloses updating the samples to continue training the model) based on the loss information of the identity swapping model (Cao Pg 13 ¶03 discloses obtaining loss information from the model) to train (Cao Pg 2 ¶17 disclose the method for training the model) the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature). See claim 18 for rationale, its parent claim.
Claims 3, 9-11, 15, and 20 are rejected under 35 U.S.C. 103 as unpatentable over Cao in view of Zhu in view of Kim in further view of Berlin et al (US Patent No 11,308,957 hereafter referred to as Berlin).
Regarding Claim 3, Cao in view of Zhu in view of Kim teaches the image processing method according to claim 2, wherein determining the feature reconstruction loss (Cao Pg 13 ¶04 and Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function) comprises:
to perform image feature extraction (Cao Pg 15 ¶04 discloses feature points from the images including eyes, nose, mouth, eyebrows) on the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character)
to perform the image feature extraction (Cao Pg 15 ¶04 discloses feature points from the images including eyes, nose, mouth, eyebrows) on the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image) to obtain a second feature extraction result (Cao Pg 15 ¶04 discloses feature points from the images including eyes, nose, mouth, eyebrows),
calculating feature differences (Cao Pg 13 ¶03 discloses calculating the facial identity feature difference)
performing summation (Cao Pg 13 ¶04 discloses performing weighted summation) of the feature differences (Cao Pg 13 ¶03 discloses calculating the facial identity feature difference) to obtain the feature reconstruction loss (Cao Pg 13 ¶04 and Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature).
Cao in view of Zhu in view of Kim does not explicitly disclose obtaining an image feature extraction network comprising a plurality of image feature extraction layers;
calling the image feature extraction network,
to obtain a first feature extraction result, the first feature extraction result comprising an identity swapping image feature extracted from each image feature extraction layer of the plurality of image feature extraction layers;
calling the image feature extraction network the second feature extraction result comprising a labeled image feature extracted from each image feature extraction layer of the plurality of image feature extraction layers;
between the identity swapping image feature and the labeled image feature that are extracted from each image feature extraction layer; and
of the image feature extraction layers.
Berlin is in the same field of image analysis to produce synthesized images or video. Further, Berlin teaches obtaining an image feature extraction network (Berlin Col 10 Lines 50-55 disclose a learning engine identifying and classifying facial features) comprising a plurality of image feature extraction layers (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers);
calling the image feature extraction network (Berlin Col 10 Lines 50-55 disclose a learning engine identifying and classifying facial features)
to obtain a first feature extraction result (Berlin Col 10 Lines 50-55 disclose a learning engine identifying and classifying facial features the result being eyes, mouth nose), the first feature extraction result comprising an identity swapping image feature (Berlin Col 10 Lines 50-60 disclose the facial features being used tin the facial swapping process) extracted from each image feature extraction layer (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers and Col 8 Lines 50-55 disclose extracting only necessary information in the encoder layer) of the plurality of image feature extraction layers (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers);
calling the image feature extraction network (Berlin Col 10 Lines 50-55 disclose a learning engine identifying and classifying facial features) the second feature extraction result comprising a labeled image (Berlin Col 10 Line 54-58 disclose automatically adds text, image, and/or graphic tags (e.g., timestamps) to the video) feature extracted from each image feature extraction layer (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers and Col 8 Lines 50-55 disclose extracting only necessary information in the encoder layer) of the plurality of image feature extraction layers (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers);
between the identity swapping image feature (Berlin Col 10 Lines 50-60 disclose the facial features being used tin the facial swapping process) and the labeled image feature (Berlin Col 10 Line 54-58 disclose automatically adds text, image, and/or graphic tags (e.g., timestamps) to the video) that are extracted from each image feature extraction layer (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers); and
of the image feature extraction layers (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Cao in view of Zhu in view of Kim by including a feature extraction network including multiple layer and calculating the mean and variance of features as taught by Berlin, to make an invention that can more accurately differentiate between the features need to complete a successful and clear swapped image; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to include techniques that may provide a more accurate model while reducing the time and computer resources needed to create the CGI model. (Berlin Col 7 Lines 40-46).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 9, Cao in view of Zhu in view of Kim teaches the image processing method according to claim 8, wherein (Cao Pg 10 ¶02-¶04 discloses a feature fusion model used to obtain target features including: the facial identity feature, attribute feature) on the identity swapping features (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the face features of the first source image (Cao Pg 8 ¶2 discloses obtaining the facial identity characteristics of the initial facial image) to obtain the encoding result (Cao Pg 8 ¶04 discloses the result of the encoding being obtaining the facial features of the first image) comprises:
of the identity swapping features (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) of the identity swapping features (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character);
of the face features (Cao Pg 8 ¶2 discloses obtaining the facial identity characteristics of the initial facial image) of the face features (Cao Pg 8 ¶2 discloses obtaining the facial identity characteristics of the initial facial image); and
performing the feature fusion processing (Cao Pg 10 ¶02-¶04 discloses a feature fusion model used to obtain target features including: the facial identity feature, attribute feature) on the identity swapping features (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the face features (Cao Pg 8 ¶2 discloses obtaining the facial identity characteristics of the initial facial image)
of the identity swapping features (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character)
of the identity swapping features (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character)
of the face features (Cao Pg 8 ¶2 discloses obtaining the facial identity characteristics of the initial of the face features (Cao Pg 8 ¶2 discloses obtaining the facial identity characteristics of the initial facial image) to obtain the encoding result (Cao Pg 8 ¶04 discloses the result of the encoding being obtaining the facial features of the first image).
Cao in view of Zhu in view of Kim does not explicitly disclose calculating a mean and a variance calculating a mean and a variance calculating a mean and a variance calculating a mean and a variance.
Berlin is in the same field of image analysis to produce synthesized images or video. Further, Berlin teaches calculating a mean (Berlin Col 14 Lines 50-55 disclose calculating the mean of the facial feature alignment) and a variance (Berlin Col 28 Lines 53-60 disclose calculating the variance) calculating a mean (Berlin Col 14 Lines 50-55 disclose calculating the mean of the facial feature alignment) and a variance (Berlin Col 28 Lines 53-60 disclose calculating the variance) calculating a mean (Berlin Col 14 Lines 50-55 disclose calculating the mean of the facial feature alignment) and a variance (Berlin Col 28 Lines 53-60 disclose calculating the variance) calculating a mean (Berlin Col 14 Lines 50-55 disclose calculating the mean of the facial feature alignment) and a variance (Berlin Col 28 Lines 53-60 disclose calculating the variance).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Cao in view of Zhu in view of Kim by including a feature extraction network including multiple layer and calculating the mean and variance of features as taught by Berlin, to make an invention that can more accurately differentiate between the features need to complete a successful and clear swapped image; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to include techniques that may provide a more accurate model while reducing the time and computer resources needed to create the CGI model. (Berlin Col 7 Lines 40-46).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 10, Cao in view of Zhu in view of Kim teaches the image processing method according to claim 1, wherein obtaining the fake template sample group (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image) comprises:
obtaining an initial source image (Zhu Pg 5 ¶2 and discloses a source image and a first image) corresponding to the first source image (Cao Pg 2 ¶04 discloses an initial facial image), and obtaining an initial labeled image (Zhu Pg 12 ¶07 discloses the first combination belonging to a non-fake image and corresponding label) corresponding to the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image);
on the initial source image (Zhu Pg 5 ¶2 and discloses a source image and a first image) corresponding to the first source image (Cao Pg 2 ¶04 discloses an initial facial image), to obtain the first source image (Cao Pg 2 ¶04 discloses an initial facial image) on the initial labeled image (Zhu Pg 12 ¶07 discloses the first combination belonging to a non-fake image and corresponding label) corresponding to the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image) , to obtain the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image);
obtaining a reference source image (Cao Pg 9 ¶04 and Pg 10 ¶01 discloses template facial image reference), and performing the identity swapping processing (Cao Pg 2 ¶14 discloses the swapping of the target feature on the facial image so that the target facial image matches the facial identity characteristics of the initial facial image, and matches the attribute characteristics of the template facial image) on the real labeled image(Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image) based on the reference source image (Cao Pg 9 ¶04 and Pg 10 ¶01 discloses template facial image reference), to obtain the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image); and
generating the fake template sample group (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image) based on the first source image(Cao Pg 2 ¶04 discloses an initial facial image), the fake template image (Cao Pg 1 ¶10-¶12 and discloses a template facial image), and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image).
Cao in view of Zhu in view of Kim does not explicitly disclose performing face region cropping, and performing the face region cropping.
Berlin is in the same field of image analysis to produce synthesized images or video. Further, Berlin teaches performing face region cropping (Berlin Col 12 Line 59-63 disclose cropping the video to remove unneeded video content ( e.g., footage that does not include a specified face)) and performing the face region cropping (Berlin Col 12 Line 59-63 disclose cropping the video to remove unneeded video content ( e.g., footage that does not include a specified face)).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Cao in view of Zhu in view of Kim by including a feature extraction network including multiple layer and calculating the mean and variance of features and performing face cropping as taught by Berlin, to make an invention that can more accurately differentiate between the features need to complete a successful and clear swapped image; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to include techniques that may provide a more accurate model while reducing the time and computer resources needed to create the CGI model. (Berlin Col 7 Lines 40-46).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 11, Cao in view of Zhu in view of Kim in further view of Berlin teaches the image processing method according to claim 10, wherein performing the face region cropping (Berlin Col 12 Line 59-63 disclose cropping the video to remove unneeded video content ( e.g., footage that does not include a specified face)) on the initial source image (Zhu Pg 5 ¶2 and discloses a source image and a first image) corresponding to the first source image (Cao Pg 2 ¶04 discloses an initial facial image) comprises:
performing face detection (Cao Pg 15 ¶08 and disclose performing face detection) on the initial source image (Zhu Pg 5 ¶2 and discloses a source image and a first image) corresponding to the first source image (Cao Pg 2 ¶04 discloses an initial facial image), to determine a face region (Cao Pg 16 ¶02- ¶04 and Pg 17 ¶09 discloses identifying a facial region) in the initial source image (Zhu Pg 5 ¶2 and discloses a source image and a first image) corresponding to the first source image (Cao Pg 2 ¶04 discloses an initial facial image);
performing, in the face region (Cao Pg 16 ¶02- ¶04 and Pg 17 ¶09 discloses identifying a facial region), face registration (Cao Pg 15 ¶08 disclose performing face registration) on the initial source image (Zhu Pg 5 ¶2 and discloses a source image and a first image) corresponding to the first source image (Cao Pg 2 ¶04 discloses an initial facial image), to determine face key points (Cao Pg 15 ¶04 disclose key points that characterize facial features) in the initial source image (Zhu Pg 5 ¶2 and discloses a source image and a first image) corresponding to the first source image (Cao Pg 2 ¶04 discloses an initial facial image); and
performing cropping processing (Berlin Col 12 Line 59-63 disclose cropping the video to remove unneeded video content ( e.g., footage that does not include a specified face)) on the initial source image (Zhu Pg 5 ¶2 and discloses a source image and a first image) corresponding to the first source image (Cao Pg 2 ¶04 discloses an initial facial image) based on the face key points (Cao Pg 15 ¶04 disclose key points that characterize facial features) , to obtain the first source image (Cao Pg 2 ¶04 discloses an initial facial image). See Claim 10 for rationale, its parent claim.
Regarding Claim 15, Cao in view of Zhu in view of Kim teaches the image processing apparatus according to claim 14, wherein the processing code (Cao Pg 2 ¶03 discloses a processor reads and executes the program) is further configured to cause at least one of the at least one processor (Cao Pg 2 ¶03 discloses a processor reads and executes the program)
to perform image feature extraction (Cao Pg 15 ¶04 discloses feature points from the images including eyes, nose, mouth, eyebrows) on the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character)
to perform the image feature extraction (Cao Pg 15 ¶04 discloses feature points from the images including eyes, nose, mouth, eyebrows) on the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image) to obtain a second feature extraction result (Cao Pg 15 ¶04 discloses feature points from the images including eyes, nose, mouth, eyebrows),
calculate feature differences (Cao Pg 13 ¶03 discloses calculating the facial identity feature difference)
perform summation (Cao Pg 13 ¶04 discloses performing weighted summation) of the feature differences (Cao Pg 13 ¶03 discloses calculating the facial identity feature difference) to obtain the feature reconstruction loss (Cao Pg 13 ¶04 and Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature).
Cao in view of Zhu in view of Kim does not explicitly disclose obtain an image feature extraction network comprising a plurality of image feature extraction layers;
calling the image feature extraction network to obtain a first feature extraction result, the first feature extraction result comprising an identity swapping image feature extracted from each image feature extraction layer of the plurality of image feature extraction layers;
calling the image feature extraction network the second feature extraction result comprising a labeled image feature extracted from each image feature extraction layer of the plurality of image feature extraction layers
between the identity swapping image feature and the labeled image feature that are extracted from each image feature extraction layer; and
of the image feature extraction layers.
Berlin is in the same field of image analysis to produce synthesized images or video. Further, Berlin teaches obtain an image feature extraction network (Berlin Col 10 Lines 50-55 disclose a learning engine identifying and classifying facial features) comprising a plurality of image feature extraction layers (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers);
calling the image feature extraction network (Berlin Col 10 Lines 50-55 disclose a learning engine identifying and classifying facial features) to obtain a first feature extraction result (Berlin Col 10 Lines 50-55 disclose a learning engine identifying and classifying facial features the result being eyes, mouth nose), the first feature extraction result comprising an identity swapping image feature (Berlin Col 10 Lines 50-60 disclose the facial features being used tin the facial swapping process) extracted from each image feature extraction layer (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers and Col 8 Lines 50-55 disclose extracting only necessary information in the encoder layer) of the plurality of image feature extraction layers (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers);
calling the image feature extraction network (Berlin Col 10 Lines 50-55 disclose a learning engine identifying and classifying facial features) the second feature extraction result comprising a labeled image (Berlin Col 10 Line 54-58 disclose automatically adds text, image, and/or graphic tags (e.g., timestamps) to the video) feature extracted from each image feature extraction layer (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers and Col 8 Lines 50-55 disclose extracting only necessary information in the encoder layer) of the plurality of image feature extraction layers (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers);
between the identity swapping image feature (Berlin Col 10 Lines 50-60 disclose the facial features being used tin the facial swapping process) and the labeled image feature (Berlin Col 10 Line 54-58 disclose automatically adds text, image, and/or graphic tags (e.g., timestamps) to the video) that are extracted from each image feature extraction layer (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers); and
of the image feature extraction layers (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Cao in view of Zhu in view of Kim by including a feature extraction network including multiple layer and calculating the mean and variance of features as taught by Berlin, to make an invention that can more accurately differentiate between the features need to complete a successful and clear swapped image; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to include techniques that may provide a more accurate model while reducing the time and computer resources needed to create the CGI model. (Berlin Col 7 Lines 40-46).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 20, Cao in view of Zhu in view of Kim teaches the non-transitory computer-readable storage medium (Cao Pg 21 ¶04 discloses the memory of the computer device includes a non-volatile storage medium and an internal memory, which is a form of non-transitory media) according to claim 19, wherein determining the feature reconstruction loss (Cao Pg 13 ¶04 and Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature) based on the feature difference (Cao Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function) between the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character) and the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image) comprises:
to perform image feature extraction (Cao Pg 15 ¶04 discloses feature points from the images including eyes, nose, mouth, eyebrows) on the first identity swapping image (Cao Fig 9 discloses how the fusion model takes facial features on a input image and fuses them with the template to create an image using the facial features of the original on a video game or movie character),
to perform the image feature extraction (Cao Pg 15 ¶04 discloses feature points from the images including eyes, nose, mouth, eyebrows) on the real labeled image (Zhu Pg 5 ¶09 and Pg 6 ¶02 and discloses labeling the first combination as a non-fake image) to obtain a second feature extraction result (Cao Pg 15 ¶04 discloses feature points from the images including eyes, nose, mouth, eyebrows),
calculating feature differences (Cao Pg 13 ¶03 discloses calculating the facial identity feature difference),
performing summation (Cao Pg 13 ¶04 discloses performing weighted summation) of the feature differences (Cao Pg 13 ¶03 discloses calculating the facial identity feature difference) to obtain the feature reconstruction loss (Cao Pg 13 ¶04 and Pg 16 ¶10 discloses the difference in face identity features used to construct the training loss function) of the identity swapping model (Cao Fig 10, 1003 and Pg 2 ¶01 discloses a fusion model that fuses the facial identity feature and the attribute feature to obtain the target feature)
wherein the training is based on a pixel difference (Kim Pg 1 Col ¶03 and Pg 2 Col 2 ¶07, and Pg 4 Col 2 ¶04 discloses training the swapping model using a loss function including pixel level change) between the second identity swapping image (Cao Pg 13 ¶08 discloses a second output image) and the fake labeled image (Zhu Pg 1 ¶10-¶12 discloses a triplet sample that includes an image with attributes including a fake image).
Cao in view of Zhu in view of Kim does not explicitly disclose obtaining an image feature extraction network comprising a plurality of image feature extraction layers;
calling the image feature extraction network to obtain a first feature extraction result, the first feature extraction result comprising an identity swapping image feature extracted from each image feature extraction layer of the plurality of image feature extraction layers,
calling the image feature extraction network the second feature extraction result comprising a labeled image feature extracted from each image feature extraction layer of the plurality of image feature extraction layers;
between the identity swapping image feature and the labeled image feature that are extracted from each image feature extraction layer; and
of the image feature extraction layers.
Berlin is in the same field of image analysis to produce synthesized images or video. Further, Berlin teaches obtaining an image feature extraction network (Berlin Col 10 Lines 50-55 disclose a learning engine identifying and classifying facial features) comprising a plurality of image feature extraction layers (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers);
calling the image feature extraction network (Berlin Col 10 Lines 50-55 disclose a learning engine identifying and classifying facial features) to obtain a first feature extraction result (Berlin Col 10 Lines 50-55 disclose a learning engine identifying and classifying facial features the result being eyes, mouth nose), the first feature extraction result comprising an identity swapping image feature (Berlin Col 10 Lines 50-60 disclose the facial features being used tin the facial swapping process) extracted from each image feature extraction layer (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers and Col 8 Lines 50-55 disclose extracting only necessary information in the encoder layer) of the plurality of image feature extraction layers (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers);
calling the image feature extraction network (Berlin Col 10 Lines 50-55 disclose a learning engine identifying and classifying facial features) the second feature extraction result comprising a labeled image (Berlin Col 10 Line 54-58 disclose automatically adds text, image, and/or graphic tags (e.g., timestamps) to the video) feature extracted from each image feature extraction layer (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers and Col 8 Lines 50-55 disclose extracting only necessary information in the encoder layer) of the plurality of image feature extraction layers (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers);
between the identity swapping image feature (Berlin Col 10 Lines 50-60 disclose the facial features being used tin the facial swapping process) and the labeled image feature (Berlin Col 10 Line 54-58 disclose automatically adds text, image, and/or graphic tags (e.g., timestamps) to the video) that are extracted from each image feature extraction layer (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers); and
of the image feature extraction layers (Berlin Col 16 Lines 1-2 disclose this engine may have one or more hidden layers).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Cao in view of Zhu in view of Kim by including a feature extraction network including multiple layer and calculating the mean and variance of features as taught by Berlin, to make an invention that can more accurately differentiate between the features need to complete a successful and clear swapped image; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to include techniques that may provide a more accurate model while reducing the time and computer resources needed to create the CGI model. (Berlin Col 7 Lines 40-46).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RACHEL ROBERTS whose telephone number is (571)272-6413. The examiner can normally be reached Monday- Friday 7:30am- 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached on (313) 446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RACHEL L ROBERTS/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights