DETAILED ACTION
Notice of Pre-AIA or AIA Status.
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
2. A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 10/15/2025 has been entered.
3. In the applicant’s submission, claims, 1, 11-13, and 21 were amended; claims 9-10, and 21-22 were cancelled. Accordingly, claims 1-8, 11-20, and 23 are pending and being examined. Claims 1, 12, and 13 are independent form.
Claim Rejections - 35 USC § 103
4. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
5. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
6. Claims 1-8, 11-20, and 23 are rejected under 35 U.S.C. 103, as being unpatentable over Natsume et al (“FSNet: An Identity-Aware Generative Model for Image-based Face Swapping”, 2018, hereinafter “Natsume”) in view of Nirkin et al (“On Face Segmentation, Face Swapping, and Face Perception”, 2018, hereinafter “Nirkin”).
Regarding claim 1, Natsume discloses a system for facilitating real time face swapping of a user, said system comprising: one or more processors operatively coupled to a plurality of user computing devices, said one or more processors comprising a memory, said memory storing instructions which when executed by the one or more processors causes the system (the face swapping system based on the FSnet; see fig.2) to:
receive a first set of data packets from the plurality of computing devices, the first set of data packets pertaining to a video stream of the user (see “Datasets” in Sec, 3.4. para.1: “The dataset for training FSNet includes four types of images.” For instance, see the datasets shown in fig.1 and the datasets from the same person shown in fig.6), the video stream comprising one or more source facial features of the user (see the users’ source face xs in fig.2 (c), i.e., one of the source faces shown in the top row of fig.6.);
receive a set of potential target facial features associated with the user from a knowledgebase associated with a centralized server (see the users’ target face xt in fig.2 (c), i.e., the corresponding target face shown in the left col. of fig.6);
extract a first set of attributes from the first set of data packets, the first set of attributes pertaining to the “Enc-dec network” extracts “the latent variables
z
θ
-
f
” from the user’s source face xs; see fig.2 (c) and pg.5, the 2nd paragraph.);
based on the extracted first set of attributes, optimize, through a face reconstruction module, the one or more source facial features of the user such that the one or more source facial features match the set of potential target facial features of the user and generate an optimized one or more facial features of the user (the generator network receives the latent variables and generates the output face image
x
s
t
'
=
G
(
x
t
-
f
-
,
z
s
-
f
,
z
t
-
f
)
; see Sec. 3.2. It should be noticed that
x
s
t
'
is reconstructed to match the target face xt based on the source variables
z
s
-
f
and the target variables
z
t
-
f
as shown fig.2(b));
color code the optimized one or more source facial features, using a Guided Generative Adversarial Network (GAN) module, based on the set of potential target facial features of the user; swap, using the GAN module, the color coded one or more facial features with the one or more source facial features to generate an accurate image of the user (the “generator network” shown in fig.2(c) receives the latent variables
z
θ
-
f
,
z
t
-
l
and the facial image
x
t
-
f
-
'
associated with the target face xt, and reconstructs the output face
x
s
t
'
in which the source face is replaced by the target face. Also see fig.6, where the face in the 2nd row and 2nd col is “the accurate image of the user” reconstructed by the “generator network” through swapping the source face located in the top row and the 2nd col. with the target face located in the 2nd row and the 1st col.) and
generate, using a machine learning (ML) model, a trained model configured to process the accurate image of the user to identify and verify the user in real time (see Sec. 3.3 “Training”, where the reconstruction image
x
θ
'
is generated by minimizing the reconstruction error/loss defined by
L
θ
r
e
c
and the identity loss defined by
L
{
s
1
,
s
2
,
t
}
i
d
in the process for training FSNet);
predict, by a ML engine, from a plurality of services received by the system, an information service associated with the swapped accurate image of the user; facilitate, by the ML engine, a response corresponding to the information service to the user based on the trained model; and auto-generate, by the ML engine, the response by the system to the user (as an example shown by fig.2, the trained face swapping system generates the swapped face (or, facilitates the face swapping)
x
s
t
'
for the user based on the user’s source face xs and the user’s target face xt received from the user. Also see figs.4-6, where each of which shows the generating face swapping results that the trained proposed network was applied to.).
Natsume does not explicitly disclose extracting a set of attributes from occluded faces as recited in the claim. However, in the same field of endeavor, that is, in the field of face swapping, Nirkin teaches the “occlusion augmentation” process which riches the dada collection by adding synthetic occlusions to the training dataset. See pg.101, the section of “occlusion augmentation” of Nirkin. It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was made to incorporate the teachings of Nirkin into the teachings of Natsume and add occlude face images to the training dataset taught by Natsume. Suggestion or motivation for doing so would have been to swap face “under unprecedented conditions” as taught by Nirkin, cf., Abstract. Therefore, the claim is unpatentable over Natsume in view of Nirkin.
Regarding claim 2, 14, the combination of Natsume and Nirkin discloses, wherein the system is further configured to align, by using a Delaunay Triangulation module, the accurate image of the user according to alignment of the set of potential target facial features of the user (Natsume, see pg.11, par.4: “Next, the face region of the source image in the mask is copy and pasted to the target image such that the two eye locations are aligned. Finally, the entire image appearance is repaired by being fed to each network.”).
Regarding claim 3, 15, the combination of Natsume and Nirkin discloses, wherein the system is further configured to convolve, by using a Pyramid Blending module, the optimized one or more facial encoding with occlusion encoding using a mask from a segmentation network module to generate a final swapped accurate image of the user (Natsume, see lateral two pyramid structures shown fig.2(b)).
Regarding claim 4, 16, the combination of Natsume and Nirkin discloses, wherein the system is further configured to preserve, by using a transfer network module, a set of finer feature details of the final swapped accurate image of the user (Natsume, as shown in fig.2(b), wherein the finer 1st layer is directly connected with the final layer).
Regarding claim 5, 17, the combination of Natsume and Nirkin discloses, wherein the system is further configured to generate, using a Hessian aided error compensation module, one or more skin regions occluded due to the one or more occlusions in the one or more facial features of the user (this feature is obvious for one of ordinary skill in the art because the Hessian error compensation is widely used in the field of object segmentation in images.).
Regarding claim 6, 18, the combination of Natsume and Nirkin discloses, wherein the system is further configured to detect the one or more source facial features using one or more face detection devices such as scanning and extraction camera sensor (this feature is obvious for one of ordinary skill in the art because face detection technologies are widely used in the field of cameras for obtaining vides. See, Nirkin, pg.101, 1st par.: “These videos portray faces of different poses, ethnicities and ages, viewed under widely varying conditions. We used 1,275 videos of subjects not included in LFW, of the 2,042 CS2 videos (309 subjects out of 500).”).
Regarding claim 7, 19, the combination of Natsume and Nirkin discloses, wherein the video stream of the user comprises a plurality of variations and diverse face profiles of the user (See, Nirkin, pg.101, 1st par.: “These videos portray faces of different poses, ethnicities and ages, viewed under widely varying conditions. We used 1,275 videos of subjects not included in LFW, of the 2,042 CS2 videos (309 subjects out of 500).”).
Regarding claim 8, 20, the combination of Natsume and Nirkin discloses, wherein the plurality of variations and diverse face profiles of the user includes a plurality of profiles such as left, right, front and back (see Natsume, a variety of face profiles shown in figs.4-6).
Regarding claim 11, 23, the combination of Natsume and Nirkin discloses, wherein the system is further configured to: store, based on a consent of the user, the one or more source facial features of the user; store based on the one or more face detection devices available in the user computing device associated with the user (this feature is obvious for one of ordinary skill in the art without any technique efforts.).
Regarding claims 12 and 13, each of which is an inherent variation of claim 1, thus it is interpreted and rejected for the reasons set forth in the rejection of claim 1.
Response to Arguments
7. Applicant’s arguments, with respects to claim 1, filed on 09/15/2025, have been fully considered but they are not persuasive.
On page 10 of applicant’s response, Applicant submits:
[7-1.] However, Natsume's ML model is trained solely to produce face-swapped images and preserve visual similarity during swapping. Natsume'sML model is not trained or configured to process an accurate image of a user for real-time identification or verification. The "identity loss" described in Natsume is merely applied to improve the realism and stability of generated swapped images, but not to authenticate and verify the true identity of a user. Therefore, Natsume does not disclose or suggest a system that identifies and verifies a user in real time, as required by amended independent claim 1.
[7-2.] Also, Natsume is directed exclusively to face swapping using FSNet, where encoder- decoder and generator networks are trained to synthesize a face-swapped image by combining latent variables for facial appearance with non-face regions of another image. The output of FSNet is limited to the generation of a synthetic face image. Natsume does not disclose or suggest predicting any information service from a plurality of services. Also, Natsume does not disclose or suggest facilitating a user-facing response, or auto-generating a response to the user. Natsume terminates at image manipulation and does not extend into service prediction or automated response generation as required by amended independent claim 1.
(The emphases added by applicant.)
Regarding [7-1], The examiner respectfully disagrees with the arguments. It is because Natsume clearly discloses that the FSNet is “an identify-aware generative model for image-based face swapping”, see the Title. Specifically,
Sec. I, item 1, on page 3, Natsume states: “While face swapping, it well preserves both the face identity in a source image and the appearances of hairstyle and background region in a target image.”
Sec. 3.3., para.1, Natsume states “the proposed network is also trained with an identity loss to preserve the face identities in the face swapping results. We define the identity loss using the triplet loss.”
On page 7, the last paragraph, Natsume states, the FSNet is trained by the “identify loss” and “the CelebA dataset, which we used in the experiments, identity labels are assigned to all the images.”
For at least the reasons found above, the reconstruction image
x
θ
'
is generated by FSNet and is “an accurate image of a user to identify and verify the user in real time” because
x
θ
'
is reconstructed by minimizing the identity loss defined by
L
{
s
1
,
s
2
,
t
}
i
d
in the process for training the FSNet. In other words, the reconstructed face swapping
x
θ
'
“well preserves both the face identity in a source image and the appearances of hairstyle and background region in a target image.” The argument therefore is unpersuasive.
Regarding [7-2], The examiner respectfully disagrees with the arguments. As explained in the rejections of the claims, the trained face swapping system facilitates and auto-generates the face swapping
x
s
t
'
for a user based on the user’s source face xs and the user’s target face xt. The argument therefore is unpersuasive.
Conclusion
8. Any inquiry concerning this communication or earlier communications from the examiner should be directed to RUIPING LI whose telephone number is (571)270-3376. The examiner can normally be reached 8:30am--5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, HENOK SHIFERAW can be reached on (571)272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit https://patentcenter.uspto.gov; https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center, and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RUIPING LI/Primary Examiner, Ph.D., Art Unit 2676