DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Remarks
This office action is responsive to the amendment filed on 03/11/2026.
Claim(s) 1-20 is/are pending in the application.
Independent claim(s) 1, 12, 20 was/were amended.
Response to Arguments
Applicant's argument(s), regarding the amended portion(s) as recited in independent claim 1 (and similarly in independent claim(s) 12, 20), filed 03/11/2026, have/has been fully considered and is/are not persuasive. Upon further consideration, Examiner still views that the prior art(s) of Kwatra, used in the previous rejection of claim(s) 1, 12, 20, can be relied upon for the aforementioned amended portion(s). To note, applicant's amendment necessitated the updated ground(s) of rejection presented in this office action.
In response to applicant’s arguments, as noted above, arguments are not persuasive. As shown previously and in the updated rejection below, Kwatra discloses in paragraphs [0088]-[0092] and corresponding Fig.7A the training of a face geometry prediction model. In step 702, a training video is obtained that includes visual data that shows a speaker and audio data that includes speech of the speaker. In step 704, 3D landmarks are detected using the visual data of the speaker. This shows that landmarks are detected in a video frame. In step 706, face geometry is predicted based on the audio data. This is viewed as corresponding to the fitting of a first 3D geometry associated with the video frame (as the audio data is from the video), which would correspond to the plurality of landmarks. In step 708 and 710, a loss term is evaluated that compares the face geometry predicted by the machine-learned face geometry model with the 3D face features generated by the 3D face landmark detector, and based on the loss term, modify one or more values of one or more parameters of the machine-learned face geometry prediction model. This shows that the first 3D geometry is modified based on a loss function, resulting in a second 3D geometry. While applicant argues, that to teach or suggest the amended claim, Kwatra would have to disclose the idea of using the machine learning model to generate the predicted facial geometry for comparison with the facial features detected in the training image, and would also have to disclose modifying the parameters of the machine learning model based on the comparison to generate additional predicted facial geometry associated with the same training image. However, examiner disagrees. Kwatra is still viewed to detect landmarks, fit a first 3D geometry, and modify the first 3D geometry based on a loss function to generate a second 3D geometry. The amended portions “associated with the video frame” is not viewed to add anything significant, or at the very least, Kwatra is still viewed to show this, as the first geometry is determined using the audio data of the video frames, and even though it is then modified to generate a second geometry, it is still based on the audio data of the video.
Claim Rejections - 35 USC § 102
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1, 4-5, 9-12, 15, 19-20 is/are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Kwatra et al. (US 2023/0343010 A1).
In regards to claim 1, Kwatra teaches a computer-implemented method for tracking faces within video frames, the method comprising:
detecting a plurality of landmarks on a face included in a video frame (e.g. [0040]: for detecting faces in the training video, and obtaining 3D facial features, a face landmark detector can be used; various facial landmark detectors (also known as three-dimensional face detectors) are known and available in the art; see also [0088]-[0089],Fig.7A: step 702, a computing system can obtain a training video that includes visual data and audio data where the visual data depicts a speaker and the audio data includes speech uttered by the speaker; step 704, the computing system can apply a three-dimensional face landmark detector to the visual data to obtain three-dimensional facial features associated with a face of the speaker);
performing one or more operations to fit a first 3D geometry associated with the video frame to the plurality of landmarks (e.g. as above, [0088]-[0089],Fig.7A: training video that includes visual data and audio data; obtain three-dimensional facial features; see also [0090],Fig.7A: step 706, the computing system can predict, using a machine-learned face geometry prediction model, a face geometry based at least in part on data descriptive of the audio data; Examiner’s note: the predicting is viewed as corresponding to the claimed “fitting”, and the face geometry is viewed as corresponding to the claimed “first 3D geometry”; step 704 shows face geometry is based on detected landmarks; the audio data is from the video, therefore the predicted 3D facial features are associated with the video frame), wherein the first 3D geometry is defined using one or more parameters (e.g. as above, [0090],Fig.7A: step 706: face geometry; see also [0030]: a face geometry prediction model can predict face geometry, e.g. which can be expressed as mesh vertex deformations versus a reference mesh; Examiner’s note: shows parameters include mesh vertex positions and deformations); and
performing one or more operations to modify the one or more parameters based on the video frame and a first loss function to generate a second 3D geometry associated with the video frame (e.g. [0091]-[0092],Fig.7A: step 708, the computing system can evaluate a loss term that compares the face geometry predicted by the machine-learned face geometry model with the three-dimensional face features generated by the three-dimensional face landmark detector; step 710, the computing system can modify one or more values of one or more parameters of the machine-learned face geometry prediction model based at least in part on the loss term; Examiner’s note: the machine-learned face geometry with one or more modified values/parameters is viewed to correspond to the claimed “second 3D geometry”; step 710 is also viewed to be based on the video frame as shown in Fig.7A in step 702, where a training video (including video frames) is obtained and used to build the face model).
In regards to media claim 12 and system claim 20, claim(s) 12, 20 recite(s) limitations that is/are similar in scope to the limitations recited in claim 1. Therefore, claim(s) 12, 20 is/are subject to rejections under the same rationale as applied hereinabove for claim 1. To note, Kwatra discloses the use of a system, comprising a processor and a memory, such as computer-readable media (see paragraph [0113]).
In regards to claim 4, Kwatra teaches a method, wherein the second 3D geometry comprises a plurality of vertices, and further comprising performing one or more operations to modify one or more positions of one or more vertices included in the plurality of vertices based on the video frame and a second loss function to generate a third 3D geometry (e.g. as above, [0091]-[0092],Fig.7A: step 708 … evaluate a loss term; step 710 … modify one or more values of one or more parameters of the machine-learned face geometry prediction model based at least in part on the loss term; Examiner’s note: in the 1st column of Fig.2, it shows that the person changes their facial expression over time in the video frames; the 2nd column in Fig.2 shows vertices being updated in response to these changing facial expressions over time; thus, the video frames will perform operations to modify positions of the vertices based on a first video frame and other additional video frames as the 3D face mesh model is updated to reflect changes in the user’s facial expression and movement over time; therefore, steps 708/710 in Fig.7A, as the loss is repeatedly calculated, this results in a second loss function to generate a third 3D geometry (where the third 3D geometry is an updated facial expression or movement that occurs over time); see also [0107]: example implementations can automatically translate and lip-sync existing videos into different languages; Examiner’s note: this further shows that the system goes through the whole video (multiple frames)).
In regards to media claim 15, claim(s) 15 recite(s) limitations that is/are similar in scope to the limitations recited in claim 4. Therefore, claim(s) 15 is/are subject to rejections under the same rationale as applied hereinabove for claim 4.
In regards to claim 5, Kwatra teaches a method, wherein the second loss function includes at least one term included in the first loss function (e.g. as above, [0091]-[0092],Fig.7A: step 708 … evaluate a loss term; step 710 … modify one or more values of one or more parameters of the machine-learned face geometry prediction model based at least in part on the loss term; Examiner’s note: as mentioned above for claim 4, the second loss function corresponds to repeated use of the loss function in step 708/710 of Fig.7A over at least a second video frame due to an updated facial change in the user who appears in the plurality of video frames; thus, the second loss function would have terms similar to what was used to calculate the first loss function because it is the same loss function being run repeatedly; for example, in the 2nd column of Fig.2, it shows normalized vertices for the updated face model being calculated for each video frame where each row corresponds to a given video frame).
In regards to claim 9, Kwatra teaches a method, wherein the second loss function penalizes a difference between the video frame and another video frame that has been rendered using the second 3D geometry (e.g. [0099],Fig.7B: step 760, the computing system can evaluate a loss term that compares the face texture predicted by the machine-learned face texture model with the training face texture; Examiner’s note: shows second loss function penalizes a difference between the video frame and another video frame that has been rendered using the at least one of a texture map).
In regards to media claim 19, claim(s) 19 recite(s) limitations that is/are similar in scope to the limitations recited in claim 9. Therefore, claim(s) 19 is/are subject to rejections under the same rationale as applied hereinabove for claim 9.
In regards to claim 10, Kwatra teaches a method, further comprising performing one or more operations to generate a dubbed media content item based on the third 3D geometry (e.g. [0107]: video translation and dubbing: even though certain example models used for experimentation were trained mostly on English videos, it turns out empirically that they are surprisingly robust to both different languages as well as TTS audio at inference time; using available transcripts or a speech recognition system to obtain captions, and subsequently a text-to-speech system to generate audio, example implementations can automatically translate and lip-sync existing videos into different languages; in conjunction with appropriate video re-timing and voice-cloning, the resulting videos look fairly convincing).
In regards to claim 11, Kwatra teaches a method, further comprising performing one or more operations to generate at least one of a texture map or a lighting map based on the face included in the video frame and a second loss function that penalizes a difference between the video frame and another video frame that has been rendered using the at least one of a texture map or a lighting map (e.g. as above, [0099],Fig.7B: step 760, the computing system can evaluate a loss term that compares the face texture predicted by the machine-learned face texture model with the training face texture; Examiner’s note: shows second loss function penalizes a difference between the video frame and another video frame that has been rendered using the at least one of a texture map).
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 2-3, 8, 13-14, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kwatra as applied to claims 1, 12 above, and further in view of Cao et al. (US 2022/0358719 A1).
In regards to claim 2, Kwatra alone does not explicitly teach the claimed limitations.
However, the combination of Kwatra and Cao teaches a method, wherein the first loss function penalizes a difference between a mapping of the first 3D geometry to a canonical space and one or more other mappings of one or more other 3D geometries associated with the face in one or more other video frames to the canonical space (e.g. Kwatra as above, [0091]-[0092],Fig.7A: step 708 … evaluate a loss term (first loss function); -- Kwatra is silent about the loss term penalizing a difference between a mapping of the first 3D geometry to a canonical space; Cao, [0055]: to provide a good initialization for the optimization, for each input image Iv, in some embodiments, facial animation model 700 detects ninety six (96) two-dimensional (2D) face landmarks {Lvk}, that correspond to face features such as mouth corner, nose tip, and/or face contour; for each landmark, k, facial animation model 700 finds the corresponding vertex index on the face mesh denoted as lk, and calculates the L2 distance between 2D face landmark and its corresponding mesh vertex projection).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings/combination of Kwatra to penalize a difference using a loss function, in the same conventional manner as taught by Cao as both deal with generating 3D facial meshes. The motivation to combine the two would be that it would help generate a face mesh that closely resembles the landmarks detected in the one or more video frames.
In regards to media claim 13, claim(s) 13 recite(s) limitations that is/are similar in scope to the limitations recited in claim 2. Therefore, claim(s) 13 is/are subject to rejections under the same rationale as applied hereinabove for claim 2.
In regards to claim 3, Kwatra alone does not explicitly teach the claimed limitations.
However, the combination of Kwatra and Cao teaches a method, wherein the first loss function penalizes one or more differences between one or more landmarks on lips of the face included in the video frame and one or more corresponding landmarks on lips associated with the first 3D geometry (e.g. Kwatra as above, [0091]-[0092],Fig.7A: step 708 … evaluate a loss term (first loss function); -- Kwatra is silent about the loss term penalizing a difference between one or more landmarks on lips of the face included in the video frame and one or more corresponding landmarks on lips associated with the first 3D geometry; Cao as above, [0055]: to provide a good initialization for the optimization, for each input image Iv, in some embodiments, facial animation model 700 detects ninety six (96) two-dimensional (2D) face landmarks {Lvk}, that correspond to face features such as mouth corner, nose tip, and/or face contour; for each landmark, k, facial animation model 700 finds the corresponding vertex index on the face mesh denoted as lk, and calculates the L2 distance between 2D face landmark and its corresponding mesh vertex projection; Examiner’s note: here, the landmarks and corresponding vertices that are located on the mouth corners are landmarks/vertices on the lips of the face in the video).
In addition, the same rationale/motivation of claim 2 is used for claim 3.
In regards to media claim 14, claim(s) 14 recite(s) limitations that is/are similar in scope to the limitations recited in claim 3. Therefore, claim(s) 14 is/are subject to rejections under the same rationale as applied hereinabove for claim 3.
In regards to claim 8, Kwatra alone does not explicitly teach the claimed limitations.
However, the combination of Kwatra and Cao teaches a method, wherein the second loss function penalizes one or more differences between the plurality of landmarks on the face included in the video frame and a plurality of corresponding landmarks associated with the second 3D geometry (e.g. Kwatra as above, [0091]-[0092],Fig.7A: step 708 … evaluate a loss term; step 710 … modify one or more values of one or more parameters of the machine-learned face geometry prediction model based at least in part on the loss term; Examiner’s note: as mentioned above for claim 1, the machine-learned face geometry with one or more modified values/parameters is viewed to correspond to the claimed “second 3D geometry”; as mentioned above for claim 4, the second loss function corresponds to repeated use of the loss function in step 708/710 of Fig.7A over at least a second video frame due to an updated facial change in the user who appears in the plurality of video frames; -- Kwatra is silent about the loss term penalizing a difference between the plurality of landmarks on the face included in the video frame and a plurality of corresponding landmarks; Cao as above, [0055]: to provide a good initialization for the optimization, for each input image Iv, in some embodiments, facial animation model 700 detects ninety six (96) two-dimensional (2D) face landmarks {Lvk}, that correspond to face features such as mouth corner, nose tip, and/or face contour; for each landmark, k, facial animation model 700 finds the corresponding vertex index on the face mesh denoted as lk, and calculates the L2 distance between 2D face landmark and its corresponding mesh vertex projection).
In addition, the same rationale/motivation of claim 2 is used for claim 8.
In regards to media claim 18, claim(s) 18 recite(s) limitations that is/are similar in scope to the limitations recited in claim 8. Therefore, claim(s) 18 is/are subject to rejections under the same rationale as applied hereinabove for claim 8.
Allowable Subject Matter
Claim(s) 6-7, 16-17 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Claim(s) 6-7, 16-17 was/were carefully reviewed and a search with regards to independent claim(s) 1, 12 and respective intervening claim(s) 4, 15 has been made. Accordingly, those claim(s) are believed to be distinct from the prior art searched.
Regarding claim(s) 6, 16 (and specifically independent claim(s) 1, 12), the prior art search was found to neither anticipate nor suggest the method of claim 4/media of claim 15, wherein the second loss function penalizes one or more differences between one or more landmarks on teeth of the face included in the video frame and one or more corresponding landmarks on teeth associated with the second 3D geometry (emphasis added).
Regarding claim(s) 7, 17 (and specifically independent claim(s) 1, 12), the prior art search was found to neither anticipate nor suggest the method of claim 4/media of claim 15, wherein the second loss function penalizes a difference between a degree to which a mouth associated with the second 3D geometry is closed and a degree to which a detected mouth associated with the face included in the video frame is closed (emphasis added).
It is viewed that any of the previously cited references or any of the prior art searched, in part or in whole, cannot be combined in such a way to render the claimed invention obvious.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JED-JUSTIN IMPERIAL whose telephone number is (571)270-5807. The examiner can normally be reached Monday to Friday, 9am - 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JED-JUSTIN IMPERIAL/Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights