Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claims 1, 6, 8, 13, 15, 20 and newly added 21 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant’s arguments with respect to claims 7 and 14 have been considered but are not persuasive. The added limitation “while the user is engaging in a conversation” does not distinguish over the proposed combination of Pandey, Steinhoff, Thies and Sipolins. Sipolins discloses augmented reality display and presenting feedback instructions. The system explicitly happens in virtual/augmented reality and teaches overlaying/superimposing visual markers on a real-time image of the face. Pandey explicitly discloses speech training with repeated utterances and feedback from a teacher/reference speaker (i.e., dialog or conversation based). Steinhoff explicitly discloses interview practice where the user responds to prompts and receives analyzed feedback tied to the recorded speech. Accordingly, presenting feedback instructions in augmented reality while engaging in a conversation is fully taught.
Claim Rejections - 35 USC § 103
2. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
3. Claims 1, 5-6, 8, 12-13, 15 and 19-21 are rejected under 35 U.S.C. 103 as being unpatentable over Pandey (WO 2023/007509), in view of Steinhoff (US 2019/0311331) and further in view of Thies et al. “FACE2FACE: Real-time Face Capture and Reenactment of RGB Videos” herein Thies.
Regarding Claim 1:
Pandey discloses a computer-implemented method comprising:
receiving a data sample of a user speaking one or more words, wherein the data sample includes video data and audio data of the user speaking (Pandey: p[0014] processes the audiovisual signals to provide time-scaled articulatory feedback);
analyzing the data sample to determine a correct articulation of a mouth when speaking the one or more words (Pandey: p[0023] Analyzes the video and audio data to assess articulatory efforts and phonetic correctness providing detailed visual feedback);
determining one or more feedback instructions (Pandey: p[0023] Analyzes the video and audio data to assess articulatory efforts and phonetic correctness providing detailed visual feedback)
generating a synthetic video of the user performing the correct articulation (Pandey: p[0038] and Fig. 4 discloses generating animations of a man, woman, boy or girl which are designed to provide visual cues on correct speech articulation. The customization options indicate that a synthetic visualization of correct articulation is generated);
presenting the synthetic video of the user to the user (Pandey: p[0038] and Fig. 4 explicitly states that graphical controls and animations are presented to the user as feedback);
and presenting the one or more feedback instructions and a live video of the user to the user while the synthetic video is presented (Pandey: Fig. 4 and p[0039] discloses video peripherals that capture live video of the user alongside the synthetic video that include feedback),
Pandey does not explicitly disclose but Steinhoff discloses analyzing the data sample to detect a tone, a volume, or a cadence of speech of the user (Steinhoff: ¶[0062] discloses processing the audio component of the recording and specifically that the audio can be analyzed for pitch, tone, quality and cadence; ¶[0045] discloses tone analysis as part of feedback);
determining one or more feedback instructions based on one or more conditional rules related to the tone, the volume, or the cadence (Steinhoff: ¶[0029]-[0038] discloses time-synced analysis/feedback that is displayed as overlays/pop-ups coordinated to the time of the triggering event, i.e., feedback instructions);
Pandey and Steinhoff are in the same general field of endeavor (computer-implemented speech interview training systems that record user audio/video, analyze the recording and present time-synced feedback. Both also address the closely related problem of improving a user’s spoken performance through automated visual feedback. Pandey provides the core speech-training pipeline with audiovisual capture and presentation of articulatory/voice related indicators, while Steinhoff teaches richer analysis outputs and presenting those outputs as feedback overlaid in coordination with the user’s recording. Steinhoff’s tone/volume/cadence analysis and feedback overlay approach with Pandey’s speech would have been an obvious improvement to Pandey’s feedback usefulness. The suggestion/motivation for doing so is “ the results of the video quality analysis 241 can provide feedback on how to improve the video quality analysis” as disclosed in ¶[0035] of Steinhoff.
Pandey does not disclose wherein a mouth articulation of the user in the live video is arranged to superimpose the correct articulation in the synthetic video. However, Thies discloses wherein a mouth articulation of the user in the live video is arranged to superimpose the correct articulation in the synthetic video (Thies: Abstract, and Introduction, discloses a real-time face, and overlay of corrected or target mouth movements from a source onto the target, the visual output is synchronized and photo-realistic, with the mouth modified to match target speech).
It would have been obvious to one of ordinary skill in the art to modify the system of Pandey, which provides audio visual feedback using animated faces to assist speech training, by incorporating the real-time overlay technique in Thies to superimpose synthetic mouth movements over a live video of the user’s face. While Thies uses a source speaker to drive the target face, one of ordinary skill would have understood that the same technique could be adapted to allow the system to overlay a corrected version of the user’s mouth articulation onto the user’s own live video feed. Doing so would enhance Pandey’s goal of providing visual speech feedback, allowing the users to re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination as disclosed by Thies abstract.
Regarding Claim 5:
The proposed combination of Pandey, Steinhoff and Thies further disclose the computer-implemented method of claim 1, further comprising presenting a visual articulation prompt when presenting the live video of the user (Pandey: p[0022-0024], Fig. 4 discloses providing visual display of the user’s articulatory efforts through vocal-tract animations and video of the user’s face and p[0028-0034] discloses real-time feedback by displaying animations of vocal tract movements aligned with the user’s speech).
Regarding Claim 6:
The proposed combination of Pandey and Thies further disclose the computer-implemented method of claim 1, wherein the one or more feedback instructions includes one or more of: a tone adjustment, a volume adjustment, and a cadence adjustment (Pandey: p[0028-0029] discloses pitch, level (volume) and cadence (timing and rhythm)), and wherein the one or more conditional rules include one or more of:
identifying the tone adjustment based on a determination of whether a tone parameter related to the tone violates a first threshold value,
identifying the volume adjustment based on a determination of whether a volume parameter related to the volume violates a second threshold value,
and identifying the cadence adjustment based on a determination of whether a cadence parameter related to the cadence violates a third threshold value (Steinhoff: ¶[0051], ¶[0061] discloses a cadence parameter, explicitly teaches audio can be analyzed for cadence, the pause threshold is directly cadence/speaking rate related because pauses are a cadence component and its explicitly contains a threshold).
Regarding Claim 8:
Claim 8 has been analyzed with regard to claim 1 (see rejection above) and
is rejected for the same reasons of obviousness used above.
It is noted Pandey discloses a computer system comprising one or more processors, and one or more computer readable storage media storing instructions at least at ¶[0043].
Regarding Claim 12:
Claim 12 has been analyzed with regard to claim 5 (see rejection above) and
is rejected for the same reasons of obviousness used above.
Regarding Claim 13:
Claim 13 has been analyzed with regard to claim 6 (see rejection above) and
is rejected for the same reasons of obviousness used above.
Regarding Claim 15:
Claim 15 has been analyzed with regard to claim 1 and (see rejection above) and
is rejected for the same reasons of obviousness used above.
It is noted that Pandey discloses a non-transitory computer readable medium at least at ¶[0043].
Regarding Claim 19:
Claim 19 has been analyzed with regard to claim 5 (see rejection above) and
is rejected for the same reasons of obviousness used above.
Regarding Claim 20:
Claim 20 has been analyzed with regard to claim 6 (see rejection above) and
is rejected for the same reasons of obviousness used above.
Regarding Claim 21:
The proposed combination of Pandey, Steinhoff and Thies further discloses the computer-implemented method of claim 1, further comprising:
determining at least one additional feedback instruction to correct the mouth articulation of the user (Pandey: ¶[0023]-[0024] discloses generating and displaying visual and articulatory feedback to correct mispronunciation), wherein the at least one additional feedback instruction includes one or more adjustments to a mouth shape of the user (Pandey: ¶[0023], ¶[0034]-[0035] discloses displaying the place of articulation, the vocal tract animation and lip movements to guide the user to adjust the mouth shape), and wherein the at least one additional feedback instruction is generated by a machine learning model that is trained based on one or more example feedback instructions and corresponding one or more samples of incorrect articulations (Pandey: ¶[0028] discloses a machine learning model for performing the analysis to determine how well the user has done and providing structured feedback).
Claims 2, 4, 9, 11, 16 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Pandey, in view of Steinhoff, further in view of Thies and further in view of Jawahar (US 2023/0290332).
Regarding Claim 2:
The proposed combination of Pandey, Steinhoff and Thies further discloses a computer-implemented method of claim 1, except wherein generating the synthetic video of the user includes processing one or more images of the user using a trained machine learning model. However, Jawahar discloses wherein generating the synthetic video of the user includes processing one or more images of the user using a trained machine learning model. However, Jawahar discloses this limitation: (Jawahar: p[0007] discloses that the synthesized video of the talking head is generated using a machine learning model).
It would have been obvious to one of ordinary skill in the art to combine Pandey and Jawahar as they both disclose related systems. Pandey discloses a system for generating audiovisual feedback for speech training by processing user video to create animations of articulatory movements. And Jawahar discloses generating a synthetic talking head videos using a trained machine learning model to process user images and synchronize lip movements with audio. Integrating Jawahar’s machine learning-based video synthesis into Pandey’s system would enhance its accuracy and realism, aligning with Pandey’s goal of providing improved feedback for speech training. Therefore, combining the two provides predictable results especially because Pandey notes in paragraph 0004 that machine learning is used within this field of endeavor and chooses not to use machine learning techniques because it is limiting to this specific field.
Regarding Claim 4:
The proposed combination of Pandey, Steinhoff, and Thies further discloses the computer-implemented method of claim 1, except further comprising identifying, using a natural language processing model, the one or more words that the user is speaking, and wherein the one or more words are used to generate the synthetic video. However, Jawahar discloses using a natural language processing model, the one or more words that the user is speaking, and wherein the one or more words are used to generate the synthetic video (Jawahar: explicitly discloses using natural language processing models to analyze speech and identify words or phrases being spoken, then further links word identification to actions such as generating synthetic visual representations).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate using a natural language processing (NLP) model for identifying speech into a system like the one disclosed in Pandey. Pandey discloses a system for generating audiovisual feedback for speech training by processing user video to create animations of articulatory movements. Jawahar discloses generating a synthetic talking head videos using a trained machine learning model to process user images and synchronize lip movements with audio. Integrating Jawahar’s NLP based word identification into Pandey’s system would be obvious because Pandey’s acoustic signal analysis forms a foundational component of NLP systems, particularly in tasks like speech-text or intelligent assistant systems. This integration could enhance Pandey’s system by enabling it to dynamically recognize the words a user is attempting to say, improving the accuracy and customization of the audiovisual feedback to support the goal of assisting hearing impaired individuals in producing intelligible speech.
Regarding Claim 9:
Claim 9 has been analyzed with regard to claim 2 (see rejection above) and
is rejected for the same reasons of obviousness as used above.
Regarding Claim 11:
Claim 11 has been analyzed with regard to claim 4 (see rejection above) and
is rejected for the same reasons of obviousness as used above.
Regarding Claim 18:
Claim 18 has been analyzed with regard to claim 4 (see rejection above) and
is rejected for the same reasons of obviousness as used above.
4. Claims 3, 7, 10, 14 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Pandey, in view of Steinhoff, further in view of Thies, and further in view of Sipolins (US 10,915,740).
Regarding Claim 3:
The proposed combination of Pandey, Steinhoff and Thies further discloses the computer-implemented method of claim 1, wherein presenting the live video comprises displaying the live video in a first layer that superimposes a second layer in which the synthetic video is displayed, and wherein the first layer is transparent. However, Sipolins teaches wherein presenting the live video comprises displaying the live video in a first layer that superimposes a second layer in which the synthetic video is displayed, and wherein the first layer is transparent (Sipolins: Col 5 lines 7-27 discloses superimposing markers or overlays on a real-time video of a user’s face).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Sipolins’ teaching of overlaying visual markers on a live video by using a transparent layer to superimpose synthetic video over live video. Sipolins already discloses overlaying visual markers on a real-time facial image, which inherently involves creating multiple layers in a multimedia presentation. Applying the concept of transparency to the live video layer to allow integration of a synthetic layer would be a logical extension to Sipolins approach if added to the combination of Pandey and Jawahar because the combination already discloses editing a synthetic layer/head, simply taking the suggestion this head gives and superimposing it onto the live face as done in Sipolins is a simple approach that. Th motivation for doing so is leveraging the real-time overlay capabilities of Sipolins and facial suggestions of Pandey and Jawahar synthetic videos to meet their goals of user-centric customization and effective guidance in speech or facial training systems.
Regarding Claim 7:
The proposed combination of Pandey, Steinhoff and Thies further discloses the computer-implemented method of claim 6, except wherein the one or more feedback instructions are presented using an augmented reality display while the user is engaging in a conversation. However, Sipolins discloses wherein the one or more feedback instructions are presented using an augmented reality display while the user is engaging in a conversation (Sipolins: Col 2 lines 47-48 discloses a facial expression feedback system on how closely the user’s expression using sensors and provides real time feedback, the system evaluates the captured data to generate confidence score and modify multimedia presentation based on these scores).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose the use of augmented reality to present feedback instructions to a user in speech training or language learning systems. Sipolins discloses providing real-time feedback on facial expression using augmented reality. Combine this with Pandey would be obvious because it could possibly enhance user experience by integrating augmented reality for intuitive and immediate visual prompts. Furthermore, integrating a program into virtual reality for one of ordinary skill in the art is a simple adaptation that would create a predictable outcome.
Regarding Claim 10:
Claim 10 has been analyzed with regard to claim 3 (see rejection above) and
is rejected for the same reasons of obviousness as used above.
Regarding Claim 14:
Claim 14 has been analyzed with regard to claim 7 (see rejection above) and
is rejected for the same reasons of obviousness as used above.
Regarding Claim 17:
Claim 17 has been analyzed with regard to claim 3 (see rejection above) and
is rejected for the same reasons of obviousness as used above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IAN SCOTT MCLEAN whose telephone number is (703)756-4599. The examiner can normally be reached "Monday - Friday 8:00-5:00 EST, off Every 2nd Friday".
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached at (571) 272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IAN SCOTT MCLEAN/Examiner, Art Unit 2654
/HAI PHAN/Supervisory Patent Examiner, Art Unit 2654