DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 9, 10, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shillingford et al (NPL: LARGE-SCALE VISUAL SPEECH RECOGNITION) in view of Ramarao (US 20250173556).
Regarding claim 1, Shillingford discloses an electronic device (pg. 2 Figure 1: The full visual speech recognition system), comprising:
circuitry (pg. 2 Figure 1: The full visual speech recognition system having processing, model, and decoding; the system inherently having circuity within these components) configured to:
receive a set of images including at least one of a plurality of human speakers (pg. 4 3 ADATAPIPELINE FOR LARGE-SCALE VISUAL SPEECH RECOGNITION: View canonicalization. We obtain canonical faces using a reference canonical face model and by applying an affine transformation on the landmarks. Then, we use a thumbnail extractor which is configured to crop the area around the lips of the canonical face.);
apply a first machine learning (ML) model on the received set of images (pg. 5 4 ANEFFICIENT SPATIOTEMPORAL MODEL OF VISUAL SPEECH RECOGNITION: Neural network architecture: Our vision module consists of 5 convolutional layers with [64,128,256,512,512] filters);
determine a first set of words, spoken by the at least one of the plurality of human speakers, based on the application of the first ML model, wherein the determined first set of words corresponds to lip movements of the at least one of the plurality of human speakers (pg. 6 4 ANEFFICIENT SPATIOTEMPORAL MODEL OF VISUAL SPEECH RECOGNITION: Neural network architecture: the architecture used in V2P is trained to predict phonemes rather than characters; see example Fig. 5 e.g. predictions of each frame);
Shillingford fails to teach where Ramarao teaches apply a first generative Artificial Intelligence (AI) model on the determined first set of words (¶69 the system provides the generative AI model with a prompt to specify a different number of output segments for different types of input segments);
break the determined first set of words into a plurality of tokens based on the application of the first generative AI model (¶35 The generative AI model 117 is trained to receive natural language as an input. The generative AI model 117 generates natural language as an output; ¶36 The attention mechanisms help neural networks in the generative AI model 117 to learn the context of words in the sequences of words. The attention mechanisms operate by breaking down a set of input data, such as a sentence or sequence of words or tokens, into keys, queries, and values);
determine a context of each of the plurality of tokens based on the application of the first generative Al model (¶36 The attention mechanisms help neural networks in the generative AI model 117 to learn the context of words in the sequences of words. The attention mechanisms operate by breaking down a set of input data, such as a sentence or sequence of words or tokens, into keys, queries, and values; ¶71 applying a text transformer model includes tokenizing an input sentence or sequence of words into word or sub-word level tokens; A resulting vector representing a sentence or sequence of words includes numerical values representing text content of tokens in a sentence or sequence of words as well as the contextual and positional information corresponding to the tokens); and
determine a first sentence corresponding to the determined first set of words, spoken by the at least one of the plurality of human speakers (¶29 Examples of interfaces include a graphical user interface (GUI), a command line interface (CLI), a haptic interface, and a voice command interface), based on the determined context of the each of the plurality of tokens and the application of the first generative Al model (¶71 The system converts the tokens into vector representations by passing the tokens though an embedding layer. The system applies a transformer encoder to the token embeddings to capture contextual and positional information of the token within a sentence or sequence of words. A resulting vector representing a sentence or sequence of words includes numerical values representing text content of tokens in a sentence or sequence of words as well as the contextual and positional information corresponding to the tokens).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of apply a first generative Artificial Intelligence (AI) model on the determined first set of words; break the determined first set of words into a plurality of tokens based on the application of the first generative AI model; determine a context of each of the plurality of tokens based on the application of the first generative Al model; and determine a first sentence corresponding to the determined first set of words, spoken by the at least one of the plurality of human speakers based on the determined context of the each of the plurality of tokens and the application of the first generative Al model from Ramarao into the electronic device as disclosed by Shillingford. The motivation for doing this is to improve machine learning model for generating text content from short prompts.
Regarding claim 9, the combination of Shillingford and Ramarao disclose the electronic device according to claim 1, wherein the circuitry is further configured to: detect a first human speaker of the plurality of human speakers, based on the received set of images (Shillingford pg. 4 3 ADATAPIPELINE FOR LARGE-SCALE VISUAL SPEECH RECOGNITION: View canonicalization. We obtain canonical faces using a reference canonical face model and by applying an affine transformation on the landmarks. Then, we use a thumbnail extractor which is configured to crop the area around the lips of the canonical face.), and determine the first set of words is further based on the detection of the first human speaker (Shillingford pg. 6 4 ANEFFICIENT SPATIOTEMPORAL MODEL OF VISUAL SPEECH RECOGNITION: Neural network architecture: the architecture used in V2P is trained to predict phonemes rather than characters; see example Fig. 5 e.g. predictions of each frame).
Regarding claim(s) 10 (drawn to a method):
The rejection/proposed combination of Shillingford and Ramarao, explained in the rejection of device claim(s) 1, anticipates/renders obvious the steps of the method of claim(s) 10 because these steps occur in the operation of the proposed combination as discussed above. Thus, the arguments similar to that presented above for claim(s) 1 is/are equally applicable to claim(s) 10.
Regarding claim(s) 17 (drawn to a CRM):
The rejection/proposed combination of Shillingford and Ramarao, explained in the rejection of device claim(s) 1, respectively, anticipates/renders obvious the steps of the computer readable medium of claim(s) 17 because these steps occur in the operation of the proposed combination as discussed above. Thus, the arguments similar to that presented above for claim(s) ) 1 is/are equally applicable to claim(s) 17. See Ramarao ¶145-146.
Claim(s) 2-4, 6-7, 11-13, 15-16 and 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Shillingford and Ramarao as applied to claims 1, 10 and 17 above, and further in view of Shah et al (US 20240155205).
Regarding claim 2, the combination of Shillingford and Ramarao disclose the electronic device according to claim 1, but fail to teach where Shah teaches wherein the circuitry is further configured to: concatenate the determined first set of words (Shah ¶32 The system then produces multiple scripts, each containing a series of concatenated text strings representing phrases along with associated inflection, emphasis, volume and emotional indicators, as well as timing and speaker identifiers, that are derived from the original audio); apply the first generative AI model on the concatenated first set of words, and determine the first sentence is further based on the application of the first generative AI model on the concatenated first set of words (Shah ¶44 At step 408, a generative artificial intelligence (AI) model may be used for the TTS; ¶45 Additionally, a vocoder model may be used for producing waveform based on the output of the generative AI model).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of concatenate the determined first set of words; apply the first generative AI model on the concatenated first set of words, and determine the first sentence is further based on the application of the first generative AI model on the concatenated first set of words from Shah into the electronic device as disclosed by the combination of Shillingford and Ramarao. The motivation for doing this is to improve the accuracy of automatic speech recognition (ASR).
Regarding claim 3, the combination of Shillingford and Ramarao disclose the electronic device according to claim 1, but fail to teach where Shah teaches wherein the determined first sentence corresponds to one of a structured sentence or an unstructured sentence (Shah ¶26 an audio stream may be transcribed into words that are grouped in phrases and sentences. Sentences include one or more phrases).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of wherein the determined first sentence corresponds to one of a structured sentence or an unstructured sentence from Shah into the electronic device as disclosed by the combination of Shillingford and Ramarao. The motivation for doing this is to improve the accuracy of automatic speech recognition (ASR).
Regarding claim 4, the combination of Shillingford and Ramarao disclose the electronic device according to claim 1, but fail to teach where Shah teaches wherein the circuitry is further configured to: apply a second ML model on the determined first set of words spoken by the at least one of the plurality of human speakers (Shah ¶27 any suitable type of machine learning algorithm(s) may be applied to extract these parameters from the original spoken audio); detect a first language associated with the determined first set of words based on the application of the second ML model (Shah ¶32 The text strings are simultaneously translated phrase-by-phrase into multiple languages by translation engine 10); apply a second generative AI model on the determined first set of words and the detected first language, and determine the first sentence based on the application of the second generative AI model (Shah ¶34 The generation of subtitles described above occurs at step 104 in FIG. 3. As discussed above, at step 104, the subtitles are initially generated by translating the caption text (generated at step 102) in its original source language).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of wherein the circuitry is further configured to: apply a second ML model on the determined first set of words spoken by the at least one of the plurality of human speakers; detect a first language associated with the determined first set of words based on the application of the second ML model; apply a second generative AI model on the determined first set of words and the detected first language, and determine the first sentence based on the application of the second generative AI model from Shah into the electronic device as disclosed by the combination of Shillingford and Ramarao. The motivation for doing this is to improve the accuracy of automatic speech recognition (ASR).
Regarding claim 6, the combination of Shillingford and Ramarao disclose the electronic device according to claim 1, but fail to teach where Shah teaches wherein the circuitry is further configured to:
receive a set of audio frames associated with the received set of images (Shah ¶20 a series of video frames and an audio track);
apply a third ML model on the received set of audio frames (Shah ¶27 any suitable type of machine learning algorithm(s) may be applied to extract these parameters from the original spoken audio), and determine the first set of words spoken by the at least one of the plurality of human speakers based on the application of the third ML model (Shah ¶32 The text strings are simultaneously translated phrase-by-phrase into multiple languages by translation engine 10).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of receive a set of audio frames associated with the received set of images; apply a third ML model on the received set of audio frames; and determine the first set of words spoken by the at least one of the plurality of human speakers based on the application of the third ML model from Shah into the electronic device as disclosed by the combination of Shillingford and Ramarao. The motivation for doing this is to improve the accuracy of automatic speech recognition (ASR).
Regarding claim 7, the combination of Shillingford and Ramarao disclose the electronic device according to claim 1, but fail to teach where Shah teaches wherein the circuitry is further configured to:
generate a group of words associated with the determined first set of words, based on the application of the first generative AI model (Shah ¶41 With reference to FIG. 6, following text-to-speech conversion at step 400, based on the speed of the subtitles translated, each subtitle at a time, dubbing engine 11 adjusts the speaking rate for the subtitle, keeping the level between a band of values; ¶44 At step 408, a generative artificial intelligence (AI) model may be used for the TTS); and determine the first sentence corresponding to the determined first set of words based on the generated group of words (Shah ¶44 At step 408, a generative artificial intelligence (AI) model may be used for the TTS; ¶45 Additionally, a vocoder model may be used for producing waveform based on the output of the generative AI model).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of generate a group of words associated with the determined first set of words, based on the application of the first generative AI model; and determine the first sentence corresponding to the determined first set of words based on the generated group of words from Shah into the electronic device as disclosed by the combination of Shillingford and Ramarao. The motivation for doing this is to improve the accuracy of automatic speech recognition (ASR).
Regarding claim 8, the combination of Shillingford, Ramarao and Shah disclose the electronic device according to claim 7, wherein the determined first sentence includes at least one of the generated group of words and the determined first set of words (Shah ¶44 At step 408, a generative artificial intelligence (AI) model may be used for the TTS; ¶45 Additionally, a vocoder model may be used for producing waveform based on the output of the generative AI model). The motivation to combine the references is discussed above in the rejection for claim 7.
Regarding claim(s) 11-13 and 15-16 (drawn to a method):
The rejection/proposed combination of Shillingford, Ramarao and Shah, explained in the rejection of device claim(s) 2-4 and 6-7, anticipates/renders obvious the steps of the method of claim(s) 11-13 and 15-16 because these steps occur in the operation of the proposed combination as discussed above. Thus, the arguments similar to that presented above for claim(s) 2-4 and 6-7 is/are equally applicable to claim(s) 11-13 and 15-16.
Regarding claim(s) 18-19 (drawn to a CRM):
The rejection/proposed combination of Shillingford, Ramarao and Shah, explained in the rejection of device claim(s) 7 and 4, respectively, anticipates/renders obvious the steps of the computer readable medium of claim(s) 18-19 because these steps occur in the operation of the proposed combination as discussed above. Thus, the arguments similar to that presented above for claim(s) ) 7 and 4 is/are equally applicable to claim(s) 18-19. See Ramarao ¶145-146.
Allowable Subject Matter
Claims 5, 14 and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 5, and similarly regarding claims 14 and 20, the prior art of record, alone or in combination, fails to teach at least “apply each ML model of a set of ML models on the determined first set of words spoken by the one of the plurality of human speakers; determine, from the determined first set of words, a second set of words in a first language and a third set of words in a second language, based on the application of each corresponding ML model of the set of ML models; apply a second generative AI model on the determined second set of words and the first language; apply a third generative AI model on the determined third set of words and the second language; determine a second sentence in the first language, based on the application of the second generative AI model; determine a third sentence in the second language, based on the application of the third generative AI model, and determine the first sentence based on each of the determination of the second sentence and the determination of the third sentence” in the specific order as claimed.
Response to Arguments
Applicant’s arguments with respect to the rejection under 35 U.S.C. 101 have been fully considered and are persuasive. The rejection under 35 U.S.C. 101 of claims 1-20 has been withdrawn.
Applicant’s arguments with respect to claim(s) 1-4, 6-13, and 15-19 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEVIN KY whose telephone number is (571)272-7648. The examiner can normally be reached Monday-Friday 9-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEVIN KY/ Primary Examiner, Art Unit 2671
Prosecution Insights