DETAILED ACTION
Introduction
Applicant's submission filed on 02/06/2025 has been entered. Claims 1-20 are pending in the application and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to mental activity abstract idea without significantly more.
According to USPTO guidelines, a claim is directed to non-statutory subject matter if:
STEP 1: the claim does not fall within one of the four statutory categories of invention (process, machine, method of manufacture, or composition of matter), or
STEP 2: the claim recites a judicial exception (e.g. an abstract idea) without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis:
STEP 2A (Prong 1): Does the claim recite an abstract idea, law of nature, or natural phenomenon? The guidelines provide three groupings of subject matter that are considered abstract ideas:
Mathematical concepts- mathematical relationships, formulas or equations, calculations
Certain methods of organizing human activity- fundamental economic principles or practices, commercial or legal interactions, managing personal behavior or relationships or interactions between people
Mental processes- concepts that are practicably performed in the human mind (including an observation, evaluation, judgement, or opinions)
STEP 2A (Prong 2): Does the claim recite additional elements that integrate the judicial exception into a practical application? The guidelines provide the following exemplary considerations that are indicative than an additional element (or combination of elements) may have integrated the judicial exception into a practical application:
an additional element reflects an improvement in the functioning of a computer, or an improvement to other technology or technical field;
an additional element that applies or uses a judicial exception to affect a particular treatment or prophylaxis for a disease or medical condition;
an additional element implements a judicial exception with, or uses a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim;
an additional element effects a transformation or reduction of a particular article to a different state or thing; and
an additional element applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception.
While the guidelines further state that the exemplary considerations are not an exhaustive list and that there may be other examples of integrating the exception into a practical application, the guidelines also list examples in which a judicial exception has not been integrated into a practical application:
an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea;
an additional element adds insignificant extra-solution activity to the judicial exception; and
an additional element does no more than generally link the use of a judicial exception to a particular technological environment or field of use.
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? Consider whether an additional element or combination of elements:
adds a specific limitation or combination of limitations that are not well-understood, routine, or conventional activity in the field, which is indicative that an inventive concept may be present; or
simply appends well-understood, routine and conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, which is indicative that an inventive concept may not be present.
Using the two-step inquiry, claim 1 is directed to an abstract idea as show below:
STEP 1: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
YES. Claim 1 is directed to a method.
STEP 2A (Prong 1): Does the claim recite an abstract idea, law of nature, or natural phenomenon?
YES. The claim recites an abstract idea:
The limitation processing the audio input using a trained encoder to generate a representation of the audio input, wherein the audio input includes speech as drafted, is a process that, under its broadest reasonable interpretation, can be performed by a human determining the written or other forms by listening to the audio containing speech and determining the speech.
The limitation of processing the representation of the audio input using a first trained adapter to generate one or more features, as drafted, is a process that, under its broadest reasonable interpretation, recites a mathematical formula or calculation based on the gathered data.
STEP 2A (Prong 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
NO.
Claim 1 recites the additional element of processing through a “trained encoder”, a “first trained adapter” and a “trained language model” which are recited at a high level of generality and amounts to merely using a computer as a tool to perform an abstract idea or mere instructions to apply the exception using a generic computer component. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the insignificant extra-solution activities abstract idea but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Claim 1 also recites processing the one or more features and text associated with the audio input using a trained language model to generate a response, as drafted, is a process that, under its broadest reasonable interpretation, recites an activity of gathering a response to the audio speech query (or extra solution activity), that adds insignificant extra-solution activity to the judicial exception. This judicial exception is not integrated into a practical application.
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
NO.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a “trained encoder”, a “first trained adapter” and a “trained language model” amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Claim 1 is not patent eligible.
Claim 2 further specifies steps to generate text associated with the audio, is a human activity and does not reflect an improvement in the functioning of a technology or computer. The claim is not patent eligible.
Claim 3 further recites specifies configuration of the computer models and does not reflect an improvement in the functioning of a technology or computer. The claim is not patent eligible.
Claim 4-5 further specifies identification of different features of speech, is a mathematical computation and does not reflect an improvement in the functioning of a technology or computer. The claim is not patent eligible.
Claim 6-7, 9 further specifies configuration and training of the computer models and does not reflect an improvement in the functioning of a technology or computer. The claim is not patent eligible.
Claim 8 further recites specifies configurations of the computer models and does not reflect an improvement in the functioning of a technology or computer. The claim is not patent eligible.
Claim 10 further recites specifies the details of identifying the speech as a question and the response to the question is a human activity and does not reflect an improvement in the functioning of a technology or computer. The claim is not patent eligible.
Claims 11-12, 14-18 are analogous to claims 1-2, 4-6, 8, 10 respectively, as directed to non-transitory computer-readable media claims to perform the operations set forth in claims 1-2, 4-6, 8, 10 and are subjected to the same rejections as claims 1-2, 4-6, 8, 10 respectively.
Claim 13 further specifies configuration and training of the computer models and does not reflect an improvement in the functioning of a technology or computer. The claim is not patent eligible.
Claim 19 further specifies the device to display the result and does not reflect an improvement in the functioning of a technology or computer. The claim is not patent eligible.
Claim 20 is analogous to claim 1 respectively, as directed to system claim to perform the operations set forth in claims 1 and are subjected to the same rejections as claim 1 respectively.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Tang, C, et al. “Salmonn: Towards generic hearing abilities for large language models.”, International Conference on Learning Representations. Vol. 2024. 2024. (cited in IDS) in view of J. Santoso, et.al., "Large Language Model-Based Emotional Speech Annotation Using Context and Acoustic Feature for Speech Emotion Recognition," ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Korea, Republic of, 2024, pp. 11026-11030.
Regarding claim 1, Tang teaches a computer-implemented method for responding to audio input, the method comprising: processing the audio input using a trained encoder to generate a representation of the audio input, wherein the audio input includes speech ( see Tang, Fig. 1, Whisper Encoder ) ; processing the representation of the audio input using a first trained adapter to generate one or more features(see Tang, Fig. 1, Q former - adapter); and processing the one or more features and text associated with the audio input using a trained language model to generate a response (see Tang, Fig. 1, A pre-trained Vicuna LLM processes the features and text associated with the audio input to generate a text response ).
Tang teaches processing the one or more features and text associated with the audio input using a trained language model to generate a response based on the Salmonn model, Sentosa further teaches processing the one or more features and text associated with the audio input using a trained language model to generate a response(see Sentosa, Fig. 1 describes conversion of acoustic features to text to describe the acoustic feature and based on the transcription, a prompt is provided to the LLM which generates an output; LLM also extracts context from the dynamics in the acoustic feature descriptors aside from context extracted from the content).
Tang and Sentosa are considered to be analogous to the claimed invention because both relate to speech features recognition and processing using large language model (LLM). Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Tang to determine automatic speech recognition and translation, auditory information-based question answering, emotion recognition, speaker verification, and music and audio captioning with the speech emotion recognition teachings of Sentosa to improve annotating emotional speech data while incorporating the information from acoustic features (see Sentosa, sect 1).
Regarding claim 2, Tang in view of Sentosa teaches the computer-implemented method of claim 1. Sentosa further teaches further comprising processing the representation of the audio input using a trained decoder to generate the text associated with the audio input (see Sentosa, Fig. 1 text content(transcription), sect 2.2 First, we extract the speech’s acoustic features and text content). Tang also teaches further comprising processing the representation of the audio input using a trained decoder to generate the text associated with the audio input (see Tang, Fig. 5, pg. 14). The same motivation to combine as claim 1 applies here.
Regarding claim 3, Tang in view of Sentosa teaches the computer-implemented method of claim 2. Sentosa further teaches wherein the trained encoder and the trained decoder are included in a trained speech model (see Sentosa, sect 2.2 First, we extract the speech’s acoustic features and text content using ASR ( trained speech model The same motivation to combine as claim 1 applies here.
Regarding claim 4, Tang in view of Sentosa teaches the computer-implemented method of claim 1. Sentosa further teaches wherein the one or more features represent at least one of a speaking style or speaker information associated with the speech (see Sentosa, Fig. 1 acoustic features description as text.). Tang also teaches wherein the one or more features represent at least one of a speaking style or speaker information associated with the speech (see Tang, Fig. 11, 13, 14 .). The same motivation to combine as claim 1 applies here.
Regarding claim 5, Tang in view of Sentosa teaches the computer-implemented method of claim 1. Sentosa further teaches wherein the one or more features represent at least one of a pitch, a volume, a speaking speed, an emotion, or a gender associated with the speech (see Sentosa, Fig. 1 acoustic features description, speaking rate, pitch, loudness, intensity, gender). Tang also teaches wherein the one or more features represent at least one of a pitch, a volume, a speaking speed, an emotion, or a gender associated with the speech (see Tang, Fig. 11, 13, 14 ). The same motivation to combine as claim 1 applies here.
Regarding claim 6, Tang in view of Sentosa teaches the computer-implemented method of claim 1. Tang further teaches wherein the trained language model comprises a second trained adapter, and the second trained adapter was trained together with the first trained adapter (see Tang, sect 3.2 To mitigate the gap between the pre-trained parameters (LLM and encoders) and the randomly initialized parameters (connection module and adaptor), a large amount of speech recognition and audio captioning data is used to pre-train the window-level Q-Former and LoRA (second trained adaptor)).
Regarding claim 7, Tang in view of Sentosa teaches the computer-implemented method of claim 6. Tang further teaches wherein the second trained adapter comprises a trained Low-Rank Adaptation of Large Language Models (LoRA) adapter (see Tang, Fig. 1, sect 3.1 LoRA (Hu et al., 2022) is a widely used parameter-efficient fine-tuning method for LLM adaptation, which is used in Salomon).
Regarding claim 8, Tang in view of Sentosa teaches the computer-implemented method of claim 1. Tang further teaches wherein processing the one or more features and the text using the trained language model comprises prompting the trained language model to respond to the text (see Tang, Fig. 1 text instruction prompt to LLM; Tang, Figs, 5, 8-10).
Regarding claim 9, Tang in view of Sentosa teaches the computer-implemented method of claim 1. Tang further teaches wherein the trained language model comprises a large language model (LLM) (see Tang, Fig. 1 LLM, sect 3.1 A pre-trained Vicuna LLM is used in this work (Chiang et al., 2023) which is a LLaMA LLM (Touvron et al., 2023) fine-tuned to follow instructions).
Regarding claim 10, Tang in view of Sentosa teaches the computer-implemented method of claim 1. Tang further teaches wherein the speech includes a question, and the response comprises text that includes an answer to the question (see Tang, Fig. 10).
Regarding claim 11, is directed to a non-transitory computer-readable media claim corresponding to the method claim presented in claim 1 and is rejected under the same grounds stated above regarding claim 1.
Regarding claim 12, is directed to a non-transitory computer-readable media claim corresponding to the method claim presented in claim 2 and is rejected under the same grounds stated above regarding claim 2.
Regarding claim 13, Tang in view of Sentosa teaches the computer-implemented method of claim 1. Tang further teaches wherein the first trained adapter is trained separately from the trained encoder and the trained language model (see Tang, sect 3.1, 3.2 describes the LLM and the encoders are pretrained & the adaptors Q-Former and LoRA are trained separately).
Regarding claim 14, is directed to a non-transitory computer-readable media claim corresponding to the method claim presented in claim 4 and is rejected under the same grounds stated above regarding claim 4.
Regarding claim 15, is directed to a non-transitory computer-readable media claim corresponding to the method claim presented in claim 5 and is rejected under the same grounds stated above regarding claim 5.
Regarding claim 16, is directed to a non-transitory computer-readable media claim corresponding to the method claim presented in claim 6 and is rejected under the same grounds stated above regarding claim 6.
Regarding claim 17, is directed to a non-transitory computer-readable media claim corresponding to the method claim presented in claim 8 and is rejected under the same grounds stated above regarding claim 8.
Regarding claim 18, is directed to a non-transitory computer-readable media claim corresponding to the method claim presented in claim 10 and is rejected under the same grounds stated above regarding claim 10.
Regarding claim 20, is directed to a system claim corresponding to the method claim presented in claim 1 and is rejected under the same grounds stated above regarding claim 1.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Tang, C, et al. “Salmonn: Towards generic hearing abilities for large language models.”, International Conference on Learning Representations. Vol. 2024. 2024. (cited in IDS) in view of J. Santoso, et.al., "Large Language Model-Based Emotional Speech Annotation Using Context and Acoustic Feature for Speech Emotion Recognition," ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Korea, Republic of, 2024, pp. 11026-11030 further in view of Jou et.al US PgPub. 2025/0349298.
Regarding claim 19, Tang in view of Sentosa teaches the computer-implemented method of claim 1. However, Tang in view of Sentosa fail to teach wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the step of outputting the response via an output device.
However, Jou teaches wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the step of outputting the response via an output device (see Jou, [0054] At operation 610, processing logic can provide the adjusted caption text for display via a graphical user interface. In some implementations, the graphical user interface can be a graphical user interface of the computing device ).
Tang in view of Sentosa teach the speech features recognition and processing using large language model, however does not teach display of the response on a output device. Jou teaches display of output results via a graphical user interface. Using the known technique of using a graphical user interface as taught by Jou (see Jou, [0054]), to provide the adjusted caption for display via the graphical user interface of the computing device in the references Tang in view of Sentosa would have been obvious to one of ordinary skill in the art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yang et. al., US PgPub. 2024/0412029 teaches techniques for the prompt is augmented based on the state of the user to generate the relevant response using generative AI (see Yang, [0049]).
Bennet, US PgPub. 2006/0122834 teaches training of the prosody analyzer with real world expected responses improves emotion modeling and the real-time identification of potential features such as emphasis, intent, attitude and semantic meaning in the speaker's utterances (see Bennet, abstract).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NANDINI SUBRAMANI whose telephone number is (571)272-3916. The examiner can normally be reached Monday - Friday 12:00pm - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh M Mehta can be reached at (571)272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NANDINI SUBRAMANI/ Examiner, Art Unit 2656
