DETAILED ACTION
This action is in response to the amendment filed on 04/15/2026.
The arguments filed on 04/15/2026 state that the newly amended claims 1 and 18 recite the following: processing, in response to determining that the audio data capturing the user speech is noisy, the audio data to generate one or more audio embeddings that represent acoustic features of the audio data. However, independent claims 1 and 18 actually recite the following: processing, in response to determining that the audio data capturing the user speech is noisy, the audio data to determine a speech recognition of the speech. Since applicant’s arguments suggest that the applicant’s invention is directed to the generation of audio embeddings based on a determination of noisy user speech, the claims can be further amended to include this limitation to further prosecution. For example, the independent claims can be amended to recite the following - in response to receiving the audio data capturing the user speech and determining that the audio data is noisy: processing the audio data to determine a speech recognition of the user speech, processing the audio data to generate one or more audio embeddings that represent acoustic features of the audio data …
Response to Amendment
Applicant’s amendment filed on 04/15/2026 has been entered. Claims 1, 4 – 9, 18 and 19 have been amended. Claims 2, 3 and 12 – 17 have been canceled. No claims have been added. Claims 1, 4 – 11 and 18 – 20 are still pending in this application, with claims 1 and 18 being independent.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4, 5 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xing et al. (US 2023/0223018) (“Xing”) in view of KO (US 2020/0143807).
For claim 1, Xing discloses a method implemented using one or more processors (Fig.1, 102; [0050 - 0051] [0054]), the method comprising: receiving audio data capturing user speech (Fig.2, 210 and Fig.3, 210 [0058] [0059] [0063]); and in response to receiving the audio data capturing the user speech: processing the audio data to determine a speech recognition (text transcript) of the user speech (An online attention CTC neural network receives speech and processes it to generate a text transcript., Fig.2, 220, 240, Fig.3, 303, 306, 312, 310, 316; [0061] [0062] [0064 – 0068] [0070]), processing the audio data (speech chunks/segments of an input speech signal, [0039] [0040]) to generate one or more audio embeddings (encoded speech embeddings, Fig.4, 414; [0041 - 0043]) that represent acoustic features of the audio data (Fig.4, 402; [0039] [0040] [0073] [0074]), and processing, using a machine learning (ML) model (neural network including a cross modal attention subnetwork, concatenator subnetwork and sequence classifier, Fig.4, 418, 422 and 426; [0006] [0009] [0030] [0074]), both (i) the one or more audio embeddings that represent the acoustic features of the audio data (encoded speech embeddings, Fig.4, 414) and (ii) a text embedding that represent the speech recognition (encoded word embeddings, Fig.4, 416; [0075]), to generate a model output (semantic prediction, Fig.2, 260 and Fig.4, 260; [0045] [0046]) ([0076 – 0083]); determining a response (command action, Fig.2, 280) to the user speech based on the model output ([0047] [0084]), and causing the response to be rendered in response to the user speech (The semantic predictions 260 may be transformed by an interpreter 270 into a command action 280 based on a predefined set of commands. A computing system or computer application running on a computing system that is capable of executing the predefined command action 280 may then be able to execute the command action 280… The streamable MLU system 200 may process the speech signal 210 to output a semantic prediction 260 that captures the speaker's intent to “turn on” “the lights”. The smart speaker may then be able to map the semantic prediction to a command action 280 from a predefined set of command actions that the user wishes to turn on the lights, and may execute the command action 280, [0060]).
Yet, Xing fails to teach the following: processing, in response to determining that the audio data capturing the user speech is noisy, the audio data to determine the speech recognition.
However, Ko discloses a method for providing a response to user’s speech or utterance (Abstract), comprising the following: determining that an audio data capturing user speech is noisy (Fig.4, S410, S420; [0089 – 0101]); and further performing automatic speech recognition and natural language understanding at a remote device (server based ASR and NLU) in response to determining that the audio data capturing user speech is noisy (Fig.4, S430, S440, S470 and S480; [0102 – 0106]).
Additionally, Xing discloses that automatic speech recognition (ASR) and natural language understanding (MLU) are performed at a remote device (The streamable MLU system is provided as service to other electronic devices, wherein a speech signal is generated by a microphone of another electronic device and communicated to the device comprising the streamable MLU system, [0048] [0059]). Furthermore, Xing discloses that the automatic speech recognition and natural language understanding comprise speech recognition of the user speech and audio embedding generation of captured audio ([0061] [0062] [0073] [0074])
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve Xing’s invention in the same way that KO’s invention has been improved to achieve the following, predictable results for the purpose of accurately and reliably processing natural language input to better capture a user’s intent to provide a desirable response (Xing, [0002] [0003]) (KO, [0009]): further determining whether the audio data capturing the user speech is noisy; and processing the audio data to determine the speech recognition in response to determining that the audio data capturing the user speech is noisy (The audio signal is transmitted to the remote device/server. Automatic speech recognition and natural language understanding are performed at the remote device/server. The automatic speech recognition comprises processing audio data to determine a speech recognition/text of the user’s speech).
For claim 4, KO further discloses, wherein determining that the audio data capturing the user speech is noisy comprises: determining that a distance between a user providing the user speech and a client device that captures the audio data is greater than a distance threshold (KO, [0078]).
For claim 5, KO further discloses, wherein determining that the audio data capturing the user speech is noisy comprises: determining that a signal-to-noise ratio (SNR) for the audio data does not satisfy a SNR threshold (KO, [0078] [0093 – 0095] [0103]).
For claim 18, Xing discloses a system (Abstract) comprising one or more processors (Fig.1, 102; [0050 - 0051] [0054]), and memory (Fig.1, 116) storing instructions that, when executed by one or more of the processors, cause one or more of the processors ([0054]) to: in response to receiving the audio data capturing the user speech (Fig.2, 210 and Fig.3, 210 [0058] [0059] [0063]): process the audio data to determine a speech recognition (text transcript) of the user speech (An online attention CTC neural network receives speech and processes it to generate a text transcript., Fig.2, 220, 240, Fig.3, 303, 306, 312, 310, 316; [0061] [0062] [0064 – 0068] [0070]), process the audio data (speech chunks/segments of an input speech signal, [0039] [0040]) to generate one or more audio embeddings (encoded speech embeddings, Fig.4, 414; [0041 - 0043]) that represent acoustic features of the audio data (Fig.4, 402; [0039] [0040] [0073] [0074]), and process, using a machine learning (ML) model (neural network including a cross modal attention subnetwork, concatenator subnetwork and sequence classifier, Fig.4, 418, 422 and 426; [0006] [0009] [0030] [0074]), both (i) the one or more audio embeddings that represent the acoustic features of the audio data (encoded speech embeddings, Fig.4, 414) and (ii) a text embedding that represent the speech recognition (encoded word embeddings, Fig.4, 416; [0075]), to generate a model output (semantic prediction, Fig.2, 260 and Fig.4, 260; [0045] [0046]) ([0076 – 0083]); determine a response (command action, Fig.2, 280) to the user speech based on the model output ([0047] [0084]), and causing the response to be rendered in response to the user speech (The semantic predictions 260 may be transformed by an interpreter 270 into a command action 280 based on a predefined set of commands. A computing system or computer application running on a computing system that is capable of executing the predefined command action 280 may then be able to execute the command action 280… The streamable MLU system 200 may process the speech signal 210 to output a semantic prediction 260 that captures the speaker's intent to “turn on” “the lights”. The smart speaker may then be able to map the semantic prediction to a command action 280 from a predefined set of command actions that the user wishes to turn on the lights, and may execute the command action 280, [0060]).
Yet, Xing fails to teach the following: processing, in response to determining that the audio data capturing the user speech is noisy, the audio data to determine the speech recognition.
However, Ko discloses a method for providing a response to user’s speech or utterance (Abstract), comprising the following: determining that an audio data capturing user speech is noisy (Fig.4, S410, S420; [0089 – 0101]); and further performing automatic speech recognition and natural language understanding at a remote device (server based ASR and NLU) in response to determining that the audio data capturing user speech is noisy (Fig.4, S430, S440, S470 and S480; [0102 – 0106]).
Additionally, Xing discloses that automatic speech recognition (ASR) and natural language understanding (MLU) are performed at a remote device (The streamable MLU system is provided as service to other electronic devices, wherein a speech signal is generated by a microphone of another electronic device and communicated to the device comprising the streamable MLU system, [0048] [0059]). Furthermore, Xing discloses that the automatic speech recognition and natural language understanding comprise speech recognition of the user speech and audio embedding generation of captured audio ([0061] [0062] [0073] [0074])
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve Xing’s invention in the same way that KO’s invention has been improved to achieve the following, predictable results for the purpose of accurately and reliably processing natural language input to better capture a user’s intent to provide a desirable response (Xing, [0002] [0003]) (KO, [0009]): further determining whether the audio data capturing the user speech is noisy; and processing the audio data to determine the speech recognition in response to determining that the audio data capturing the user speech is noisy (The audio signal is transmitted to the remote device/server. Automatic speech recognition and natural language understanding are performed at the remote device/server. The automatic speech recognition comprises processing audio data to determine a speech recognition/text of the user’s speech).
Claim(s) 6 and 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xing et al. (US 2023/0223018) (“Xing”) in view of KO (US 2020/0143807) and further in view of Graciarena et al. (US 2025/0046333) (“Graciarena”).
For claim 6, the combination of Xing and KO fails to teach, wherein processing the audio data to generate one or more the audio embedding that represent the acoustic features of the audio data comprises: generating a spectrogram from the audio data capturing the user speech, and processing the spectrogram to extract spectrogram features corresponding to the audio data as the acoustic features.
However, Graciarena discloses a system and method to automatically identity and classify audio input (Abstract), comprising the following: generating a spectrogram from audio data capturing user speech (An input audio waveform is converted to a spectrogram, [0004] [0019 – 0021]); and processing the spectrogram to extract spectrogram features corresponding to the audio data as acoustic features (The audio spectrogram is applied to a log Mel filter bank to generate acoustic features, [0041 – 0046]).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve the invention disclosed by the combination of Xing and KO in the same way that Graciarena’s invention has been improved to achieve the following, predictable results for the purpose of accurately and reliably processing natural language input to better capture a user’s intent to provide a desirable response (Xing, [0002] [0003]): further generating a spectrogram from the audio data capturing the user speech, and processing the spectrogram to extract spectrogram features corresponding to the audio data as the acoustic features (Xing, The speech features are extracted busing a 80-dimenensional
log Mel filter bank, [0040]).
For claim 7, Xing and Graciarena further disclose, wherein processing the audio data to generate the one or more audio embeddings that represent acoustic features of the audio data comprises: processing the spectrogram features, using an audio encoder, to generate the one or more audio embeddings (Xing, [0040] [0058 – 0061] [0073] [0074]) (Graciarena, [0041 – 0043] [0047] [0048]).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xing et al. (US 2023/0223018) (“Xing”) in view of KO (US 2020/0143807) and further in view of Shabat et al. (US 2024/0203404) (“Shabat”).
For claim 8, the combination of Xing and Ko fails to teach that the ML model (neural network, [0006] [0009] ) is a transformer-based large language model (LLM).
However, Shabat discloses a system and method for the purpose of enabling large language model-based spoken language understanding systems to leverage both audio and textual data (Abstract), wherein a machine learning model (Fine-Tuned LLM/NLU Module) which receives audio (e.g. speech) input and text input generated by ASR and outputs natural language understanding data(e.g. intent) is a transformer based large language model (Fig.1B, 160 and Fig 3A, 300; [0003] [0032 – 0040] [0054 – 0056]).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s to improve the invention disclosed by the combination of Xing and KO in the same way that Shabat’s invention has been improved to achieve the predictable results of the ML model, which receives and processes audio and text input to generate a spoken language understanding output, further comprising a transformer based LLM for the purpose of leveraging both audio and textual data to predict semantics information contained in received speech to generate a desirable response using LLMs, wherein LLMs enable transfer learning of general-purpose knowledge into specific NLP tasks (Xing, [0002] [0003] [0008]) (Shabat, [0001 – 0003] [0006]).
Claim(s) 9, 10,11, 19 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xing et al. (US 2023/0223018) (“Xing”) in view of KO (US 2020/0143807), and further in view of Shabat et al. (US 2024/0203404) (“Shabat”), and further in view of Liu et al. (US 2023/0368796) (“Liu”) and further in view of Jaber et al. (US 2024/0054342) (“Jaber”).
For claims 9 and 19, the combination of Xing and Ko fails to teach, wherein processing both (i) the one or more audio embeddings that represents the acoustic features of the audio data and (ii) the text embedding that represent the speech recognition comprises: processing the text embedding, using a multi-head attention mechanism, to generate intermediate attention features, and providing the intermediate attention features and the one or more audio embeddings to an additional multi-head attention mechanism.
However, Shabat discloses a system and method for the purpose of enabling large language model-based spoken language understanding systems to leverage both audio and textual data (Abstract), wherein a machine learning model (Fine-Tuned LLM/NLU Module) which receives audio (e.g. speech) input and text input generated from an ASR and outputs natural language understanding data(e.g. intent) is a large language model (Fig.1B, 160 and Fig 3A, 300 and Fig.3B; [0003] [0032 – 0040] [0054 – 0057] [0061]). Additionally, Shabat discloses providing one or more text embeddings and one or more audio embeddings to a multi-head attention mechanism (Fig.3C, 341; [0057] [0061]).
Moreover, Liu discloses a system and method for performing spoken language understanding (Abstract), comprising the following: a text encoder further comprises a transform encoder and a text embedder (Fig.5, 420, 522 and 524; [0106] [0107]); and text embeddings generated by the text embedder are further processed by the transform encoder of the text encoder to generate intermediate features which are forwarded to a decoding process ([0107]).
Furthermore, Jaber discloses a system and method for processing input using a machine learning model (Abstract), wherein input (e.g. text) is processed by a transformer encoder (Fig.2B, 250) comprising a multi-headed attention section ([0052]).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s to improve the invention disclosed by the combination of Xing and KO in the same way that Shabat’s invention has been improved to achieve the predictable results of the ML model, which receives and processes audio and text input to generate a spoken language understanding output, further comprising a transformer based LLM, wherein one or more text embeddings and one or more audio embeddings are provided to a multi-head attention mechanism in the LLM model for the purpose of leveraging both audio and textual data to predict semantics information contained in received speech to generate a desirable response using LLMs, wherein LLMs enable transfer learning of general-purpose knowledge into specific NLP tasks (Xing, [0002] [0003] [0008]) (Shabat, [0001 – 0003] [0006]).
Additionally, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve the invention disclosed by the combination of Xing, KO and Shabat in the same way that Liu’s invention has been improved to achieve the following, predictable results for the purpose of leveraging both audio and textual data to predict semantics information contained in received speech to generate a desirable response using LLMs which enable transfer learning of general-purpose knowledge into specific NLP tasks (Xing, [0002] [0003] [0008]) (Shabat, [0001 – 0003] [0006]): a text encoder of the LLM model (Shabat, Fig.3A, 310) further comprises a transform encoder and text embedder; and text embeddings generated by the text embedder are further processed by the transform encoder of the text encoder to generate intermediate features which are provided to the multi-head attention mechanism.
Moreover, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve the invention disclosed by the combination of Xing, KO, Shabat and Liu in the same way that Jaber’s invention has been improved to achieve the following, predictable results for the purpose of leveraging both audio and textual data to predict semantics information contained in received speech to generate a desirable response using LLMs which enable transfer learning of general-purpose knowledge into specific NLP tasks (Xing, [0002] [0003] [0008]) (Shabat, [0001 – 0003] [0006]: the transformer encoder further comprises a multi-head attention mechanism.
For claim 10, Jaber further discloses, wherein the multi-head attention mechanism or the additional multi-head attention mechanism includes multiple attention heads each having a query matrix, a key matrix, and a value matrix (Jaber, [0052]).
For claims 11 and 20, Shabat, Liu and Jaber further disclose, wherein providing the intermediate attention features and the one or more audio embeddings to an additional multi-head attention mechanism causes the intermediate attention features to be multiplied with the query matrix, and the one or more audio embeddings to be multiplied with the key matrix and the value matrix, respectively (Shabat, [0054 – 0057] [0061]) (Liu, [0106] [0107]) (Fig.5, 430 and 532; [0108] [0109]) (Jaber, All inputs features are multiplied with the query matrix, key matrix and value matrix, [0052]).
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 4 – 11 and 18 – 20 have been considered but are moot in view of the new ground(s) of rejection. The new ground(s) of rejected were necessitated by amendment based on a change in scope of the independent claims. Previously, independent claims 1 and 18 recited, “in response to receiving the audio data capturing the user speech, processing the audio data to determine a speech recognition of the user speech.” Independent claims 1 and 18 now recite, “processing, in response to determining that the audio data capturing the user speech is noisy, the audio data to determine a speech recognition of the user speech.”
Although applicant’s arguments state that this new limitation was previously recited in claims 2 and 3, claims 2 and 3 previously recited the following limitations: determining whether the audio data capturing the user speech is noisy, wherein processing the audio data to generate the one or more audio embeddings is performed in response to determining that the audio data capturing the user speech is noisy.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Hariri Nokob et al. (US 12,620,389) (teaches the inventive concept of using audio encodings to improve/correct ASR transcriptions
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SONIA L GAY whose telephone number is (571)270-1951. The examiner can normally be reached Monday-Friday 9-5 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at 571-272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SONIA L GAY/Primary Examiner, Art Unit 2657
Prosecution Insights