DETAILED ACTION
This office action is in response to Applicant’s Amendment/Request for Reconsideration, received on 03/13/2026. Claims 1-2, 14-15, and 21 have been amended. Claim 24 has been added. Claims 1-24 are pending and have been considered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see pg. 10, filed 03/13/2026, with respect to objections to the specification have been fully considered and are persuasive. The objection of [0028] of the specification has been withdrawn. The examiner would like to note that Applicant appears to make no explicit remarks regarding the objection to the specification, though an amendment to resolve the identified objection has been submitted.
Applicant’s arguments, see pgs. 10-11, filed 03/13/2026, with respect to “35 U.S.C. 112 Rejections” have been fully considered and are persuasive. The rejections of claims 2-5, 11-13, and 15-17 under 35 U.S.C. 112(b) have been withdrawn.
Applicant’s arguments, see pgs. 11-16, filed 03/13/2026, with respect to the rejection(s) of claim(s) 1, 14, and 21 under 35 U.S.C. 102(a)(2) (claims 1, 14)/103 (claim 21) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Zheng (US-12050882-B2). Zheng discloses “Representation learning for text and speech has improved many language-related tasks. However, existing methods only learn from one input modality, while a unified representation for both speech and text is needed for tasks such as end-to-end speech translation. Consequently, these methods cannot exploit various large-scale text and speech data and their performance is limited by the scarcity of parallel speech translation data. To address these problems, embodiments of a fused acoustic and text masked language model (FAT-MLM) are disclosed. FAT-MLM embodiments jointly learn a unified representation for both acoustic and text input from various types of corpora including parallel data for speech recognition and machine translation, and pure speech and text data” (abstract). Zheng was previously referenced for disclosing elements of claims 5 and 22. See updated rejections below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Qian et al. (US-20240127001-A1), hereinafter Qian, in view of Zheng et al. (US-12050882-B2), hereinafter Zheng.
Regarding claim 1, Qian discloses: a method for enhancing speech modality in a large language model (LLM) ([0008] techniques for transferring few-shot learning ability to the audio-text setting, [0030] the autoregressive language model is a general-purpose learner containing the text embedder such as generative pre-trained Transformer 2 (GPT-2) which is a neural network machine learning model trained using internet data that translates text, answers questions, summarizes passages, and generates text output, [Wherein audio-text is a multimodality setting to be enhanced by Qian and a GPT language model will inherently be large in the context of solving multiple language tasks]), the method comprising:
obtaining a first set of training data comprising tuples of a sample of speech combined with synthetically generated pairings of speech comprehension test questions and answers (SQA) that correspond to the sample of speech ([0032] the audio encoder is trained as part of an automatic speech recognition system with the goal being that the audio encoder learns to convert speech or non-speech audio utterances in the audio demonstration tasks into embeddings digestible by the autoregressive language model. As highlighted above, the audio understanding task demonstration are each in the form of a triplet containing an audio utterance, a text question/prompt, and a text answer, [An audio encoder responsible for generating speech into embeddings digestible by an autoregressive language model, i.e. text embeddings (see [0037]), indicates the embeddings are representative of synthetically generated (by the audio encoder, i.e. generating embeddings using a digital, e.g. synthetic, processing element, see [0033] defining the audio encoder to be a multi-layer convolutional neural network, necessarily requires the embeddings output to by synthetic) pairings (paired with respect to the triple, reasonably understood to be synonymous with a tuple) of speech comprehension test questions and answers that correspond to the sample of speech received by the encoder. Further, training the audio encoder to perform this task indicates a required set of training data received for the training operation]);
obtaining a second set of training data comprising pairings of automatic speech recognition (ASR) data ([0042] the audio encoder is pretrained as part of an automatic speech recognition system using publicly available datasets, so that the audio encoder learns to convert the audio utterances in the audio understanding task demonstrations (e.g., in the form of triplets including an audio utterance, a text prompt and a text answer), [The automatic speech recognition data is comprised of pairings of speech to text]);
generating a first set of encodings of the first set of training data ([0037] the text embedder converts the text question/prompt(s) and the text answer(s) into text embeddings, [Embeddings generated through a text embedder to be combined with audio encodings (see Fig. 3B) indicates the text embeddings to be functionally equivalent to encodings]) and a second set of encodings of the second set of training data ([0029] The audio encoder is pretrained as part of an automatic speech recognition system, so that it learns to convert the audio in text answer demonstrations into embeddings that are understandable to the autoregressive language model, [Embeddings generated through an audio encoder indicates the embeddings to be encodings]).
Qian does not disclose:
aligning the first set of encodings with the second set of encodings by adjusting one or more parameters of the LLM to minimize the difference between the first set of encodings, the second set of encodings, and one or more internal representations of the first set of training data and the second set of training data used by the LLM.
Zheng discloses:
aligning the first set of encodings with the second set of encodings by adjusting one or more parameters of the LLM to minimize the difference between the first set of encodings, the second set of encodings, and one or more internal representations of the first set of training data and the second set of training data used by the LLM ([Col. 8, Lines 30-35] To fully utilize the corpora for different tasks, FAT-MLM may take any combination of speech, transcription, translation triplets D.sub.2.sub.{s,x,y} as input, which is the power set of {s,x,y} triplets, [Col. 10, Lines 45-50] speech translation is enabled to encode both acoustic and text features as input by simply adapting the architecture of monolingual FAT-MLM to a fused acoustic and text speech translation (FAT-ST) model, [Col. 10, Lines 60-67], [Col. 11, Lines 1-5] In one or more embodiments, the acoustic embeddings 902 may be masked acoustic embeddings when the FAT-ST model is running in a training procedure or original acoustic embeddings without any masks when the FAT-ST model is deployed for inference. The representation 912 may be an acoustic representation when the transformer encoder receives only acoustic embeddings 902, or a unified representation when the transformer encoder receives both the acoustic embeddings 902 and the text embeddings 904. In one or more embodiments, positional embeddings 906 may be used to align the text embeddings 904 for transcription alignment., [Col. 11, Lines 55-60] A final FAT-ST loss function may then be obtained (1015) based on a combination of the direct speech translation loss, the machine translation loss, and the FAT-MLM loss, [Col. 11, Lines 65-67] One or more model parameters of the FAT-ST model may be optimized or updated (1020) using the FAT-ST loss function, [Updating/optimizing model parameters using a loss function indicates the loss function to be minimized in order to be properly optimized (see loss minimization [Col. 10, Lines 25-30]), wherein parameters track to internal representations of training data used by the LLM. Further, consider the triplet data being sent to the FAT-MLM comprising acoustic and text features (in view of the triplets of Qian) indicating that a loss function which updates model parameters based on FAT-MLM loss is an alignment of the first set of encodings, i.e. based on the triples the FAT-MLM receives, with the second set of encodings (transcription, i.e. ASR, and/or translation data as gathered from the translation loss(es)) based on updating the model employing the loss function, i.e. the FAT-ST which is adapted from/is comprised of the FAT-MLM]).
Qian and Zheng are considered analogous art within speech processing using large language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian to incorporate the teachings of Zheng, because of the novel way to generate a unified representation for both speech and text for performing tasks such as end-to-end speech translation, allowing for the exploitation of large-scale text and speech data for improved cross-modal model performance and translation quality (Zheng, Abstract, [Col. 1, Lines 15-35]).
Qian further discloses:
training the LLM on a greater amount of the first set of training data than the second set of training data ([Fig. 3A, Number of embeddings resulting from the audio encoder and text embedder respectively], [0044] In the illustrative, non-limiting example shown in FIG. 3A, the question ‘what did the speaker say?’ is used as a prompt during pretraining. In that case, the output from the autoregressive language model must then match the audio utterance of the speaker, e.g., ‘to catch a glimpse of the expected train’ in order to validate the training of the audio encoder, [0030] gradients are backpropagated through the autoregressive language model, [0047] Optionally, prior to inference, the autoregressive language model can be calibrated to maximize its performance, [Comparing the number of embeddings output from the first set of training data, i.e. text embedder representing question-answer pairs, (13 embeddings) to the number of embeddings output from the second set of training data, i.e. ASR data, (4 embeddings) indicates that the training of the Autoregressive Language Model will be trained on a greater amount of the first set of training data than the second set (with respect to the total number of embeddings represented) through performance maximization calibration (reasonably understood to be consisting of training). The examiner would like to note that [0047] of Qian discloses that the autoregressive language model “does not need to change the (fixed) parameters of the…model” (emphasis added to underlined portion), though this disclosure does not necessarily require the parameters to remain fixed. Calibrating different output distributions indicates some change in the model producing the output, indicative of training]); and,
using the trained LLM to perform a natural language processing task ([0038] the autoregressive language model has to answer the question using the form specified in the audio understanding task demonstrations, assuming that at least one audio understanding task demonstration is included in the prompt sequence, [In view of the examiner’s previous assertion that the autoregressive language model is functionally equivalent to a LLM, audio understanding task is a natural language processing task]).
Regarding claim 14, Qian discloses:
a system ([0035] The system with its (now fixed) pretrained audio encoder can then be used for performing audio understanding tasks) comprising:
one or more processors ([0063] processor set 110 (including processing circuitry 120); and
a hardware storage system storing computer-executable instructions that are executable by the one or more processors ([0062] A “storage device” is any tangible device that can retain and store instructions for use by a computer processor) for causing the system to perform a method for enhancing speech modality in a large language model (LLM) ([0008] techniques for transferring few-shot learning ability to the audio-text setting, [0030] the autoregressive language model is a general-purpose learner containing the text embedder such as generative pre-trained Transformer 2 (GPT-2) which is a neural network machine learning model trained using internet data that translates text, answers questions, summarizes passages, and generates text output, [Wherein audio-text is a multimodality setting to be enhanced by Qian and a GPT language model will inherently be large in the context of solving multiple language tasks]), the method comprising:
obtaining a first set of training data comprising tuples of a sample of speech combined with synthetically generated pairings of speech comprehension test questions and answers (SQA) that correspond to the sample of speech ([0032] the audio encoder is trained as part of an automatic speech recognition system with the goal being that the audio encoder learns to convert speech or non-speech audio utterances in the audio demonstration tasks into embeddings digestible by the autoregressive language model. As highlighted above, the audio understanding task demonstration are each in the form of a triplet containing an audio utterance, a text question/prompt, and a text answer, [An audio encoder responsible for generating speech into embeddings digestible by an autoregressive language model, i.e. text embeddings (see [0037]), indicates the embeddings are representative of synthetically generated (by the audio encoder, i.e. generating embeddings using a digital, e.g. synthetic, processing element, see [0033] defining the audio encoder to be a multi-layer convolutional neural network, necessarily requires the embeddings output to by synthetic) pairings (paired with respect to the triple, reasonably understood to be synonymous with a tuple) of speech comprehension test questions and answers that correspond to the sample of speech received by the encoder. Further, training the audio encoder to perform this task indicates a required set of training data received for the training operation]);
obtaining a second set of training data comprising pairings of automatic speech recognition (ASR) data ([0042] the audio encoder is pretrained as part of an automatic speech recognition system using publicly available datasets, so that the audio encoder learns to convert the audio utterances in the audio understanding task demonstrations (e.g., in the form of triplets including an audio utterance, a text prompt and a text answer), [The automatic speech recognition data is comprised of pairings of speech to text]);
generating a first set of encodings of the first set of training data ([0037] the text embedder converts the text question/prompt(s) and the text answer(s) into text embeddings, [Embeddings generated through a text embedder to be combined with audio encodings (see Fig. 3B) indicates the text embeddings to be functionally equivalent to encodings]) and a second set of encodings of the second set of training data ([0029] The audio encoder is pretrained as part of an automatic speech recognition system, so that it learns to convert the audio in text answer demonstrations into embeddings that are understandable to the autoregressive language model, [Embeddings generated through an audio encoder indicates the embeddings to be encodings]).
Qian does not disclose:
aligning the first set of encodings with the second set of encodings by adjusting one or more parameters of the LLM to minimize the difference between the first set of encodings, the second set of encodings, and one or more internal representations of the first set of training data and the second set of training data used by the LLM.
Zheng discloses:
aligning the first set of encodings with the second set of encodings by adjusting one or more parameters of the LLM to minimize the difference between the first set of encodings, the second set of encodings, and one or more internal representations of the first set of training data and the second set of training data used by the LLM ([Col. 8, Lines 30-35] To fully utilize the corpora for different tasks, FAT-MLM may take any combination of speech, transcription, translation triplets D.sub.2.sub.{s,x,y} as input, which is the power set of {s,x,y} triplets, [Col. 10, Lines 45-50] speech translation is enabled to encode both acoustic and text features as input by simply adapting the architecture of monolingual FAT-MLM to a fused acoustic and text speech translation (FAT-ST) model, [Col. 10, Lines 60-67], [Col. 11, Lines 1-5] In one or more embodiments, the acoustic embeddings 902 may be masked acoustic embeddings when the FAT-ST model is running in a training procedure or original acoustic embeddings without any masks when the FAT-ST model is deployed for inference. The representation 912 may be an acoustic representation when the transformer encoder receives only acoustic embeddings 902, or a unified representation when the transformer encoder receives both the acoustic embeddings 902 and the text embeddings 904. In one or more embodiments, positional embeddings 906 may be used to align the text embeddings 904 for transcription alignment., [Col. 11, Lines 55-60] A final FAT-ST loss function may then be obtained (1015) based on a combination of the direct speech translation loss, the machine translation loss, and the FAT-MLM loss, [Col. 11, Lines 65-67] One or more model parameters of the FAT-ST model may be optimized or updated (1020) using the FAT-ST loss function, [Updating/optimizing model parameters using a loss function indicates the loss function to be minimized in order to be properly optimized (see loss minimization [Col. 10, Lines 25-30]), wherein parameters track to internal representations of training data used by the LLM. Further, consider the triplet data being sent to the FAT-MLM comprising acoustic and text features (in view of the triplets of Qian) indicating that a loss function which updates model parameters based on FAT-MLM loss is an alignment of the first set of encodings, i.e. based on the triples the FAT-MLM receives, with the second set of encodings (transcription, i.e. ASR, and/or translation data as gathered from the translation loss(es)) based on updating the model employing the loss function, i.e. the FAT-ST which is adapted from/is comprised of the FAT-MLM]).
Qian and Zheng are considered analogous art within speech processing using large language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian to incorporate the teachings of Zheng, because of the novel way to generate a unified representation for both speech and text for performing tasks such as end-to-end speech translation, allowing for the exploitation of large-scale text and speech data for improved cross-modal model performance and translation quality (Zheng, Abstract, [Col. 1, Lines 15-35]).
Qian further discloses:
training the LLM on a greater amount of the first set of training data than the second set of training data ([Fig. 3A, Number of embeddings resulting from the audio encoder and text embedder respectively], [0044] In the illustrative, non-limiting example shown in FIG. 3A, the question ‘what did the speaker say?’ is used as a prompt during pretraining. In that case, the output from the autoregressive language model must then match the audio utterance of the speaker, e.g., ‘to catch a glimpse of the expected train’ in order to validate the training of the audio encoder, [0030] gradients are backpropagated through the autoregressive language model, [0047] Optionally, prior to inference, the autoregressive language model can be calibrated to maximize its performance, [Comparing the number of embeddings output from the first set of training data, i.e. text embedder representing question-answer pairs, (13 embeddings) to the number of embeddings output from the second set of training data, i.e. ASR data, (4 embeddings) indicates that the training of the Autoregressive Language Model will be trained on a greater amount of the first set of training data than the second set (with respect to the total number of embeddings represented) through performance maximization calibration (reasonably understood to be consisting of training). The examiner would like to note that [0047] of Qian discloses that the autoregressive language model “does not need to change the (fixed) parameters of the…model” (emphasis added to underlined portion), though this disclosure does not necessarily require the parameters to remain fixed. Calibrating different output distributions indicates some change in the model producing the output, indicative of training]); and,
using the trained LLM to perform a natural language processing task ([0038] the autoregressive language model has to answer the question using the form specified in the audio understanding task demonstrations, assuming that at least one audio understanding task demonstration is included in the prompt sequence, [In view of the examiner’s previous assertion that the autoregressive language model is functionally equivalent to a LLM, audio understanding task is a natural language processing task]).
Claim(s) 2-5, 8-9, 15-17, 19, 21-22, 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Qian in view of Zheng, further in view of Balasubramaniam et al. (US-12431131-B1), hereinafter Balasubramaniam.
Regarding claim 2, Qian in view of Zheng discloses: the method of claim 1.
Qian further discloses:
wherein the natural language processing task comprises a speech-to-text task ([0032] the audio encoder learns to convert speech or non-speech audio utterances in the audio demonstration tasks into embeddings digestible by the autoregressive language model, [0053] speech was converted into text using an automatic speech recognition system).
Qian in view of Zheng does not disclose:
wherein the method further comprises:
fine-tuning the LLM to perform the natural language processing task with a single-shot training prompt.
Balasubramaniam discloses:
wherein the method further comprises:
fine-tuning the LLM to perform the natural language processing task with a single-shot training prompt ([Col. 16, Lines 55-60] Few-shot learning may require fewer amount of training data than implementing other fine-tuning techniques. For further example, in some embodiments, the language models may be configured using one-shot learning, [Col. 20, Lines 62-66] In other embodiments, the personalized context component 765 may be/implement an LLM. In such embodiments, the personalized context component 765 may be finetuned on personalized information for one or more users, as is discussed in more detail herein below, [Col. 43, Lines 49-51] task-specific fine-tuning where the pre-trained model is fine-tuned on a specific task using a task-specific dataset, [Wherein tasks performed using a LLM based on user information indicates the tasks to be natural language processing tasks in view of the previously disclosed LLM of Qian which does process specific natural language tasks. Further, single-shot and one-shot trainings are synonymous]).
Qian, Zheng, and Balasubramaniam are considered analogous art within speech processing using large language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Balasubramaniam, because of the novel way to combine speech recognition with natural language understanding processing techniques, enabling improved speech-based user control of computer devices to perform tasks based on the spoken input (Balasubramaniam, [Col. 1, Lines 5-20]).
Regarding claim 3, Qian in view of Zheng, further in view of Balasubramaniam discloses: the method of claim 2.
Balasubramaniam further discloses:
wherein the single-shot training prompt comprises a natural language input ([Col. 16, Lines 34-35] the input to the LLM may be in the form of a prompt. A prompt may be a natural language input, [Col. 16, Lines 57-58] the language models may be configured using one-shot learning).
Regarding claim 4, Qian in view of Zheng, further in view of Balasubramanian discloses: the method of claim 3.
Qian further discloses:
wherein the natural language input comprises a speech or audio sample provided as a reference with the prompt ([0032] the audio understanding task demonstration are each in the form of a triplet containing an audio utterance, a text question/prompt, and a text answer, [Storing a speech/audio sample in the same structure as the prompt indicates the utterance to be provided as a reference with the prompt]).
Regarding claim 5, Qian in view of Zheng, further in view of Balasubramaniam discloses: the method of claim 2.
Zheng further discloses:
wherein the speech-to-text task comprises converting a sample of audio into a translated text ([Col. 9, Lines 58-59] FIG. 8B shows that FAT-MLM is able to learn a clear monotonic speech-to-text cross-modal attention, [Wherein the task of Fig. 8B is clearly translation]).
Regarding claim 8, Qian in view of Zheng discloses: the method of claim 1.
Qian in view of Zheng does not disclose:
wherein the LLM is fine-tuned to perform unseen tasks in a zero-shot setting.
Balasubramaniam discloses:
wherein the LLM is fine-tuned to perform unseen tasks in a zero-shot setting ([Col. 16, Lines 55-65] Few-shot learning may require fewer amount of training data than implementing other fine-tuning techniques. As another example, in some embodiments, the language models may be configured using zero-shot learning. In zero-shot learning, the model solves the given problem without examples of how to solve the specific/similar problem and just based on the model's training dataset, [Disclosing few-shot to be a fine-tuning technique with a further example being zero-shot indicates zero-shot learning to be a fine-tuning technique]).
Qian, Zheng, and Balasubramaniam are considered analogous art within speech processing using large language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Balasubramaniam, because of the novel way to combine speech recognition with natural language understanding processing techniques, enabling improved speech-based user control of computer devices to perform tasks based on the spoken input (Balasubramaniam, [Col. 1, Lines 5-20]).
Regarding claim 9, Qian in view of Zheng discloses: the method of claim 1.
Qian in view of Zheng does not disclose:
wherein the LLM is fine-tuned to perform unseen tasks in a few-shot setting.
Balasubramaniam discloses:
wherein the LLM is fine-tuned to perform unseen tasks in a few-shot setting ([Col. 9, Lines 48-50] the language models may be configured using few-shot learning).
Qian, Zheng, and Balasubramaniam are considered analogous art within speech processing using large language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Balasubramaniam, because of the novel way to combine speech recognition with natural language understanding processing techniques, enabling improved speech-based user control of computer devices to perform tasks based on the spoken input (Balasubramaniam, [Col. 1, Lines 5-20]).
Regarding claim 15, Qian in view of Zheng discloses: the system of claim 14.
Qian further discloses:
wherein the natural language processing task comprises a speech-to-text task ([0032] the audio encoder learns to convert speech or non-speech audio utterances in the audio demonstration tasks into embeddings digestible by the autoregressive language model, [0053] speech was converted into text using an automatic speech recognition system).
Qian in view of Zheng does not disclose:
wherein the method further comprises:
fine-tuning the LLM to perform the natural language processing task with a single-shot training prompt.
Balasubramaniam discloses:
wherein the method further comprises:
fine-tuning the LLM to perform the natural language processing task with a single-shot training prompt ([Col. 16, Lines 55-60] Few-shot learning may require fewer amount of training data than implementing other fine-tuning techniques. For further example, in some embodiments, the language models may be configured using one-shot learning, [Col. 20, Lines 62-66] In other embodiments, the personalized context component 765 may be/implement an LLM. In such embodiments, the personalized context component 765 may be finetuned on personalized information for one or more users, as is discussed in more detail herein below, [Col. 43, Lines 49-51] task-specific fine-tuning where the pre-trained model is fine-tuned on a specific task using a task-specific dataset, [Wherein tasks performed using a LLM based on user information indicates the tasks to be natural language processing tasks in view of the previously disclosed LLM of Qian which does process specific natural language tasks. Further, single-shot and one-shot trainings are synonymous]).
Qian, Zheng, and Balasubramaniam are considered analogous art within speech processing using large language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Balasubramaniam, because of the novel way to combine speech recognition with natural language understanding processing techniques, enabling improved speech-based user control of computer devices to perform tasks based on the spoken input (Balasubramaniam, [Col. 1, Lines 5-20]).
Regarding claim 16, Qian in view of Zheng, further in view of Balasubramaniam discloses: the system of claim 15.
Balasubramaniam further discloses:
wherein the prompt comprises a natural language input ([Col. 16, Lines 34-35] the input to the LLM may be in the form of a prompt. A prompt may be a natural language input, [Col. 16, Lines 57-58] the language models may be configured using one-shot learning).
Regarding claim 17, Qian in view of Zheng, further in view of Balasubramanian discloses: the system of claim 16.
Qian further discloses:
wherein the natural language input comprises a speech or audio sample provided as a reference with the prompt ([0032] the audio understanding task demonstration are each in the form of a triplet containing an audio utterance, a text question/prompt, and a text answer, [Storing a speech/audio sample in the same structure as the prompt indicates the utterance to be provided as a reference with the prompt]).
Regarding claim 19, Qian in view of Zheng discloses: the system of claim 14.
Qian in view of Zheng does not disclose:
wherein the LLM is fine-tuned to perform unseen tasks in a zero-shot setting.
Balasubramaniam discloses:
wherein the LLM is fine-tuned to perform unseen tasks in a zero-shot setting ([Col. 16, Lines 55-65] Few-shot learning may require fewer amount of training data than implementing other fine-tuning techniques. As another example, in some embodiments, the language models may be configured using zero-shot learning. In zero-shot learning, the model solves the given problem without examples of how to solve the specific/similar problem and just based on the model's training dataset, [Disclosing few-shot to be a fine-tuning technique with a further example being zero-shot indicates zero-shot learning to be a fine-tuning technique]).
Qian, Zheng, and Balasubramaniam are considered analogous art within speech processing using large language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Balasubramaniam, because of the novel way to combine speech recognition with natural language understanding processing techniques, enabling improved speech-based user control of computer devices to perform tasks based on the spoken input (Balasubramaniam, [Col. 1, Lines 5-20]).
Regarding claim 21, Qian discloses: A method for using a large language model (LLM) to perform an unseen task ([0035] For instance, in step 13 a prompt sequence is received that contains few audio understanding task demonstrations (few-shot) or no audio understanding task demonstrations (zero-shot) of a new task, [0030] the autoregressive language model is a general-purpose learner containing the text embedder such as generative pre-trained Transformer 2 (GPT-2) which is a neural network machine learning model trained using internet data that translates text, answers questions, summarizes passages, and generates text output), the method comprising:
obtaining an LLM that was (i) initially trained on a mono-lingual task-independent training dataset ([0030] the autoregressive language model is a general-purpose learner containing the text embedder such as generative pre-trained Transformer 2 (GPT-2) which is a neural network machine learning model trained using internet data, [Using internet data, wherein internet data is reasonably understood to be containing unlabeled, i.e. task-independent, data (while Qian makes no reference to translation other than stating is as a functionality of GPT models) indicates the data to be mono-lingual task-independent]), (ii) subsequently trained on a combination of automatic speech recognition (ASR) training data and speech comprehension (SQA) training data ([0042] the audio encoder is pretrained as part of an automatic speech recognition system using publicly available datasets, [0044] the output from the autoregressive language model must then match the audio utterance of the speaker, e.g., ‘to catch a glimpse of the expected train’ in order to validate the training of the audio encoder, [Wherein training of the audio encoder based on a comparison to LLM output, i.e. comparing the speech comprehension results from the LLM (autoregressive language model) to an audio encoding, e.g. ASR data, for validation indicates a required training of the LLM based on a comparison of these results, see backpropagation of autoregressive language model, [0042]]),
the LLM being trained on a greater amount of the SQA training data than the ASR training data ([Fig. 3A, Number of embeddings resulting from the audio encoder and text embedder respectively], [0044] In the illustrative, non-limiting example shown in FIG. 3A, the question ‘what did the speaker say?’ is used as a prompt during pretraining. In that case, the output from the autoregressive language model must then match the audio utterance of the speaker, e.g., ‘to catch a glimpse of the expected train’ in order to validate the training of the audio encoder, [0030] gradients are backpropagated through the autoregressive language model, [0047] Optionally, prior to inference, the autoregressive language model can be calibrated to maximize its performance, [Comparing the number of embeddings output from the first set of training data, i.e. text embedder representing question-answer pairs, (13 embeddings) to the number of embeddings output from the second set of training data, i.e. ASR data, (4 embeddings) indicates that the training of the Autoregressive Language Model will be trained on a greater amount of the first set of training data than the second set (with respect to the total number of embeddings represented) through performance maximization calibration (reasonably understood to be consisting of training). The examiner would like to note that [0047] of Qian discloses that the autoregressive language model “does not need to change the (fixed) parameters of the…model” (emphasis added to underlined portion), though this disclosure does not necessarily require the parameters to remain fixed. Calibrating different output distributions indicates some change in the model producing the output, indicative of training]).
Qian does not disclose:
the SQA training data having been aligned with the ASR training data prior to the training of the LLM with the combination of the SQA training data and the ASR training data, the aligning occurring by adjusting one or more parameters of the LLM to minimize the difference between the SQA training data, the ASR training data and one or more internal representations of the ASR training data and the SQA training data used by the LLM.
Zheng discloses:
the SQA training data having been aligned with the ASR training data prior to the training of the LLM with the combination of the SQA training data and the ASR training data, the aligning occurring by adjusting one or more parameters of the LLM to minimize the difference between the SQA training data, the ASR training data and one or more internal representations of the ASR training data and the SQA training data used by the LLM ([Col. 8, Lines 30-35] To fully utilize the corpora for different tasks, FAT-MLM may take any combination of speech, transcription, translation triplets D.sub.2.sub.{s,x,y} as input, which is the power set of {s,x,y} triplets, [Col. 10, Lines 45-50] speech translation is enabled to encode both acoustic and text features as input by simply adapting the architecture of monolingual FAT-MLM to a fused acoustic and text speech translation (FAT-ST) model, [Col. 10, Lines 60-67], [Col. 11, Lines 1-5] In one or more embodiments, the acoustic embeddings 902 may be masked acoustic embeddings when the FAT-ST model is running in a training procedure or original acoustic embeddings without any masks when the FAT-ST model is deployed for inference. The representation 912 may be an acoustic representation when the transformer encoder receives only acoustic embeddings 902, or a unified representation when the transformer encoder receives both the acoustic embeddings 902 and the text embeddings 904. In one or more embodiments, positional embeddings 906 may be used to align the text embeddings 904 for transcription alignment., [Col. 11, Lines 55-60] A final FAT-ST loss function may then be obtained (1015) based on a combination of the direct speech translation loss, the machine translation loss, and the FAT-MLM loss, [Col. 11, Lines 65-67] One or more model parameters of the FAT-ST model may be optimized or updated (1020) using the FAT-ST loss function, [Updating/optimizing model parameters using a loss function indicates the loss function to be minimized in order to be properly optimized (see loss minimization [Col. 10, Lines 25-30]), wherein parameters track to internal representations of training data used by the LLM. Further, consider the triplet data being sent to the FAT-MLM comprising acoustic and text features (in view of the triplets of Qian) indicating that a loss function which updates model parameters based on FAT-MLM loss is an alignment of the first set of encodings, i.e. based on the triples the FAT-MLM receives, with the second set of encodings (transcription, i.e. ASR, and/or translation data as gathered from the translation loss(es)) based on updating the model employing the loss function, i.e. the FAT-ST which is adapted from/is comprised of the FAT-MLM]).
Qian and Zheng are considered analogous art within speech processing using large language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian to incorporate the teachings of Zheng, because of the novel way to generate a unified representation for both speech and text for performing tasks such as end-to-end speech translation, allowing for the exploitation of large-scale text and speech data for improved cross-modal model performance and translation quality (Zheng, Abstract, [Col. 1, Lines 15-35]).
Qian further discloses:
providing a new input and new instructional prompt to perform the unseen task on the new input to cause the LLM to generate a corresponding output for the new input ([0055] language models can perform zero-shot learning on speech understanding tasks, [Zero-shot learning indicates a new input, prompt, and unseen task to qualify as zero-shot in view of the previously disclosed question answering output functionality of Qian]); and
generate the corresponding output for the new input based on performing the previously unseen task ([0038] The new question similarly contains a new audio utterance and new text question/prompt, but no new answer. In that case, the autoregressive language model would be tasked with providing a text answer).
Qian in view of Zheng does not disclose:
obtaining an LLM that was (iii) then fine-tuned using a one-shot training data sample comprising an input-output pair and instructional prompt representing an unseen task.
Balasubramaniam discloses:
obtaining an LLM that was (iii) then fine-tuned using a one-shot training data sample comprising an input-output pair and instructional prompt representing an unseen task ([Col. 16, Lines 55-60] Few-shot learning may require fewer amount of training data than implementing other fine-tuning techniques. For further example, in some embodiments, the language models may be configured using one-shot learning, [Col. 20, Lines 62-66] In other embodiments, the personalized context component 765 may be/implement an LLM. In such embodiments, the personalized context component 765 may be finetuned on personalized information for one or more users, as is discussed in more detail herein below, [Col. 43, Lines 49-51] task-specific fine-tuning where the pre-trained model is fine-tuned on a specific task using a task-specific dataset, [Wherein tasks performed using a LLM based on user information indicates the tasks to be natural language processing tasks in view of the previously disclosed LLM of Qian which does process specific natural language tasks. Further, single-shot and one-shot trainings are synonymous]).
Qian, Zheng, and Balasubramaniam are considered analogous art within speech processing using large language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Balasubramaniam, because of the novel way to combine speech recognition with natural language understanding processing techniques, enabling improved speech-based user control of computer devices to perform tasks based on the spoken input (Balasubramaniam, [Col. 1, Lines 5-20]).
Regarding claim 22, Qian in view of Zheng, further in view of Balasubramaniam discloses: the method of claim 21.
Zheng further discloses:
wherein the unseen task is machine translation of audio in a first language represented in the mono-lingual task-independent training dataset and combination of automatic speech recognition training data and speech comprehension training data to a text-based transcription in a second language represented in the output of the one-shot training data sample ([Col. 9, Lines 58-59] FIG. 8B shows that FAT-MLM is able to learn a clear monotonic speech-to-text cross-modal attention, [In view of the previously disclosed unseen tasks and training datasets of Qian in view of Balasubramaniam. Wherein the task of Fig. 8B is clearly translation. Further, wherein speech-to-text is a well-known method of automatic speech recognition indicating the translation of Fig. 8A to be a combination of automatic speech recognition training data, i.e. “and you know what?” and speech comprehension training data, i.e. the translations generated, e.g. “und wissen sie was?”, represented in output]).
Regarding claim 24, Qian in view of Zheng discloses: the method of claim 1.
Qian in view of Zheng does not disclose:
dynamically obtaining new ASR training data and new SQA training data; and,
continuously and dynamically updating training of the LLM using the new AST training data and the new SQA training data.
Balasubramaniam discloses:
dynamically obtaining new ASR training data and new SQA training data ([Fig. 4B, Encode new portions of prompt 424], [Col. 11, Lines 50-60] Referring to FIG. 4B, next few steps described here may be performed on a loop until a Response step is generated by the LLM container 150. The LLM orchestrator 130 may instruct (420) the LLM container 150 to respond to the first utterance, where such instruction is provided via a prompt. This prompt may include the prompt portion previously sent (406) to the LLM container 150 for encoding and may also include the first transcription representing the first utterance. The prompt may also include other information that may be available for processing, [Providing a prompt to a model on a loop, wherein a current iteration of a prompt contains a last prompt AND additional, new information, indicates the updated prompt per iteration to be obtained new training data, wherein the data is clearly in a QA system (see Fig. 6A), and includes a transcribed audio indicating ASR data]); and,
continuously and dynamically updating training of the LLM using the new ASR training data and the new SQA training data ([Fig. 4B, Perform Iterative Processing 426], [Col. 12, Lines 5-10] LLM container 150 may perform (426) iterative processing using the encoded prompt—the encoded new portions of the prompt and the encoded prompt portion retrieved from the cache 170, [Combining newly encoded prompt portions with previously encoded portions gathered from a cache indicates the combination to be a dynamically updated prompt for training, continuously, i.e. iteratively, processed, wherein the training data consists of SQA and ASR data as previously disclosed]).
Qian, Zheng, and Balasubramaniam are considered analogous art within speech processing using large language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Balasubramaniam, because of the novel way to combine speech recognition with natural language understanding processing techniques, enabling improved speech-based user control of computer devices to perform tasks based on the spoken input (Balasubramaniam, [Col. 1, Lines 5-20]).
Zheng further discloses:
aligning the new SQA training data with the new ASR training data ([Col. 8, Lines 30-35] To fully utilize the corpora for different tasks, FAT-MLM may take any combination of speech, transcription, translation triplets D.sub.2.sub.{s,x,y} as input, which is the power set of {s,x,y} triplets, [Col. 10, Lines 45-50] speech translation is enabled to encode both acoustic and text features as input by simply adapting the architecture of monolingual FAT-MLM to a fused acoustic and text speech translation (FAT-ST) model, [Col. 10, Lines 60-67], [Col. 11, Lines 1-5] In one or more embodiments, the acoustic embeddings 902 may be masked acoustic embeddings when the FAT-ST model is running in a training procedure or original acoustic embeddings without any masks when the FAT-ST model is deployed for inference. The representation 912 may be an acoustic representation when the transformer encoder receives only acoustic embeddings 902, or a unified representation when the transformer encoder receives both the acoustic embeddings 902 and the text embeddings 904. In one or more embodiments, positional embeddings 906 may be used to align the text embeddings 904 for transcription alignment., [Col. 11, Lines 55-60] A final FAT-ST loss function may then be obtained (1015) based on a combination of the direct speech translation loss, the machine translation loss, and the FAT-MLM loss, [Col. 11, Lines 65-67] One or more model parameters of the FAT-ST model may be optimized or updated (1020) using the FAT-ST loss function, [In view of the iteratively updating data of Balasubrmaniam. Consider the triplet data being sent to the FAT-MLM comprising acoustic and text features (in view of the triplets of Qian) indicating that a loss function which updates model parameters based on FAT-MLM loss is an alignment of the first set of encodings, i.e. based on the triples containing SQA data which the FAT-MLM receives, with the second set of encodings (transcription, i.e. ASR, and/or translation data as gathered from the translation loss(es)) based on updating the model employing the loss function, i.e. the FAT-ST which is adapted from/comprises the FAT-MLM]).
Claim(s) 6-7, 10-11, 18, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Qian in view of Zheng, further in view of Zhao et al. (US-11244167-B2), hereinafter Zhao.
Regarding claim 6, Qian in view of Zheng discloses: the method of claim 1.
Qian in view of Zheng does not disclose:
generating the synthetic speech comprehension test questions and answers based on transcripts of the sample of speech using a generative machine learning model.
Zhao discloses:
generating the synthetic speech comprehension test questions and answers based on transcripts of the sample of speech using a generative machine learning model ([Col. 4, Lines 1-5] the query-response system may convert the question from uttered speech to digital text (e.g., via a speech-to-text mechanism), [Col. 5, Lines 4-7] the query-response system can utilize the transcript layers to generate textual-context vectors representing the transcript for corresponding frames in the video segment, [Col. 5, Lines 63-65] the query-response system utilizes response-network layers of the query-response-neural network to generate the candidate-response vectors, [Col. 8, Lines 28-30] the question-network layers can intelligently generate query vectors (representing a question received during display or playback of a video segment, [Generating candidate response and query vectors from a neural network, see [Col. 8, Lines 5-25] defining the neural network to be machine learning model which will clearly be received natural language input questions/queries, indicating this model to be applicable to the LLM of Qian, based on received transcripts of input questions indicates the response/query vectors to be synthetically generated, i.e. by the neural network/LLM, comprehension test question and answers. See Fig. 4, “Open Dialogues” 432, plurality emphasized, indicating at least one test question and answer]).
Qian, Zheng, and Zhao are considered analogous art within question answering with language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Zhao, because of the novel way to incorporate query context into a multimodal query vector representing an original question to be answered by a computing system, improving the quality and amount of responses generated by that computing system based on a domain-knowledge base or other data source (Zhao, [Col. 2, Lines 10-35]).
Regarding claim 7, Qian in view of Zheng discloses: the method of claim 1.
Qian in view of Zheng does not disclose:
wherein the synthetic speech comprehension test questions and answers form a one-to-many mapping from input speech to target text, thereby enhancing alignment between the speech modality and the text modality.
Zhao discloses:
wherein the synthetic speech comprehension test questions and answers form a one-to-many mapping from input speech to target text, thereby enhancing alignment between the speech modality and the text modality ([Col. 5, Lines 63-65] the query-response system utilizes response-network layers of the query-response-neural network to generate the candidate-response vectors, [Col. 8, Lines 28-30] the question-network layers can intelligently generate query vectors (representing a question received during display or playback of a video segment, [Col. 4, Lines 5-7] the question-network layers can transform the question into one or more word embeddings or other formats as query vectors., [Generating a plurality of query response vectors and query vectors, wherein each vector represents its own query/response, based on one original query (see Fig. 11, 1108) indicates the mapping of input speech, i.e. received query, to target text, i.e. question/answer, to be a one-to-many mapping]).
Qian, Zheng, and Zhao are considered analogous art within question answering with language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Zhao, because of the novel way to incorporate query context into a multimodal query vector representing an original question to be answered by a computing system, improving the quality and amount of responses generated by that computing system based on a domain-knowledge base or other data source (Zhao, [Col. 2, Lines 10-35]).
Regarding claim 10, Qian in view of Zheng discloses: the method of claim 1.
Qian in view of Zheng does not disclose:
wherein the LLM is fine-tuned to perform domain adaptation based on a single audio example and corresponding text target.
Zhao discloses:
wherein the LLM is fine-tuned to perform domain adaptation based on a single audio example and corresponding text target ([Col. 9, Lines 10-15] in some embodiments, the response-network layers generate the candidate-response vectors utilizing pre-trained vectors based on external domain knowledge to modify, weight, and/or filter candidate responses to a question, [Modifying candidate responses, i.e. text targets, to a received question, i.e. a single audio example, based on external domain knowledge indicates the modification to be a domain adaptation]).
Qian, Zheng, and Zhao are considered analogous art within question answering with language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Zhao, because of the novel way to incorporate query context into a multimodal query vector representing an original question to be answered by a computing system, improving the quality and amount of responses generated by that computing system based on a domain-knowledge base or other data source (Zhao, [Col. 2, Lines 10-35]).
Regarding claim 11, Qian in view of Zheng discloses: the method of claim 1.
Qian in view of Zheng does not disclose:
wherein the LLM is applied to at least two times the SQA training data than the ASR training data.
Zhao discloses:
wherein the LLM is applied to at least two times the SQA training data than the ASR training data ([Col. 4, Lines 1-3] the query-response system may convert the question from uttered speech to digital text (e.g., via a speech-to-text mechanism), [Col. 18, Lines 17-20] the query-response system 106 can, at a matching act 438, compare the query-context vector 428 with the candidate-response vectors 436 to determine the selected response 440, [Fig. 4, Matching Act 438 of candidate response vectors 436 (plurality emphasized) to query-context vector 428], [Col. 18, Lines 35-38 The query-response system 106 may train the neural network using triplets including a question-response pair, [In view of Fig. 4 which is comparing multiple responses 436 to a singular query 428, i.e. that transformed into text via speech-to-text-mechanism, e.g. ASR, indicating that each response being compared to the query vector forms a query-response, i.e. SQA training data, wherein generation of multiple query-answer pairs for one original received question indicates the LLM to be applied to at least two times the SQA training data that ASR training data]).
Qian, Zheng, and Zhao are considered analogous art within question answering with language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Zhao, because of the novel way to incorporate query context into a multimodal query vector representing an original question to be answered by a computing system, improving the quality and amount of responses generated by that computing system based on a domain-knowledge base or other data source (Zhao, [Col. 2, Lines 10-35]).
Regarding claim 18, Qian in view of Zheng discloses: the system of claim 14.
Qian in view of Zheng does not disclose:
wherein the synthetic speech comprehension test questions and answers form a one-to-many mapping from input speech to target text, thereby enhancing alignment between the speech modality and the text modality.
Zhao discloses:
wherein the synthetic speech comprehension test questions and answers form a one-to-many mapping from input speech to target text, thereby enhancing alignment between the speech modality and the text modality ([Col. 5, Lines 63-65] the query-response system utilizes response-network layers of the query-response-neural network to generate the candidate-response vectors, [Col. 8, Lines 28-30] the question-network layers can intelligently generate query vectors (representing a question received during display or playback of a video segment, [Col. 4, Lines 5-7] the question-network layers can transform the question into one or more word embeddings or other formats as query vectors., [Generating a plurality of query response vectors and query vectors, wherein each vector represents its own query/response, based on one original query (see Fig. 11, 1108) indicates the mapping of input speech, i.e. received query, to target text, i.e. question/answer, to be a one-to-many mapping]).
Qian, Zheng, and Zhao are considered analogous art within question answering with language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Zhao, because of the novel way to incorporate query context into a multimodal query vector representing an original question to be answered by a computing system, improving the quality and amount of responses generated by that computing system based on a domain-knowledge base or other data source (Zhao, [Col. 2, Lines 10-35]).
Regarding claim 20, Qian in view of Zheng discloses: the system of claim 14.
Qian in view of Zheng does not disclose:
wherein the LLM is fine-tuned to perform domain adaptation based on a single audio example and corresponding text target.
Zhao discloses:
wherein the LLM is fine-tuned to perform domain adaptation based on a single audio example and corresponding text target ([Col. 9, Lines 10-15] in some embodiments, the response-network layers generate the candidate-response vectors utilizing pre-trained vectors based on external domain knowledge to modify, weight, and/or filter candidate responses to a question, [Modifying candidate responses, i.e. text targets, to a received question, i.e. a single audio example, based on external domain knowledge indicates the modification to be a domain adaptation]).
Qian, Zheng, and Zhao are considered analogous art within question answering with language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Zhao, because of the novel way to incorporate query context into a multimodal query vector representing an original question to be answered by a computing system, improving the quality and amount of responses generated by that computing system based on a domain-knowledge base or other data source (Zhao, [Col. 2, Lines 10-35]).
Claim(s) 12, 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Qian in view of Zheng, further in view of Jiang et al. (“LibriSQA: Advancing Free-form and Open-ended Spoken Question Answering with a Novel Dataset and Framework”), hereinafter Jiang.
Regarding claim 12, Qian in view of Zheng discloses: the method of claim 1.
Qian in view of Zheng does not disclose:
wherein the LLM is applied to at least four times the SQA training data than the ASR training data.
Jiang discloses:
wherein the LLM is applied to at least four times the SQA training data than the ASR training data ([pg. 3, “II. The LibriSQA Dataset”, par. 1] we collect LibriSQA based on LibriSpeech [21]. LibriSpeech, a comprehensive dataset, offers approximately 1000 hours of 16kHz English read speech, [pg. 5, “III. Method”, par. 1] Throughout the training phase, the parameters of the pre-trained speech models remain frozen, with only the linear layer engaging in the training procedure, [pg. 6, “C. The Results of ASR”, Par. 1] The experiments in the second section are conducted to demonstrate the benefits of the SQA task for the ASR task. Our training doesn’t involve any ASR datasets. The results indicate that solely training on the SQA dataset can accomplish the ASR task during inference, though the outcomes are considerably influenced compared to the first section, which is expected, [Only performing partial pre-training on the ASR and only training the SQA indicates the SQA training data to be at least four times the ASR data, i.e. 1000hrs of training data, none of which is applied to the ASR pre-training indicates at least four times the training data being applied to the SQA as compared to during ASR, i.e. 1000 hours to no hours, in view of the pre-training which reasonably tracks to an ASR training]).
Qian, Zheng, and Jiang are considered analogous art within spoken question-answering with speech recognition. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Jiang, because of the novel way to train an LLM to accommodate free-form, open-ended questions through integration of both speech and text into LLMs, obviating the need for ASR modules, improving the capabilities of the LLM to interpret and process speech autonomously (Jiang, [pg. 2, pars. 3-6]).
Regarding claim 13, Qian in view of Zheng discloses: the method of claim 1.
Qian in view of Zheng does not disclose:
wherein the LLM is applied to at least sixteen times the SQA training data than the ASR training data.
Jiang discloses:
wherein the LLM is applied to at least sixteen times the SQA training data than the ASR training data ([pg. 3, “II. The LibriSQA Dataset”, par. 1] we collect LibriSQA based on LibriSpeech [21]. LibriSpeech, a comprehensive dataset, offers approximately 1000 hours of 16kHz English read speech, [pg. 5, “III. Method”, par. 1] Throughout the training phase, the parameters of the pre-trained speech models remain frozen, with only the linear layer engaging in the training procedure, [pg. 6, “C. The Results of ASR”, Par. 1] The experiments in the second section are conducted to demonstrate the benefits of the SQA task for the ASR task. Our training doesn’t involve any ASR datasets. The results indicate that solely training on the SQA dataset can accomplish the ASR task during inference, though the outcomes are considerably influenced compared to the first section, which is expected, [Only performing partial pre-training on the ASR and only training the SQA indicates the SQA training data to be at least four times the ASR data, i.e. 1000hrs of training data, none of which is applied to the ASR pre-training indicates at least sixteen times the training data being applied to the SQA as compared to during ASR, i.e. 1000 hours to no hours, in view of the pre-training which reasonably tracks to an ASR training]).
Qian, Zheng, and Jiang are considered analogous art within spoken question-answering with speech recognition. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng to incorporate the teachings of Jiang, because of the novel way to train an LLM to accommodate free-form, open-ended questions through integration of both speech and text into LLMs, obviating the need for ASR modules, improving the capabilities of the LLM to interpret and process speech autonomously (Jiang, [pg. 2, pars. 3-6]).
Claim(s) 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Qian in view of Zheng, further in view of Balasubramaniam, further in view of Zhao.
Regarding claim 23, Qian in view of Zheng, further in view of Balasubramaniam discloses: the method of claim 21.
Qian in view of Zheng, further in view of Balasubramaniam does not disclose:
wherein the unseen task is domain adaptation to a new domain represented in the one-shot training data sample that is different than a previously seen domain represented in the mono-lingual task-independent training dataset.
Zhao discloses:
wherein the unseen task is domain adaptation to a new domain represented in the one-shot training data sample that is different than a previously seen domain represented in the mono-lingual task-independent training dataset ([Col. 9, Lines 10-15] in some embodiments, the response-network layers generate the candidate-response vectors utilizing pre-trained vectors based on external domain knowledge to modify, weight, and/or filter candidate responses to a question, [Modifying candidate responses, i.e. text targets, to a received question, i.e. a single audio example, based on external domain knowledge indicates the modification to be a domain adaptation]).
Qian, Zheng, Balasubramaniam, and Zhao are considered analogous art within question answering with language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Qian in view of Zheng, further in view of Balasubramaniam to incorporate the teachings of Zhao, because of the novel way to incorporate query context into a multimodal query vector representing an original question to be answered by a computing system, improving the quality and amount of responses generated by that computing system based on a domain-knowledge base or other data source (Zhao, [Col. 2, Lines 10-35]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zhuang et al. (“ToolQA: A Dataset for LLM Question Answering with External Tools”) discloses “Large Language Models (LLMs) have demonstrated impressive performance in various NLP tasks, but they still suffer from challenges such as hallucination and weak numerical reasoning. To overcome these challenges, external tools can be used to enhance LLMs’ question-answering abilities. However, current evaluation methods do not distinguish between questions that can be answered using LLMs’ internal knowledge and those that require external information through tool use. To address this issue, we introduce a new dataset called ToolQA, which is designed to faithfully evaluate LLMs’ ability to use external tools for question answering. Our development of ToolQA involved a scalable, automated process for dataset curation, along with 13 specialized tools designed for interaction with external knowledge in order to answer questions. Importantly, we strive to minimize the overlap between our benchmark data and LLMs’ pre-training data, enabling a more precise evaluation of LLMs’ tool-use reasoning abilities” (abstract). See entire document. The examiner does not believe this qualifies as valid prior art due to the publication date but is being cited for explaining the state of the art.
Gardner et al. (“LLARK: A multimodal Foundation Model for Music”) discloses “an instruction-tuned multimodal model for music understanding. We detail our process for dataset creation, which involves augmenting the annotations of diverse open-source music datasets and converting them to a unified instruction-tuning format. We propose a multimodal architecture for LLARK, integrating a pretrained generative model for music with a pretrained language model. In evaluations on three types of tasks (music understanding, captioning, and reasoning), we show that our model matches or outperforms existing baselines in zero-shot generalization for music understanding, and that humans show a high degree of agreement with the model’s responses in captioning and reasoning tasks.” (abstract). See entire document.
Kim (US-20210012222-A1) discloses “In implementations of answering questions during video playback, a video system can receive a question related to a video at a timepoint of the video during playback of the video, and determine audio sentences of the video that occur within a segment of the video that includes the timepoint. The video system can generate a classification vector from words of the question and the audio sentences, and determine an answer to the question utilizing the classification vector. The video system can obtain answer candidates, and the answer to the question can be selected as one of the answer candidates based on matching the classification vector to one of the answer vectors.” (abstract). Specifically, [0101] discloses training an encoder module using random initialization triplets of questions, answers, and audio sentences, wherein [0109] discloses a matching system based on the triples. See entire document.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THEODORE JOHN WITHEY whose telephone number is (703)756-1754. The examiner can normally be reached Monday - Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571) 272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THEODORE WITHEY/Examiner, Art Unit 2655 /ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655
Prosecution Insights