DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 07/16/2026 and 08/14/2026 were filed in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
1. Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1, “A computerized system” is recited, which is directed to one of the four statutory categories of invention (machine; Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts:
accessing speech data: a person writes down information about speech they hear
…generating aligned embeddings of the speech data based on length matching and latent space matching of the speech data to text data: generating aligned embeddings via length and latent space matching amounts to mathematical concepts
based on the aligned embeddings, generating an input sequence for a Large Language Model (LLM): a person writes down an input sequence based on the aligned embeddings
…generating an output for the speech data: a person writes down a response to speech data (e.g. answers a question in the speech)
Claim 1 does not contain any additional elements which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). The only additional limitations are “A computerized system comprising: one or more computer processors; and computer memory storing computer-usable instructions that, when used by the one or more computer processors, cause the one or more computer processors to perform operations, the operations comprising”, “using a speech-text alignment engine, generating…” “using the LLM and the input sequence, generating…”. These limitations are recited at a high level of generality and amount to mere instructions to implement the judicial exception using a generic computer which, even when viewed in combination with the claim as a whole, do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the abstract ideas. Therefore, claim 1 is directed to abstract ideas.
Claim 1 does not contain any additional elements which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations are mere instructions to implement the judicial exception using a generic computer. Even when viewed in combination with the claim as a whole, mere instructions to implement the judicial exception using a generic computer do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claim 1 is not patent eligible.
Regarding claims 2-7, “The system” is recited, which is directed to one of the four statutory categories of invention (machine; Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite further mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite further mental processes or mathematical concepts:
Claim 2:
captures audio associated with one or more spoken languages and transforms the audio into the speech data comprising speech embeddings representing acoustic features of speech: a person writes down information about speech they hear
Claim 2 contains the additional limitation “wherein a speech encoder captures…”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 3:
wherein generating the aligned embeddings based on the length matching comprises computing alignment information associated with a non-blank token sequence of the speech data and a speech input sequence of the speech data…: computing alignment information amounts to mathematical concepts
Claim 3 contains the additional limitation “using a Connectionist Temporal Classification (CTC) layer”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 4:
wherein the length matching comprises determining a length of non-blank tokens of the speech data and a frame coverage of the non-blank tokens: determining lengths and frame coverages of non-blank tokens amounts to mathematical concepts
Claim 5:
wherein generating the aligned embeddings based on latent space matching comprises generating…the aligned embeddings based on frames corresponding to non-blank tokens of the speech data, the latent space matching is based on vectorized representation of the text data that is transformed into a shared latent space to facilitate alignment with speech data: generating aligned embeddings via latent space matching amounts to mathematical concepts.
Claim 5 contains the additional limitation “using a modified q-former layer”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 6:
wherein the LLM is trained based on speech-derived embeddings and text-derived embeddings associated with token swapping”: training via token swapping amounts to mathematical concepts
Claim 7:
concatenating the aligned embeddings with prompt embeddings to generate the input sequence for the LLM, wherein a prompt embedding is a pre-defined or dynamically generated vector that encodes instructions or context for the LLM: concatenating aligned embeddings and prompt embeddings amounts to mere instructions to implement the judicial exception using a generic computer.
Claims 2-7 do not contain any additional elements which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). As discussed above, the additional limitation amount to mere instructions to implement the judicial exception using a generic computer, which even when viewed in combination with the claims as a whole, do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the abstract ideas. Therefore, claims 2-7 are directed to abstract ideas.
Claims 2-7 do not contain any additional elements which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations amount to mere instructions to implement the judicial exception using a generic computer which, even when viewed in combination with the claims as a whole, do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claims 2-7 are not patent eligible.
Regarding claim 8, “One or more computer-storage media” is recited, which is directed to one of the four statutory categories of invention (article of manufacture, Applicant’s specification provides a special definition for computer-storage media which excludes signals per se (see para. 00149); Step 1: YES). However, the claim limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts:
communicating speech data associated with a speech-based task…receiving a response associated with the speech data and the speech-based task, …generates aligned embeddings of the speech data based on length matching and latent space matching of the speech data to text data, wherein the response is generated … and generating the response to the speech-based task …: a person listens to speech, and generates a response using aligned embeddings; generating aligned embeddings via length and latent space matching amounts to mathematical concepts
Claim 8 contains the additional limitation “One or more computer-storage media having computer-executable instructions embodied thereon that, when executed by a computing system having a processor and memory, cause the processor to perform operations, the operations comprising: …for a client device…wherein the response is associated with a speech-text alignment engine…wherein the response is generated using a Large Language Model (LLM)…generating the response to the speech-based task at the client device”. These limitations are recited at a high level of generality and amount to mere instructions to implement the judicial exception using a generic computer which, even when viewed in combination, do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the abstract ideas. Therefore, claim 8 is directed to abstract ideas.
Claim 8 does not contain any additional elements which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the additional limitation amounts to mere instructions to implement the judicial exception using a generic computer which, even when viewed in combination with the claim as a whole, does not amount to significantly more than the judicial exception as it does not provide an inventive concept. Therefore, claim 8 is not patent eligible.
Regarding claims 9-14, “The media” is recited, which is directed to one of the four statutory categories of invention (article of manufacture; Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite further mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite further mental processes or mathematical concepts:
Claim 9:
wherein the speech-based task is associated with speech translation, speech summarization, or speech-based question answering: a person can translate speech, summarize speech, and can answer questions in speech
Claim 10:
wherein the length matching comprises determining length of non-blank tokens of the speech data and a frame coverage of the non- blank tokens: length matching by determining length of non-blank tokens and frame converage amounts to mathematical concepts
Claim 11:
wherein latent space matching comprises determining frames corresponding to non-blank tokens of the speech data: latent space matching amounts to mathematical concepts
Claim 12:
wherein the LLM is trained on next token prediction loss and CTC loss: determining losses amounts to mathematical concepts
Claim 13:
wherein the LLM is trained based on a token swapping technique where speech-derived embeddings are randomly swapped with corresponding text embeddings: training a model via token swapping amounts to mathematical concepts
Claim 14:
Claim 14 contains the additional limitation “wherein the response is a text-based LLM output that is converted to speech”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claims 9-14 do not contain any additional elements which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). As discussed above, the only additional limitations amount to mere instructions to implement the judicial exception using a generic computer which, even when viewed in combination with the claims as a whole, do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the abstract ideas. Therefore, claims 9-14 recite abstract ideas.
Claims 9-14 do not contain any additional elements which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations amount to mere instructions to implement the judicial exception using a generic computer which, even when viewed in combination with the claims as a whole, do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claims 9-14 are not patent eligible.
Regarding claim 15, “A computer-implemented method” is recited, which is directed to one of the four statutory categories of invention (process; Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts:
accessing Automatic Speech Recognition (ASR) training data, the ASR training data comprising speech data and text data: a person listens and writes down information regarding speech data
…generating alignment information associated with a non-blank token sequence of the speech data and a speech input sequence of the speech data: generating alignment information amounts to a mathematical concept
based on the alignment information and the speech data…generate aligned embeddings: generating aligned embeddings based on alignment information amounts to mathematical concepts
using the aligned embeddings, training a Large Language Model (LLM): training an LLM amounts to mathematical concepts
Claim 15 does not contain any additional elements which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). The only additional limitations are “using a Connectionist Temporal Classification (CTC) layer, generating…”, “using a modified -former layer to generate”, and “deploying the LLM”. These limitations are each recited at a high level of generality and amount to mere instructions to implement the judicial exception using a generic computer which, even when viewed in combination with the claim as a whole, do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the abstract ideas. Therefore, claim 15 is directed to abstract ideas.
Claim 15 does not contain any additional elements which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations are mere instructions to implement the judicial exception using a generic computer which, even when viewed in combination with the claim as a whole, does not amount to significantly more than the judicial exception as it does not provide an inventive concept. Therefore, claim 15 is not patent eligible.
Regarding claims 16-20, “The method” is recited, which is directed to one of the four statutory categories of invention (process; Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite further mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite further mental processes or mathematical concepts:
Claim 16:
…length matching comprising determining a length of non-blank tokens of the speech data and a frame coverage of the non-blank tokens: determining a length and frame coverage amounts to mathematical concepts
Claim 16 recites the additional limitation “wherein the CTC supports…”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 17:
…latent space matching comprising determining frames corresponding to non-blank tokens of the speech data: performing latent space matching amounts to mathematical concepts
Claim 17 recites the additional limitation “wherein the modified q-former layer supports…”.
Claim 18:
wherein training the LLM is based on next token prediction loss and CTC loss: training on loss functions amounts to mathematical concepts
Claim 19:
wherein training the LLM comprises a token swapping technique where speech-derived embeddings are randomly swapped with corresponding text embeddings: training via token swapping amounts to mathematical concepts
Claim 20:
concatenating the aligned embeddings with prompt embeddings to generate an input sequence for the LLM, wherein a prompt embedding is a pre-defined or dynamically generated vector that encodes instructions or context for the LLM: concatenating aligned embeddings and prompt embeddings amounts to mathematical concepts
Claims 16-20 do not contain any additional elements which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). As discussed above, the only additional limitations amount to mere instructions to implement the judicial exception using a generic computer which, even when viewed in combination with the claims as a whole, do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the abstract ideas. Therefore, claims 16-20 recite abstract ideas.
Claims 16-20 do not contain any additional elements which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations amount to mere instructions to implement the judicial exception using a generic computer which, even when viewed in combination with the claims as a whole, do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claims 16-20 are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
2. Claims 1-5, 8-11, and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Tang et al. (NPL “SALMONN: Towards Generic Hearing Abilities for Large Language Models”) in view of Jin et al. (NPL “CTC-aligned Audio-Text Embeddings for Streaming Open-Vocabulary Keyword Spotting”, hereinafter Jin).
Regarding claim 1, Tang discloses A computerized system comprising: one or more computer processors; and computer memory storing computer-usable instructions that, when used by the one or more computer processors, cause the one or more computer processors to perform operations, the operations comprising (Tang references source code (see pg. 10, section 7) and experimental results, which inherently reads on a system comprising processor, memory, and program instructions to perform the disclosed operations): accessing speech data (Fig. 1, speech features input to Dual Auditory Encoders (Whisper Encoder and BEATs Encoder); pg. 3, section ‘Dual Auditory Encoders’ Equation 1, speech feature X “…Since both encoders have the same output frame rate of 50Hz, the concatenated output feature are [Equation 1], where X is a variable-length general audio input sequence…”); using a speech-text alignment engine, generating aligned embeddings of the speech data based on …latent space matching of the speech data to text data (Fig. 1, ‘Window-level Q-former’ generates aligned embeddings of speech data to text data based on latent space matching (map speech to same latent space as text in form of textual tokens); pg. 3, section ‘Window-level Q-Former’, 1st para. – 2nd para. “The Q-Former structure is commonly used to convert the output of an image encoder into a fixed number of textual input tokens of an LLM (Li et al., 2023a), which requires modification when applied to handle audio inputs of variable lengths. Specifically, regarding the encoder output of an input image l as a Zl, Q-Former employs a fixed number of N trainable queries Q to transform Zl into N textual tokens Hl using a number of stacked Q-Former blocks…”; see also Fig. 1 caption “A window-level Q-former is used as the connection module to fuse the outputs from a Whisper speech encoder and a BEATs audio encoder as augmented audio tokens, which are aligned with the LLM input space…”); based on the aligned embeddings, generating an input sequence for a Large Language Model (LLM) (Fig. 1, aligned embeddings input to LLM along with textual embeddings from text prompt); and using the LLM and the input sequence, generating an output for the speech data (Fig. 1, text response is generated for LLM input).
Tang does not specifically disclose [using a speech-text alignment engine, generating aligned embeddings of the speech data based on] length matching.
Jin teaches generating aligned embeddings of the speech data based on length matching (Fig. 1, CTC-alignment performed such that speech data frames (hframeAE) are aligned in length with the text tokens (i.e. httokenAE having a matching length with textual tokens htokenTE)).
Tang and Jin are considered to be analogous to the claimed invention as
they both are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tang to incorporate the teachings of Jin in order to generate the aligned embeddings based on length matching. Doing so would be beneficial, as this would resolve sequence length mismatches between the frame level audio features and text token domains (see Fig. 1).
Regarding claim 2, Tang in view of Jin discloses wherein a speech encoder captures audio associated with one or more spoken language and transforms the audio into the speech data comprising speech embeddings representing acoustic features of speech (Tang, Fig. 1, “Whisper Encoder” encodes input speech into speech embeddings: pg. 3, section 3.1, ‘Dual Auditory Encoders’, 1st para. “…The Whisper model is trained for speech recognition and translation based on a large amount of weakly supervised data, whose encoder output features are suitable to model speech and include information about the background noises…”).
Regarding claim 3, Tang in view of Jin discloses wherein generating the aligned embeddings based on the length matching comprises computing alignment information associated with a non-blank token sequence of the speech data and a speech input sequence of the speech data (alignment information obtained ( (i.e. length of non-blank tokens of speech httokenAE is determined, see Eq. 8, where a number of token-level acoustic embeddings corresponding to number of non-blank text tokens) and frame coverage, (see Fig. 1, CTC Aligner determines which frame level acoustic embeddings cover the same token to obtain httokenAE pg. 3, 2nd para.) based on a non-blank token sequence (Fig. 1, htokenTE) and speech input sequence (Fig. 1, hframeAE)) using a Connectionist Temporal Classification (CTC) layer (Jin, Fig. 1, CTC Aligner).
Tang and Jin are considered to be analogous to the claimed invention as
they both are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tang to incorporate the teachings of Jin in order to compute alignment information associated with a non-blank token sequence and a speech input sequence. Doing so would be beneficial, given the same rationale as claim 1.
Regarding claim 4, Tang in view of Jin discloses wherein the length matching comprises determining a length of non-blank tokens of the speech data and a frame coverage of the non-blank tokens (alignment information obtained ( (i.e. length of non-blank tokens of speech httokenAE is determined, see Eq. 8, where a number of token-level acoustic embeddings corresponding to number of non-blank text tokens) and frame coverage, (see Fig. 1, CTC Aligner determines which frame level acoustic embeddings cover the same non-blank tokens to obtain httokenAE pg. 3, 2nd para.) based on a non-blank token sequence (Fig. 1, htokenTE) and speech input sequence (Fig. 1, hframeAE)).
Tang and Jin are considered to be analogous to the claimed invention as
they both are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tang to incorporate the teachings of Jin in order to determine a length of non-blank tokens of the speech data and a frame coverage of the non-blank tokens. Doing so would be beneficial, given the same rationale as claim 1.
Regarding claim 5, Tang in view of Jin discloses wherein generating the aligned embeddings based on latent space matching comprises generating, using a modified q-former layer (Tang, Fig. 1, “Window-level Q-former”), the aligned embeddings (Tang, Fig. 1, “Auditory Embeddings” generated from window-level q-former layer, comprising aligned embeddings) based on frames corresponding to non-blank tokens of the speech data (Jin teaches aligning embeddings based on frames corresponding to non-blank tokens of the speech data: Fig. 1, CTC-alignment performed such that speech data frames (hframeAE) are aligned in length with the non-blank text tokens (i.e. httokenAE having a matching length with non-blank textual tokens htokenTE)), the latent space matching is based on vectorized representation of the text data that is transformed into a shared latent space to facilitate alignment with speech data (Tang, Fig. 1 caption “A window-level Q-former is used as the connection module to fuse the outputs from a Whisper speech encoder and a BEATs audio encoder as augmented audio tokens, which are aligned with the LLM input space…”; pg. 3, section ‘Window-level Q-Former’, 1st para. – 2nd para. “The Q-Former structure is commonly used to convert the output of an image encoder into a fixed number of textual input tokens of an LLM (Li et al., 2023a), which requires modification when applied to handle audio inputs of variable lengths. Specifically, regarding the encoder output of an input image l as a Zl, Q-Former employs a fixed number of N trainable queries Q to transform Zl into N textual tokens Hl using a number of stacked Q-Former blocks…”).
Tang and Jin are considered to be analogous to the claimed invention as
they both are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tang to incorporate the teachings of Jin in order to generate aligned embeddings based on frames corresponding to non-blank tokens of the speech data. Doing so would be beneficial, given the same rationale as claim 1.
Regarding claim 8, Tang discloses One or more computer-storage media having computer-executable instructions embodied thereon that, when executed by a computing system having a processor and memory, cause the processor to perform operations, the operations comprising (Tang references source code (see pg. 10, section 7) and experimental results, which inherently reads on a CRM with instructions which can be executed by a computing system): communicating speech data with a speech-based task for a client device (Fig. 1, speech features input to Dual Auditory Encoders (Whisper Encoder and BEATs Encoder); pg. 3, section ‘Dual Auditory Encoders’ Equation 1, speech feature X “…Since both encoders have the same output frame rate of 50Hz, the concatenated output feature are [Equation 1], where X is a variable-length general audio input sequence…”); see Table 2 for list of speech-based tasks performed at inference; as discussed above, the referenced source code and experimental results inherently read on a device being used to perform the operations recited); based on communicating the speech data, receiving a response associated with the speech data and the speech-based task (Fig. 1, text response is generated for LLM input), wherein the response is associated with a speech-text alignment engine that generates aligned embeddings of the speech data based on…latent space matching of the speech data to text data (Fig. 1, ‘Window-level Q-former’ generates aligned embeddings of speech data to text data based on latent space matching (map speech to same latent space as text in form of textual tokens); pg. 3, section ‘Window-level Q-Former’, 1st para. – 2nd para. “The Q-Former structure is commonly used to convert the output of an image encoder into a fixed number of textual input tokens of an LLM (Li et al., 2023a), which requires modification when applied to handle audio inputs of variable lengths. Specifically, regarding the encoder output of an input image l as a Zl, Q-Former employs a fixed number of N trainable queries Q to transform Zl into N textual tokens Hl using a number of stacked Q-Former blocks…”; see also Fig. 1 caption “A window-level Q-former is used as the connection module to fuse the outputs from a Whisper speech encoder and a BEATs audio encoder as augmented audio tokens, which are aligned with the LLM input space…”), wherein the response is generated using a Large Language Model (LLM) (Fig. 1, text response is generated for LLM input using the LLM); and generating the response to the speech-based task at the client device (Fig. 1, text response is generated for LLM input using the LLM; as discussed above, the referenced source code and experimental results inherently read on a device being used to perform the operations recited).
Tang does not specifically disclose [a speech-text alignment engine that generates aligned embeddings of the speech data based on] length matching.
Jin teaches generating aligned embeddings of the speech data based on length matching (Fig. 1, CTC-alignment performed such that speech data frames (hframeAE) are aligned in length with the text tokens (i.e. httokenAE having a matching length with textual tokens htokenTE)).
Tang and Jin are considered to be analogous to the claimed invention as
they both are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tang to incorporate the teachings of Jin in order to generate the aligned embeddings based on length matching. Doing so would be beneficial, given the same rationale as claim 1.
Regarding claim 9, Tang in view of Jin discloses wherein the speech-based task is associated with speech translation, speech summarization, or speech-based question-answering (Tang, see Table 2 tasks, also pg. 6, section 4.3, tasks include spoken-query-based question answering SQQA).
Regarding claim 10, claim 10 is rejected for analogous reasons to claim 4.
Regarding claim 11, Tang in view of Jin discloses wherein latent space matching (Tang teaches latent space matching) comprises determining frames corresponding to non-blank tokens of the speech data (Jin teaches determining frames corresponding to non-blank tokens of speech data: Fig. 1, CTC-alignment performed such that speech data frames (hframeAE) are aligned in length with the non-blank text tokens (i.e. httokenAE having a matching length with non-blank textual tokens htokenTE); Fig. 1, specific frames are determined as corresponding to non-blank text tokens).
Tang and Jin are considered to be analogous to the claimed invention as
they both are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tang to incorporate the teachings of Jin in order to generate aligned embeddings with latent space matching by determining frames corresponding to non-blank tokens of the speech data. Doing so would be beneficial, given the same rationale as claim 1.
Regarding claim 15, Tang discloses A computer-implemented method, the method comprising: accessing Automatic Speech Recognition (ASR) training data, the ASR training data comprising speech data and text data (pg. 5, section 4.2, “…The data used for the first pre-traiing stage consists of…1000-hour GigaSpeech M-set (Chen et al., 2021) for speech recognition…”; see also Table 1, ASR task)…based on the alignment information and the speech data, using a modified q-former layer to generate aligned embeddings (Fig. 1, using q-former queries and speech data (output of dual auditory encoders), aligned embeddings are generated (Fig. 1, ‘Auditory Embeddings’); ig. 1, ‘Window-level Q-former’ generates aligned embeddings of speech data to text data based on latent space matching (map speech to same latent space as text in form of textual tokens); pg. 3, section ‘Window-level Q-Former’, 1st para. – 2nd para. “The Q-Former structure is commonly used to convert the output of an image encoder into a fixed number of textual input tokens of an LLM (Li et al., 2023a), which requires modification when applied to handle audio inputs of variable lengths. Specifically, regarding the encoder output of an input image l as a Zl, Q-Former employs a fixed number of N trainable queries Q to transform Zl into N textual tokens Hl using a number of stacked Q-Former blocks…”; see also Fig. 1 caption “A window-level Q-former is used as the connection module to fuse the outputs from a Whisper speech encoder and a BEATs audio encoder as augmented audio tokens, which are aligned with the LLM input space…”); using the aligned embeddings, training a Large Language Model (LLM) (training performed on LoRA adaptor portion of LLM (Fig. 1), pg. 4, section 3.2 “Pre-training Stage:…a large amount to speech recognition and audio captioning data is used to pre-train the window-level Q-former and LoRA…”); and deploying the LLM (see Table 2, test/infererence performed for multiple speech-based tasks using the LLM).
Tang does not specifically disclose using a Connectionist Temporal Classification (CTC) layer, generating alignment information associated with a non-blank token sequence of the speech data and a speech input sequence of the speech data.
Jin teaches using a Connectionist Temporal Classification (CTC) layer (Jin, Fig. 1, CTC Aligner), generating alignment information associated with a non-blank token sequence of the speech data and a speech input sequence of the speech data (alignment information obtained ( (i.e. length of non-blank tokens of speech httokenAE is determined, see Eq. 8, where a number of token-level acoustic embeddings corresponding to number of non-blank text tokens) and frame coverage, (see Fig. 1, CTC Aligner determines which frame level acoustic embeddings cover the same token to obtain httokenAE pg. 3, 2nd para.) based on a non-blank token sequence (Fig. 1, htokenTE) and speech input sequence (Fig. 1, hframeAE)).
Tang and Jin are considered to be analogous to the claimed invention as
they both are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tang to incorporate the teachings of Jin in order to generate alignment information using a CTC layer, the alignment information associated with a non-blank token sequence of the speech data and a speech input sequence of the speech data. Doing so would be beneficial, given the same rationale as claim 1.
Regarding claim 16, claim 16 is rejected for analogous reasons to claim 4.
Regarding claim 17, claim 17 is rejected for analogous reasons to claim 11.
3. Claims 6, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Tang in view of Jin, and further in view of Nguyen et al. (NPL “SpiritLM: Interleaved Spoken and Written Language Model”, hereinafter Nguyen).
Regarding claim 6, Tang in view of Jin does not specifically disclose wherein the LLM is trained based on speech-derived embeddings and text-derived embeddings associated with token swapping.
Nguyen teaches wherein the LLM is trained based on speech-derived embeddings and text-derived embeddings associated with token swapping (Fig. 1, caption “…Spirit LM models are trained on mix of text-only sequences, speech-only sequences, and interleaved speech-text sequences. B. Speech text interleaving scheme. Speech is encoded into tokens (pink) using clusterized speech units (Hubert, Pitch, or Style tokens), and text (blue) using BPE. We use special tokens [TEXT] to prefix text and [SPEECH] for speech tokens. During training, a change of modality is randomly triggered at word boundaries in aligned speech-text corpora. Speech tokens are deduplicated and interleaved with text tokens at the modality change boundary…”).
Tang, Jin, and Nguyen are considered to be analogous to the claimed invention as they are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tang in view of Jin to incorporate the teachings of Nguyen in order to train the LLM using speech-derived and text-derived embeddings associated with token swapping. Doing so would be beneficial, as such training process would preserve text generation abilities of the model while also leading to better speech understanding and generation performance (Nguyen, pg. 7, section 4.2, 4th para.).
Regarding claim 13, Tang in view of Jin does not specifically disclose wherein the LLM is trained based on a token swapping technique where speech-derived embeddings are randomly swapped with corresponding text embeddings.
Nguyen teaches wherein the LLM is trained based on a token swapping technique where speech-derived embeddings are randomly swapped with corresponding text embeddings (Fig. 1, caption “…Spirit LM models are trained on mix of text-only sequences, speech-only sequences, and interleaved speech-text sequences. B. Speech text interleaving scheme. Speech is encoded into tokens (pink) using clusterized speech units (Hubert, Pitch, or Style tokens), and text (blue) using BPE. We use special tokens [TEXT] to prefix text and [SPEECH] for speech tokens. During training, a change of modality is randomly triggered at word boundaries in aligned speech-text corpora. Speech tokens are deduplicated and interleaved with text tokens at the modality change boundary…”).
Tang, Jin, and Nguyen are considered to be analogous to the claimed invention as they are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tang in view of Jin to incorporate the teachings of Nguyen in order to train the LLM using a token swapping technique where speech-derived embeddings are randomly swapped with corresponding text embeddings. Doing so would be beneficial, as such training process would preserve text generation abilities of the model while also leading to better speech understanding and generation performance (Nguyen, pg. 7, section 4.2, 4th para.).
Regarding claim 19, claim 19 is rejected for analogous reasons to claim 13.
4. Claims 7 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Tang in view of Jin, and further in view of Wu et al. (NPL “Transferable speech-to-text large language model alignment module”, hereinafter Wu).
Regarding claim 7, Tang in view of Jin discloses wherein a prompt embedding is a pre-defined or dynamically generated vector that encodes instructions or context for the LLM (Tang, Fig. 1, text instruction prompt converted to “Textual embeddings” for instructions to LLM).
Tang in view of Jin does not specifically disclose concatenating the aligned embeddings with prompt embeddings to generate the input sequence for the LLM.
Wu teaches concatenating the aligned embeddings with prompt embeddings to generate the input sequence for the LLM (Fig. 1 and caption “An overview of our proposed speech-text bimodal architecture. Alignment module is used to map the speech features into text feature space. Speech encoder is frozen all the time. LLM embedding will extract text features form prompt. The speech and text modal features are concatenated as LLM’s input…”).
Tang, Jin, and Wu are considered to be analogous to the claimed invention as they are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tang in view of Jin to incorporate the teachings of Wu in order to concatenate the aligned embeddings and prompt embeddings as input to the LLM. Doing so would be beneficial, as such operation would allow the LLM to learn the information between the two modalities autonomously (Li et al., NPL “A Transcription Prompt-based Efficient Audio Large Language Model for Robust Speech Recognition”, see pg. 2, section 3.1).
Regarding claim 20, claim 20 is rejected for analogous reasons to claim 7.
5. Claims 12 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Tang in view of Jin, and further in view of Hono et al. (NPL “Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition”, hereinafter Hono).
Regarding claim 12, Tang in view of Jin does not specifically disclose wherein the LLM is trained on next token prediction loss and CTC loss.
Hono teaches wherein the LLM is trained on next token prediction loss and CTC loss (pg. 4, section 3.3, equation 3, which combines a causal language modeling loss (i.e. next token prediction loss) with a CTC-based loss to train a large language model (see Fig. 1 for model)).
Tang, Jin, and Hono are considered to be analogous to the claimed invention as they are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tang in view of Jin to incorporate the teachings of Hono in order to train the LLM using next token prediction loss and CTC loss. Doing so would be beneficial, as next token prediction loss would enable the LLM to learn how to generate predict the textual responses, while the CTC loss would enable CTC compression to be applied in order to align the text and speech embeddings.
Regarding claim 18, claim 18 is rejected for analogous reasons to claim 12.
5. Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Tang in view of Jin, and further in view of Rossi & Peretti (US 2024/0347058 A1, hereinafter Rossi).
Regarding claim 14, Tang in view of Jin does not specifically disclose wherein the response is a text-based LLM output that is converted to speech.
Rossi teaches wherein the response is a text-based LLM output that is converted to speech (para. 0032 “Upon receiving the text prompt from the conversation system 110, the LLM system 120 generates a text response based on the text prompt and sends the text response to the conversation system 110. The conversation system 110 then converts the text response into a speech response and sends the speech response to the client device 130, causing the client device 130 to play the speech response to the user.”).
Tang, Jin, and Rossi are considered to be analogous to the claimed invention as they are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tang in view of Jin to incorporate the teachings of Rossi in order to convert the text-based LLM output to speech as the response. Doing so would be beneficial, as this would enable verbal conversations with LLMs that mimic human interaction (Rossi, para. 0030).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Chen et al. (NPL “Maestro: Matched Speech Text Representations through Modality Matching”): learning unified representations from speech and text (Fig. 1)
Shang et al. (NPL “An End-to-End Speech Summarization Using Large Language Model”): use of Q-former layer to generate aligned speech embeddings for use in LLM speech-based task (i.e. speech summarization) (Fig. 1)
Tang et al. (NPL “Can Large Language Models Understand Spatial Audio?”): window-level q-former layer to generate aligned embeddings for LLM (Fig. 1)
Chen et al. (NPL “SALM: Speech-Augmented Language Model with In-Context Learning for Speech Recognition and Translation”): modality adapter for speech input to match the different information rate and modeling space between text and speech (pg. 2, 2nd para. and Fig. 1)
Ghosh et al. (US 2025/0336389 A1): aligning inputs with outputs for LLM using CTC losses (Abstract)
Thomas et al. (US 2024/0371361 A1): utilizing cross attention to align at a token level sentence embeddings with speech based embeddings (Abstract)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CODY DOUGLAS HUTCHESON whose telephone number is (703)756-1601. The examiner can normally be reached M-F 8:00AM-5:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at (571)-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CODY DOUGLAS HUTCHESON/Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659
