DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 02 January 2025, 02 January 2025, and 28 January 2026, respectively, are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-8, 10-14, and 16-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over “On Decoder-Only Architecture for Speech-to-Text and Large Language Model Integration”, hereinafter referred to as Wu J et al., in view of US 20240403564, hereinafter referred to as Bendersky et al.
Regarding claim 1, Wu J et al. discloses a method comprising:
receiving audio from a user (Wu J et al., fig. 1 – acoustic features input.);
generating, via a trained audio encoder based upon the received audio, an audio embedding sequence (“In this work, we design an architecture named Speech-LLaMA where a text-LLM can also accept acoustic embedding as well as text as conditional prompts for text generation,” Wu et al., section 3, first sentence. See also Wu J et al., fig. 1 – audio embedding is output from audio encoder block.);
receiving, via a trained large language model (LLM), the generated audio embedding sequence and a text embedding sequence, wherein the text embedding sequence is arranged before or after the generated audio embedding sequence (Wu J et al., fig. 1 and fig. 2. Here, the decoder-only model comprises the LLM text embedding sequence is arranged before the generated audio embedding sequence.);
producing, via the trained LLM based upon the text embedding sequence, a textual response associated with the audio received from the user (Wu J et al., figs. 1 and 2 – predicted tokens.).
Wu J et al., though, does not disclose causing to display, via a user interface of the user, the produced textual response.
Bendersky et al. is cited to disclose causing to display, via a user interface of the user, the produced textual response (Bendersky et al., fig. 1A.). Bendersky et al. benefits Wu J et al. by providing an interface whereby a user may view the text of their audio prompt and responses, thereby allowing a user to verify that the prompt is accurate. Therefore, it would be obvious for one skilled in the art to combine the teachings of Wu J et al. with those of Bendersky et al. to enhance the decoder-only architecture for speech-to-text and LLM integration as described by Wu J et al.
As to claim 12, system claim 12 and method claim 1 are related as method and system of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 12 is similarly rejected under the same rationale as applied above with respect to method claim. Also, Bendersky et al., para [0044]-[0045], describes CRM, memory, processor(s), and processor-executable instructions.
As to claim 20, CRM claim 20 and method claim 1 are related as method and CRM of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 20 is similarly rejected under the same rationale as applied above with respect to method claim. Also, Bendersky et al., para [0044]-[0045], describes CRM, memory, processor(s), and processor-executable instructions.
Regarding claim 2, Wu J et al., as modified by Bendersky et al., discloses the method of claim 1, wherein the text embedding sequence is arranged before and after the generated audio embedding sequence (The user of an LLM usually issues a sequence of prompts that will, together with the response of the LLM, form a dialogue between the user and the LLM. See Bendersky et al., figs. 1 and 2, for example.).
As to claim 13, system claim 13 and method claim 2 are related as method and system of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 13 is similarly rejected under the same rationale as applied above with respect to method claim. Also, Bendersky et al., para [0044]-[0045], describes CRM, memory, processor(s), and processor-executable instructions.
Regarding claim 3, Wu J et al., as modified by Bendersky et al., discloses the method of claim 1, wherein the audio embedding sequence is devoid of an intermediate step of converting the audio embedding sequence into a textual representation (Wu J et al., section 3, “CTC compressor” and “Audio encoder”.).
As to claim 14, system claim 14 and method claim 3 are related as method and system of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 14 is similarly rejected under the same rationale as applied above with respect to method claim. Also, Bendersky et al., para [0044]-[0045], describes CRM, memory, processor(s), and processor-executable instructions.
Regarding claim 5, Wu J et al., as modified by Bendersky et al., discloses the method of claim 1, wherein the LLM uses the text embedding sequence to interpret the audio embedding sequence (Wu J et al., fig. 1 and section 3, first paragraph.).
As to claim 16, system claim 16 and method claim 5 are related as method and system of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 16 is similarly rejected under the same rationale as applied above with respect to method claim. Also, Bendersky et al., para [0044]-[0045], describes CRM, memory, processor(s), and processor-executable instructions.
Regarding claim 6, Wu J et al., as modified by Bendersky et al., discloses the method of claim 1, wherein the text embedding sequence includes a conversation history associated with the user (The user of an LLM usually issues a sequence of prompts that will, together with the response of the LLM, form a dialogue/conversation between the user and the LLM. See Bendersky et al., para [0038] and figs. 1 and 2, for example.).
As to claim 17, system claim 17 and method claim 6 are related as method and system of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 17 is similarly rejected under the same rationale as applied above with respect to method claim. Also, Bendersky et al., para [0044]-[0045], describes CRM, memory, processor(s), and processor-executable instructions.
Regarding claim 7, Wu J et al., as modified by Bendersky et al., discloses the method of claim 1, further comprising:
receiving, via the trained audio encoder, supplemental audio from the user (The user of an LLM usually issues a sequence of prompts that will, together with the response of the LLM, form a dialogue/conversation between the user and the LLM. See Bendersky et al., para [0038] and figs. 1 and 2, for example.);
generating a supplemental audio embedding sequence (The user of an LLM usually issues a sequence of prompts that will, together with the response of the LLM, form a dialogue/conversation between the user and the LLM. See Bendersky et al., para [0038] and figs. 1 and 2, for example.); and
producing a supplemental textual response based upon the audio embedding
sequence (The user of an LLM usually issues a sequence of prompts that will, together with the response of the LLM, form a dialogue/conversation between the user and the LLM. See Bendersky et al., para [0038] and figs. 1 and 2, for example.).
As to claim 18, system claim 18 and method claim 7 are related as method and system of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 18 is similarly rejected under the same rationale as applied above with respect to method claim. Also, Bendersky et al., para [0044]-[0045], describes CRM, memory, processor(s), and processor-executable instructions.
Regarding claim 8, Wu J et al., as modified by Bendersky et al., discloses the method of claim 1, wherein the audio encoder is trained on a textual output of the LLM, and wherein the textual output is derived from automatic speech recognition (ASR) data (“We pre-trained CTC compressor with the paired speech and text data (i.e., ASR task) from 13 languages using the CTC objective function because that in our preliminary experiments, the BLEU score with ASR task training is much better than the one with the ST task training. Once CTC compressor is trained, the parameters are frozen during later training stages,” Wu J et al., section 4.2.1.).
As to claim 19, system claim 19 and method claim 8 are related as method and system of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 19 is similarly rejected under the same rationale as applied above with respect to method claim. Also, Bendersky et al., para [0044]-[0045], describes CRM, memory, processor(s), and processor-executable instructions.
Regarding claim 10, Wu J et al., as modified by Bendersky et al., discloses the method of claim 1, further comprising:
controlling, via the trained encoder, an audio resolution of the audio embedding sequence prior to being received by the LLM (Wu J. et al. section 4.2.2, first paragraph, i.e controlling the resolution of the audio embeddings to match the resolution of the text embeddings.).
Regarding claim 11, Wu J et al., as modified by Bendersky et al., discloses the method of claim 1, wherein the audio encoder includes a convolutional feature extractor with an output frame rate of 80 milliseconds (Wu J. et al., section 4.2.1. Note that 80 milliseconds is a design choice.).
Claim(s) 4, 9, and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over “On Decoder-Only Architecture for Speech-to-Text and Large Language Model Integration”, hereinafter referred to as Wu J et al., in view of US 20240403564, hereinafter referred to as Bendersky et al., and further in view of “Prompting Large Language Models with Speech Recognition Abilities, hereinafter referred to as Fathullah et al.
Regarding claim 4, Wu J et al., as modified by Bendersky et al., discloses the method of claim 1, but not wherein the audio embedding sequence is monotonically aligned with the text embedding sequence. Fathullah et al. is cited to disclose wherein the audio embedding sequence is monotonically aligned with the text embedding sequence (Fathullah et al., section 4.). Fathullah et al. benefits Wu J et al. by training the output to be monotonically aligned to the text, thereby improving speech recognition results. Therefore, it would be obvious for one skilled in the art to combine the teachings of Wu J et al. with those of Fathullah et al. to improve the decoder-only architecture for speech-to-text and LLM integration as described by Wu J et al.
As to claim 15, system claim 15 and method claim 4 are related as method and system of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 15 is similarly rejected under the same rationale as applied above with respect to method claim. Also, Bendersky et al., para [0044]-[0045], describes CRM, memory, processor(s), and processor-executable instructions.
Regarding claim 9, Wu J et al., as modified by Bendersky et al., discloses the method of claim 8, but not wherein the ASR data includes audio data and labeled text associated with the audio data. Fathullah et al. is cited to disclose wherein the ASR data includes audio data and labeled text associated with the audio data (Fathullah et al., section 3.2, paragraph entitled "Training"; table 1: supervised learning.). Fathullah et al. benefits Wu J et al. by training the output to be monotonically aligned to the text, thereby improving speech recognition results. Therefore, it would be obvious for one skilled in the art to combine the teachings of Wu J et al. with those of Fathullah et al. to improve the decoder-only architecture for speech-to-text and LLM integration as described by Wu J et al.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See attached PTO-892. In particular, the examiner notes, Tang et al., Zhao et al., and Wang et al., all of which describe end-to-end speech-input conversational LLMs similar to that proposed by the applicant.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANNE L THOMAS-HOMESCU whose telephone number is (571)272-0899. The examiner can normally be reached Mon-Fri 8-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh M Mehta can be reached on 5712727453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANNE L THOMAS-HOMESCU/Primary Examiner, Art Unit 2656
Prosecution Insights