DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The prior 35 U.S.C. 112 rejection of claims 4 and 14 (10/21/25) is withdrawn in light of amendments to the claim.
Response to Arguments
Applicant's arguments filed 3/23/26 have been fully considered but they are not persuasive. Regarding the 35 U.S.C. 102 rejection of claim 20 with reference Baeumi, Applicant argues that Baeumi fails to disclose limitation “detecting a second speech input from the user during the playing of the first speech response signal and in response to detecting an intent to interrupt the first speech response signal with the second speech input: stopping playing the first speech response signal” (Arguments, pg. 12 – pg. 13, third para.). Examiner respectfully disagrees as Baeumi explicitly describes a user using voice input to select a halt streaming output button on its client’s display while the client device is rendering a stream of NL based output response, thereby halting the rendering of subsequent segments of the NL based output response (para. [0077]), corresponding to “detecting a second speech input from the user during the playing of the first speech response signal and in response to detecting an intent to interrupt the first speech response signal with the second speech input: stopping playing the first speech response signal”.
Applicant’s arguments with respect to subsequent limitations in the independent claims now being performed “in response to detecting an intent to interrupt the first speech response signal with the second speech input:” including “generating, by the LLM system, a second speech response signal based on the second speech input”, and the arguments provided above for claims 5 and 15 (Arguments, pg. 12 – pg. 13, third para.) have been considered but are moot in light of new grounds of rejection with reference Krishnan (PTO 892, 10/21/25) as presented below.
Regarding the rejection of dependent claims 3, 6-10, 13 and 16-19 with additional references Choo, Mehrabani and Rajbhandari, Applicant argues that the dependent claims are allowable based on their dependency from the independent claims as well as the arguments provided above for the independent claims (Arguments, pg. 13).
Examiner respectfully disagrees as provided above for the independent claims, and absent any argument as to why the cited portions of the additional references fail to address the limitations of the dependent claims, Examiner maintains that the rejections of the dependent claims are appropriate.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
1. Claims 1-5, 11-15 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Baeumi et al US 2024/0311402 A1 (“Baeumi”) in view of Krishnan et al US 2022/0093101 A1 (“Krishnan”)
Per claim 1, Baeumi discloses a method for managing interruptions during a verbal conversation between users and Large Language Models (LLMs), comprising:
receiving a first speech input from a client device of a user (the system receives NL based input associated with a client device. In some implementations, the NL based input can be one formulated based on explicit user interface input at a client device (e.g., detected via the user input engine 111), such as typed input, voice input …, para. [0032]);
converting the first speech input into first text (In some implementations, when the query includes content that is not in textual format, the system can convert the query to a textual format or other format…., para. [0032]);
generating a first prompt for an LLM based on the first text (Abstract; At block 254, the system processes, using a LLM, the NL based input to generate a stream of LLM output…., para. [0035], input provided to LLM as prompt);
transmitting the first prompt to the LLM, causing the LLM to generate a first text response (At block 254, the system processes, using a LLM, the NL based input to generate a stream of LLM output…., para. [0035]);
receiving the first text response from the LLM (para. [0046]);
converting the first text response into a first speech response (para. [0046]; para. [0077]);
transmitting the first speech response back to the client device, causing the client to play the first speech response to the user (the stream of NL based output can be visually rendered via a display of the client device (e.g., via the rendering engine 112) ... In additional or alternative implementations, the NL based output can audibly rendered via speaker(s) of the client device (e.g., via the rendering engine 112) …, para. [0046]);
detecting a second speech input comprising a second speech input from the user during the play of the first speech response (In various implementations of the method 300 of FIG. 3 or the method 500 of FIG. 5 where the NL based output that that is responsive to the NL based input 652 is generated and rendered in a streaming manner, and while the stream of NL based output is being rendered at the client device 110, the client device 110 can cause a halt streaming selectable element 656 to be rendered via the display 180 of the client device 110. The halt streaming selectable element 656, when selected by the user of the client device 110 (e.g., via voice input and/or touch input), can cause the generating and/or rendering of the stream of NL based output to be halted…., para. [0077], voice input as second speech input);
in response to detecting an intent to interrupt the first speech response signal with the second speech input: causing the client device to stop playing the first speech response (the client device 110 can cause a halt streaming selectable element 656 to be rendered via the display 180 of the client device 110. The halt streaming selectable element 656, when selected by the user of the client device 110 (e.g., via voice input and/or touch input), can cause the generating and/or rendering of the stream of NL based output to be halted…., para. [0077]);
Baeumi discloses the use of an LLM, generating and transmitting prompts to the LLM as well as receiving and generating from the LLM (fig. 2)
Baeumi does not explicitly disclose in response to detecting an intent to interrupt the first speech response signal with the second speech input: generating a second prompt to the LLM based on the second speech input, transmitting the second prompt to the LLM, causing the LLM to generate a second text response, receiving the second text response from the LLM, converting the second text response into a second speech response or transmitting the second speech response back to the client device, causing the client device to play the second speech response to the user
However, these features are taught/suggested by Krishnan:
in response to detecting an intent to interrupt the first speech response signal with the second speech input: generating a second prompt to the language model based on the second speech input (That one looks good!, fig. 15F; para. [0055]; The output audio may correspond to a list of entries, for example a TTS output responsive to a previous user request that called for the system to output the list. …, para. [0058]-[0059]; The system may also cease processing with regard to the second audio or may continue processing the second audio for purposes of updating the system's dialog data …, para. [0056]; para. [0077]; The NLG component 279 may include a trained model. The NLG component 279 generates text data 2110 from dialog data received by the dialog manager 272 such that the output text data 2110 has a natural feel …, para. [0078]; para. [0078]; para. [0079]; if the system is outputting data corresponding to a turn in a conversation (e.g., paying back audio representing TTS synthesized speech), if a user interruption (sometimes referred to as a “barge-in”) is detected, the system may discontinue or lower the volume of the TTS output to ensure processing (and outward recognition) of the user interruption …, para. [0350]; para. [0352]; para. [0364], detected barge-in as intent to interrupt, received dialog data as input/prompt to trained NLG/language model 279, trained NLG/language model 279 as suggesting large language model (LLM));
transmitting the second prompt to the language model, causing the language model to generate a second text response (The language output component 293 includes a natural language generation (NLG) component 279 and a text-to-speech (TTS) component 280 …, para. [0077]; The NLG component 279 may include a trained model. The NLG component 279 generates text data 2110 from dialog data received …, para. [0078]; Barge-in support may include the system being able to process the user interrupting the system with a request, including anaphoric selection of a currently displayed or audibly indicated item (e.g., the user speaking “that one” in the middle of a list being output through TTS) or ordinal selection of any item (e.g., the user speaking “the second one” in response to a list being output through TTS or displayed on a screen) … if the system is unable to determine which item the user is referring to with sufficient confidence, it may ask a follow up question such as “did you mean the pepperoni pizza?” …, para. [0352], received dialog data as input/prompt provided/transmitted to trained NLG/language model 279, trained NLG/language model 279 as suggesting large language model (LLM));
receiving the second text response from the language model (The text of a system generated response may be sent to a TTS component 280 for creation of audio data corresponding to the response…., para. [0074]; para. [0077]; The NLG component 279 may include a trained model. The NLG component 279 generates text data 2110 from dialog data received by the dialog manager 272 such that the output text data 2110 has a natural feel …, para. [0078]; The system may be configured to recover when a user barge-in is unclear. For example, using entity confirmation of the candidate barge-in entity (“you meant salmon, right?”). Alternatively, the system may perform an operation for general clarification (“I did not catch that, can you repeat the selection?”) …, para. [0353], trained NLG/language model 279 as suggesting large language model (LLM));
converting the second text response into a second speech response (The text of a system generated response may be sent to a TTS component 280 for creation of audio data corresponding to the response…., para. [0074]; The language output component 293 includes a natural language generation (NLG) component 279 and a text-to-speech (TTS) component 280 …, para. [0077]); and
transmitting the second speech response back to the client device, causing the client device to play the second speech response to the user (fig. 2; para. [0043]; The text of a system generated response may be sent to a TTS component 280 for creation of audio data corresponding to the response. The audio data may then be sent to a user device (e.g., device 110) for ultimate output to the user …, para. [0074]; para. [0080]; para. [0493])
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Krishnan with the method of Baeumi in arriving at the missing features of Baeumi, because such combination would have resulted in an improved ability to handle conversational overlap, leading to a better user experience (Krishnan, para. [0350]). Furthermore, it would have been obvious to one of ordinary skill in the art to try to implement/substitute a LLM (such as the LLM described by Baeumi) instead of/with the language model (LM)/NLG 279 of Krishnan with the suggestion/motivation of reducing latency while performing various natural language processing (NLP) tasks.
Per claim 2, Baeumi in view of Krishnan discloses the method of claim 1,
Krishnan discloses: determining a timing of the second speech input (The system 120 may receive (194) the input audio data and the time data…., para. [0059]);
generating the second prompt based on the timing of the second speech input (fig. 1C; The system 120 may receive (194) the input audio data and the time data…., para. [0059]; fig. 15F; para. [0077]-[0079]).
Per claim 3, Baeumi in view of Krishnan discloses the method of claim 2,
Krishnan discloses: wherein determining the timing of the second speech input comprises: determining whether the second speech input is received within a first threshold time period after the first speech response starts (Abstract; para. [0352]; para. [0383])
responsive to determining that the second speech input is received within the first threshold time period after the first speech response starts, generating a continuer response to signal the user to continue speaking (para. [0352]; para. [0383]).
Per claim 4, Baeumi in view of Krishnan discloses the method of claim 2,
Krishnan discloses wherein determining the timing of the second speech input comprises: determining whether the second speech input is received after a playback threshold time period, wherein the playback threshold time period occurs during a speech playing state (fig. 13; fig. 15F; para. [0364]); and
responsive to determining that the second speech input is received after the playback threshold time period: canceling a previous conversation (fig. 13; fig. 15F; para. [0364]); and
restarting a new conversation (fig. 13; fig. 15F; para. [0364]).
Per claim 5, Baeumi in view of Krishnan discloses the method of claim 1,
Krishnan discloses: determining a state of the conversation when the second speech input is received (para. [0059]; fig. 15F; para. [0077]-[0079]; para. [0297]; The system may use the transcript time information and the offset time data TO 1309 to determine that the user barged-in while the device 110 was outputting audio corresponding to a particular item on the list, e.g., the “one pan salmon from Easy Kitchen” illustrated in FIG. 13…. The system may then determine the entity item included at the appropriate output time aligned with the user's barge-in (e.g., “one pan salmon”) and may thus identify and use that entity for further processing, para. [0364]; para. [0387]-[0388]); and
generating the second prompt to the language model based on the determined state of the current conversation (fig. 1C; The system 120 may receive (194) the input audio data and the time data…., para. [0059]; fig. 15F; para. [0077]-[0079]; para. [0297]; he entity resolver component 1170 may then use the utterance data (including the timing data) as well as the context embedding to determine that the user's utterance 1402 indicated selection of the first entry in the list (e.g., “one pan salmon”)…., para. [0388], trained NLG/language model 279 as suggesting large language model (LLM)).
Per claim 11, Baeumi discloses a computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:
receive a first speech input from a client device of a user (the system receives NL based input associated with a client device. In some implementations, the NL based input can be one formulated based on explicit user interface input at a client device (e.g., detected via the user input engine 111), such as typed input, voice input …, para. [0032]);
convert the first speech input into first text (In some implementations, when the query includes content that is not in textual format, the system can convert the query to a textual format or other format…., para. [0032]);
generate a first prompt for an LLM based on the first text (Abstract; At block 254, the system processes, using a LLM, the NL based input to generate a stream of LLM output…., para. [0035], input provided to LLM as prompt);
transmit the first prompt to the LLM, causing the LLM to generate a first text response (At block 254, the system processes, using a LLM, the NL based input to generate a stream of LLM output…., para. [0035]);
receive the first text response from the LLM (para. [0046]);
convert the first text response into a first speech response (para. [0046]; para. [0077]);
transmit the first speech response back to the client device, causing the client to play the first speech response to the user (the stream of NL based output can be visually rendered via a display of the client device (e.g., via the rendering engine 112) ... In additional or alternative implementations, the NL based output can audibly rendered via speaker(s) of the client device (e.g., via the rendering engine 112) …, para. [0046]);
detect a second speech input from the user during the play of the first speech response (In various implementations of the method 300 of FIG. 3 or the method 500 of FIG. 5 where the NL based output that that is responsive to the NL based input 652 is generated and rendered in a streaming manner, and while the stream of NL based output is being rendered at the client device 110, the client device 110 can cause a halt streaming selectable element 656 to be rendered via the display 180 of the client device 110. The halt streaming selectable element 656, when selected by the user of the client device 110 (e.g., via voice input and/or touch input), can cause the generating and/or rendering of the stream of NL based output to be halted…., para. [0077], voice input as second speech input);
in response to detecting an intent to interrupt the first speech response with the second speech input: cause the client device to stop playing the first speech response (the client device 110 can cause a halt streaming selectable element 656 to be rendered via the display 180 of the client device 110. The halt streaming selectable element 656, when selected by the user of the client device 110 (e.g., via voice input and/or touch input), can cause the generating and/or rendering of the stream of NL based output to be halted…., para. [0077]);
Baeumi discloses the use of an LLM, generating and transmitting prompts to the LLM as well as receiving and generating from the LLM (fig. 2)
Beaum does not explicitly disclose in response to detecting an intent to interrupt the first speech response with the second speech input: generate a second prompt to the LLM based on the second speech input, transmit the second prompt to the LLM, causing the LLM to generate a second text response, receive the second text response from the LLM, convert the second text response into a second speech response or transmit the second speech response back to the client device, causing the client device to play the second speech response to the user
However, these features are taught/suggested by Krishnan:
in response to detecting an intent to interrupt the first speech response with the second speech input: generate a second prompt to the language model based on the second speech input (That one looks good!, fig. 15F; para. [0055]; The output audio may correspond to a list of entries, for example a TTS output responsive to a previous user request that called for the system to output the list. …, para. [0058]-[0059]; The system may also cease processing with regard to the second audio or may continue processing the second audio for purposes of updating the system's dialog data …, para. [0056]; para. [0077]; The NLG component 279 may include a trained model. The NLG component 279 generates text data 2110 from dialog data received by the dialog manager 272 such that the output text data 2110 has a natural feel …, para. [0078]; para. [0078]; para. [0079]; if the system is outputting data corresponding to a turn in a conversation (e.g., paying back audio representing TTS synthesized speech), if a user interruption (sometimes referred to as a “barge-in”) is detected, the system may discontinue or lower the volume of the TTS output to ensure processing (and outward recognition) of the user interruption …, para. [0350]; para. [0352]; para. [0364], detected barge-in as intent to interrupt, received dialog data as input/prompt to trained NLG/language model 279, trained NLG/language model 279 as suggesting large language model (LLM));
transmit the second prompt to the language model, causing the language model to generate a second text response (The language output component 293 includes a natural language generation (NLG) component 279 and a text-to-speech (TTS) component 280 …, para. [0077]; The NLG component 279 may include a trained model. The NLG component 279 generates text data 2110 from dialog data received …, para. [0078]; Barge-in support may include the system being able to process the user interrupting the system with a request, including anaphoric selection of a currently displayed or audibly indicated item (e.g., the user speaking “that one” in the middle of a list being output through TTS) or ordinal selection of any item (e.g., the user speaking “the second one” in response to a list being output through TTS or displayed on a screen) … if the system is unable to determine which item the user is referring to with sufficient confidence, it may ask a follow up question such as “did you mean the pepperoni pizza?” …, para. [0352], received dialog data as input/prompt provided/transmitted to trained NLG/language model 279, trained NLG/language model 279 as suggesting large language model (LLM));
receive the second text response from the language model (The text of a system generated response may be sent to a TTS component 280 for creation of audio data corresponding to the response…., para. [0074]; para. [0077]; The NLG component 279 may include a trained model. The NLG component 279 generates text data 2110 from dialog data received by the dialog manager 272 such that the output text data 2110 has a natural feel …, para. [0078]; The system may be configured to recover when a user barge-in is unclear. For example, using entity confirmation of the candidate barge-in entity (“you meant salmon, right?”). Alternatively, the system may perform an operation for general clarification (“I did not catch that, can you repeat the selection?”) …, para. [0353], trained NLG/language model 279 as suggesting large language model (LLM));
convert the second text response into a second speech response (The text of a system generated response may be sent to a TTS component 280 for creation of audio data corresponding to the response…., para. [0074]; The language output component 293 includes a natural language generation (NLG) component 279 and a text-to-speech (TTS) component 280 …, para. [0077]); and
transmit the second speech response back to the client device, causing the client device to play the second speech response to the user (fig. 2; para. [0043]; The text of a system generated response may be sent to a TTS component 280 for creation of audio data corresponding to the response. The audio data may then be sent to a user device (e.g., device 110) for ultimate output to the user …, para. [0074]; para. [0080]; para. [0493])
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Krishnan with the product of Baeumi in arriving at the missing features of Baeumi, because such combination would have resulted in an improved ability to handle conversational overlap, leading to a better user experience (Krishnan, para. [0350]). Furthermore, it would have been obvious to one of ordinary skill in the art to try to implement/substitute a LLM (such as the LLM described by Baeumi) instead of/with the language model (LM)/NLG 279 of Krishnan with the suggestion/motivation of reducing latency while performing various natural language processing (NLP) tasks.
Per claim 12, Baeumi in view of Krishnan discloses the computer program product of claim 11,
Product claim 12 and method claim 2 are related as product and the method of using same, with each claimed element's function corresponding to the claimed method step. Accordingly claim 12 is similarly rejected under the same rationale as applied above with respect to claim 2.
Per claim 13, Baeumi in view of Krishnan discloses the computer program product of claim 12,
Product claim 13 and method claim 3 are related as product and the method of using same, with each claimed element's function corresponding to the claimed method step. Accordingly claim 13 is similarly rejected under the same rationale as applied above with respect to claim 3.
Per claim 14, Baeumi in view of Krishnan discloses the computer program product of claim 12,
Product claim 14 and method claim 4 are related as product and the method of using same, with each claimed element's function corresponding to the claimed method step. Accordingly claim 14 is similarly rejected under the same rationale as applied above with respect to claim 4.
Per claim 15, Baeumi in view of Krishnan discloses the method of claim 11,
Krishnan discloses: determining a state of the conversation when the second speech input is received (para. [0059]; fig. 15F; para. [0077]-[0079]; para. [0297]; The system may use the transcript time information and the offset time data TO 1309 to determine that the user barged-in while the device 110 was outputting audio corresponding to a particular item on the list, e.g., the “one pan salmon from Easy Kitchen” illustrated in FIG. 13…. The system may then determine the entity item included at the appropriate output time aligned with the user's barge-in (e.g., “one pan salmon”) and may thus identify and use that entity for further processing, para. [0364]; para. [0387]-[0388]); and
generating the second prompt to the language model based on the determined state of the current conversation (fig. 1C; The system 120 may receive (194) the input audio data and the time data…., para. [0059]; fig. 15F; para. [0077]-[0079]; para. [0297]; he entity resolver component 1170 may then use the utterance data (including the timing data) as well as the context embedding to determine that the user's utterance 1402 indicated selection of the first entry in the list (e.g., “one pan salmon”)…., para. [0388], trained NLG/language model 279 as suggesting large language model (LLM)).
It would have been obvious to one of ordinary skill in the art to try to implement/substitute a LLM (such as the LLM described by Baeumi) instead of/with the language model (LM)/NLG 279 of Krishnan with the suggestion/motivation of reducing latency while performing various natural language processing (NLP) tasks.
Per claim 20, Baeumi discloses a method for managing interruptions during verbal conversations between users and Large Language Models (LLMs), comprising:
playing a first speech response signal on a client device of a user during a vocal conversation between the user and an LLM system (the stream of NL based output can be visually rendered via a display of the client device (e.g., via the rendering engine 112) ... In additional or alternative implementations, the NL based output can audibly rendered via speaker(s) of the client device (e.g., via the rendering engine 112) …, para. [0046]);
detecting second speech input from the user during the playing of the first speech response signal and before an end of the playing of the first speech response signal (fig. 6A; para. [0042]-[0045]; while the stream of NL based output is being rendered at the client device 110, the client device 110 can cause a halt streaming selectable element 656 to be rendered via the display 180 of the client device 110. The halt streaming selectable element 656, when selected by the user of the client device 110 (e.g., via voice input and/or touch input), can cause the generating and/or rendering of the stream of NL based output to be halted …., para. [0077], selection of halt streaming selectable button 656 by voice input as second speech input);
in response to detecting an intent to interrupt the first speech response signal with the second speech input: stopping playing the first speech response signal (The halt streaming selectable element 656, when selected by the user of the client device 110 (e.g., via voice input and/or touch input), can cause the generating and/or rendering of the stream of NL based output to be halted …., para. [0077], halting as stopping playing the first speech response signal);
Baeumi generating, by the LLM system, a second speech response signal (fig. 2)
Baeumi does not explicitly disclose in response to detecting an intent to interrupt the first speech response signal with the second speech input: generating, by the LLM system, a second speech response signal based on the second speech input or playing the second speech response signal on the client device of the user
However, these features are taught/suggested by Krishnan:
in response to detecting an intent to interrupt the first speech response signal with the second speech input: generating, by the language model system, a second speech response signal based on the second speech input (fig. 15F; The language output component 293 includes a natural language generation (NLG) component 279 and a text-to-speech (TTS) component 280 …, para. [0077]; The NLG component 279 may include a trained model. The NLG component 279 generates text data 2110 from dialog data received …, para. [0078]; Barge-in support may include the system being able to process the user interrupting the system with a request, including anaphoric selection of a currently displayed or audibly indicated item (e.g., the user speaking “that one” in the middle of a list being output through TTS) or ordinal selection of any item (e.g., the user speaking “the second one” in response to a list being output through TTS or displayed on a screen) … if the system is unable to determine which item the user is referring to with sufficient confidence, it may ask a follow up question such as “did you mean the pepperoni pizza?” …, para. [0352], trained NLG/language model 279 as suggesting large language model (LLM), system response to barge-in user input as second speech response signal); and
playing the second speech response signal on the client device of the user (fig. 2; para. [0043]; The text of a system generated response may be sent to a TTS component 280 for creation of audio data corresponding to the response. The audio data may then be sent to a user device (e.g., device 110) for ultimate output to the user …, para. [0074]; para. [0077]-[0079]; para. [0493])
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Krishnan with the method of Baeumi in arriving at the missing features of Baeumi, because such combination would have resulted in an improved ability to handle conversational overlap, leading to a better user experience (Krishnan, para. [0350]). Furthermore, it would have been obvious to one of ordinary skill in the art to try to implement/substitute a LLM (such as the LLM described by Baeumi) instead of/with the language model (LM)/NLG 279 of Krishnan with the suggestion/motivation of reducing latency while performing various natural language processing (NLP) tasks.
2. Claims 6, 7, 16 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Baeumi in view of Krishnan as applied to claims 1 and 11 above, and further in view of Mehrabani et al US 12,243,517 B1 (“Mehrabani”) and Rajbhandari et al US 2022/0059086 A1 (“Rajbhandari”)
Per claim 6, Baeumi in view of Krishnan discloses the method of claim 1,
Baeumi does not explicitly disclose applying a machine-learning model to data associated with the conversation to determine a speech completion score indicating a likelihood of whether the user has finished speaking, wherein the machine-learning model is trained over historical conversation data comprising features associated with historical conversations between users and LLMs and interruptions during the conversations, or responsive to the speech completion score greater than a threshold, generating the first prompt for the LLM.
However, these features are suggested by Mehrabani:
applying a machine-learning model to data associated with the conversation to determine a speech completion score indicating a likelihood of whether the user has finished speaking (the endpointing model 160 is a rules-based model that specifies an endpoint is present in a user utterance increment based on values of the speech attributes or confidence scores satisfying one or more tests. For example, a test of the rules-based model specifies that an endpoint is present in an utterance …, col. 13, ln 35-59),
wherein the machine-learning model is trained over historical conversation data comprising features associated with historical conversations between users and LLMs (the model training engine 145 may use historical conversations collected across one or more users and derived speech attributes to train a machine learning model…., col. 10, ln 19-43, machine-learning model as suggesting LLMs);
responsive to the speech completion score greater than a threshold, generating the first prompt for the LLM (If the confidence score output by the endpointing model 160 meets or exceeds a threshold score indicating that the given portion of the user utterance is the endpoint of the user utterance, the endpointing system 101 may cause the VA to generate a response utterance to respond to the user …, col. 13, ln 35-59, machine-learning model as suggesting LLMs)
Baeumi in view of Mehrabani does not explicitly disclose wherein the machine-learning model is trained over historical conversation data comprising features associated with interruptions during the historical conversations
However, this feature is taught by Rajbhandari (para. [0040]-[0041]; para. [0102]; The model building component 510 may at least partially train a machine learning model(s) using behavioral data 525.…, para. [0104]; The behavioral data 525 may include barge-in data. Barge-in data may represent instances when the system detects a wakeword while the system is performing an action believed responsive to a user input (e.g., the user interrupts or “barges in” with a subsequent user input while the system is performing an action) …, para. [0106]; para. [0118]; para. [0132])
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Mehrabani with the method of Baeumi in arriving at the missing features of Baeumi, as well as to combine the teachings of Rajbhandari with the method of Baeumi in view of Mehrabani in arriving at the missing features of Baeumi in view of Mehrabani, because such combination would have resulted in enabling a system to determine when to act next without interrupting a user (Mehrabani, col. 2, ln 19-31) as well as in automatically improving a system’s understanding of user inputs (Rajbhandari, para. [0025]). Furthermore, it would have been obvious to one of ordinary skill in the art to try to implement/substitute a LLM (such as the LLM described by Baeumi) instead of/with the NLP model and the NLU model of Mehrabani and Rajbhandari with the suggestion/motivation of reducing latency while performing various natural language processing (NLP) tasks.
Per claim 7, Baeumi in view of Krishnan discloses the method of claim 1, further comprising:
Baeumi does not explicitly disclose applying a machine-learning model to data associated with the conversation to determine a complexity score indicating a level of complexity of the conversation, wherein the machine-learning model is trained over historical conversation data comprising features associated with historical conversations between users and LLMs and interruptions during the historical conversations or responsive to the complexity score greater than a threshold, generating the first prompt for the LLM, causing the LLM to summarize the conversation, thereby verifying with the user that the conversation has been correctly understood
However, these features are suggested by Mehrabani:
applying a machine-learning model to data associated with the conversation to determine a complexity score indicating a level of complexity of the conversation (col. 14, ln 17-37; The endpointing system can use machine learning to determine an endpoint in a user utterance increment, training endpointing models 147 using the model training engine 145. Furthermore, the conversational/dialogue context could also be taken into account to build an endpointing model, where depending on the dialog state, utterances, and semantics information captured so far from the previous turns of a conversation (e.g., the conversation history stored in data store 150), the endpointing model 160 outputs different confidence scores, col. 16, ln 29-51, conversation/utterance requiring clarification as complex conversation),
wherein the machine-learning model is trained over historical conversation data comprising features associated with historical conversations between users and LLMs (col. 5, ln 56 – col. 6, ln 29; col. 13, ln 17-24, machine-learning model as suggesting LLMs); and
responsive to the complexity score greater than a threshold, generating the first prompt for the LLM, causing the LLM to summarize the conversation, thereby verifying with the user that the conversation has been correctly understood (The endpointing system 101 may cause the VA to generate a response utterance (e.g., a clarifying question, an instruction to provide more information, or a confirmation that the user utterance was correctly understood)…., col. 14, ln 17-37, machine-learning model as suggesting LLMs)
Baeumi in view of Mehrabani does not explicitly disclose wherein the machine-learning model is trained over historical conversation data comprising features associated with interruptions during the historical conversations
However, this feature is taught by Rajbhandari (para. [0040]-[0041]; para. [0102]; The model building component 510 may at least partially train a machine learning model(s) using behavioral data 525. Behavioral data 525 may represent one or more characteristics of one or more user inputs..…, para. [0104]; The behavioral data 525 may include barge-in data. Barge-in data may represent instances when the system detects a wakeword while the system is performing an action believed responsive to a user input (e.g., the user interrupts or “barges in” with a subsequent user input while the system is performing an action) …, para. [0106]; para. [0118]; para. [0132])
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Mehrabani with the method of Baeumi in arriving at the missing features of Baeumi, as well as to combine the teachings of Rajbhandari with the method of Baeumi in view of Mehrabani in arriving at the missing features of Baeumi in view of Mehrabani, because such combination would have resulted in enabling a system to determine when to act next without interrupting a user (Mehrabani, col. 2, ln 19-31) as well as in automatically improving a systems' understanding of user inputs (Rajbhandari, para. [0025]). Furthermore, it would have been obvious to one of ordinary skill in the art to try to implement/substitute a LLM (such as the LLM described by Baeumi) instead of/with the NLP model and the NLU model of Mehrabani and Rajbhandari with the suggestion/motivation of reducing latency while performing various natural language processing (NLP) tasks.
Per claim 16, Baeumi in view of Krishnan discloses the computer program product of claim 11,
Product claim 16 and method claim 6 are related as product and the method of using same, with each claimed element's function corresponding to the claimed method step. Accordingly claim 16 is similarly rejected under the same rationale as applied above with respect to claim 6.
Per claim 17, Baeumi in view of Krishnan discloses the method of claim 11, further comprising:
Baeumi does not explicitly disclose applying a machine-learning model to data associated with the conversation to determine a complexity score indicating a level of complexity of the current conversation, wherein the machine-learning model is trained over historical conversation data comprising features associated with historical conversations between users and LLMs and interruptions during the historical conversations or responsive to the complexity score greater than a threshold, generating the first prompt for the LLM, causing the LLM to summarize the current conversation, thereby verifying with the user that the current conversation has been correctly understood
However, these features are suggested by Mehrabani:
applying a machine-learning model to data associated with the conversation to determine a complexity score indicating a level of complexity of the current conversation (col. 14, ln 17-37; The endpointing system can use machine learning to determine an endpoint in a user utterance increment, training endpointing models 147 using the model training engine 145. Furthermore, the conversational/dialogue context could also be taken into account to build an endpointing model, where depending on the dialog state, utterances, and semantics information captured so far from the previous turns of a conversation (e.g., the conversation history stored in data store 150), the endpointing model 160 outputs different confidence scores, col. 16, ln 29-51, conversation/utterance requiring clarification as complex conversation),
wherein the machine-learning model is trained over historical conversation data comprising features associated with historical conversations between users and LLMs (col. 5, ln 56 – col. 6, ln 29; col. 13, ln 17-24, machine-learning model as suggesting LLMs); and
responsive to the complexity score greater than a threshold, generating the first prompt for the LLM, causing the LLM to summarize the current conversation, thereby verifying with the user that the current conversation has been correctly understood (The endpointing system 101 may cause the VA to generate a response utterance (e.g., a clarifying question, an instruction to provide more information, or a confirmation that the user utterance was correctly understood)…., col. 14, ln 17-37, machine-learning model as suggesting LLMs)
Baeumi in view of Mehrabani does not explicitly disclose wherein the machine-learning model is trained over historical conversation data comprising features associated with interruptions during the historical conversations
However, this feature is taught by Rajbhandari (para. [0040]-[0041]; para. [0102]; The model building component 510 may at least partially train a machine learning model(s) using behavioral data 525.…, para. [0104]; The behavioral data 525 may include barge-in data. Barge-in data may represent instances when the system detects a wakeword while the system is performing an action believed responsive to a user input (e.g., the user interrupts or “barges in” with a subsequent user input while the system is performing an action) …, para. [0106]; para. [0118]; para. [0132])
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Mehrabani with the method of Baeumi in arriving at the missing features of Baeumi, as well as to combine the teachings of Rajbhandari with the method of Baeumi in view of Mehrabani in arriving at the missing features of Baeumi in view of Mehrabani, because such combination would have resulted in enabling a system to determine when to act next without interrupting a user (Mehrabani, col. 2, ln 19-31) as well as in automatically improving a systems' understanding of user inputs (Rajbhandari, para. [0025]). Furthermore, it would have been obvious to one of ordinary skill in the art to try to implement/substitute a LLM (such as the LLM described by Baeumi) instead of/with the NLU model of Rajbhandari with the suggestion/motivation of reducing latency while performing various natural language processing (NLP) tasks.
3. Claims 8-10, 18 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Baeumi in view of Rajbhandari
Per claim 8, Baeumi in view of Krishnan discloses the method of claim 1,
Baeumi discloses: applying a machine-learning model to data associated with the second speech input to determine a type of vocal interruption (para. [0032]; The halt streaming selectable element 656, when selected by the user of the client device 110 (e.g., via voice input and/or touch input), can cause the generating and/or rendering of the stream of NL based output to be halted…., para. [0077]; causing the one or more processors to halt processing of the NL based input and/or halt rendering of a current segment of the stream of NL based output that is being rendered when the halt streaming selectable element is selected, wherein the current segment of the stream of NL based output is one of: the first segment of the stream of NL based output or the second segment of the stream of NL based output, para. [0102]; generating, based on processing the NL based input using a large language model (LLM), a stream of NL based output that is responsive to the NL based input …, para. [0113], halting streaming of first segment or second segment as types of interruption),
Krishnan discloses generating the second prompt based on the determined type of vocal interruption (fig. 15F; para. [0364]; para. [0387])
Baeumi does not explicitly disclose wherein the machine-learning model is trained over historical conversation data comprising features associated with historical conversations between users and language models and interruptions during the historical conversations
However, this feature is taught by Rajbhandari (para. [0040]-[0041]; The model building component 510 may at least partially train a machine learning model(s) using behavioral data 525.…, para. [0104]; The behavioral data 525 may include barge-in data. Barge-in data may represent instances when the system detects a wakeword while the system is performing an action believed responsive to a user input (e.g., the user interrupts or “barges in” with a subsequent user input while the system is performing an action), para. [0106]; para. [0118]; para. [0132])
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Rajbhandari with the method of Baeumi in arriving at the missing features of Baeumi, because such combination would have resulted in automatically improving a systems' understanding of user inputs (Rajbhandari, para. [0025]). Furthermore, it would have been obvious to one of ordinary skill in the art to try to implement/substitute a LLM (such as the LLM described by Baeumi) instead of/with the NLU model of Rajbhandari with the suggestion/motivation of reducing latency while performing various natural language processing (NLP) tasks.
Per claim 9, Baeumi in view of Krishnan discloses the method of claim 1,
Krishnan disclose collecting data associated with the conversation and second speech input (fig. 15F; para. [0146]; para. [0306])
Baeumi in view of Krishnan does not explicitly disclose retraining or fine-tuning the LLM based on the collected data
However, this feature is taught by Rajbhandari (para. [0040]-[0041]; The behavioral data 525 may include barge-in data. Barge-in data may represent instances when the system detects a wakeword while the system is performing an action believed responsive to a user input (e.g., the user interrupts or “barges in” with a subsequent user input while the system is performing an action) …, para. [0104]; para. [0118]; para. [0106]; para. [0132]-[0133], NLU model as suggesting LLM)
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Rajbhandari with the method of Baeumi in arriving at the missing features of Baeumi, because such combination would have resulted in automatically improving a systems' understanding of user inputs (Rajbhandari, para. [0025]). Furthermore, it would have been obvious to one of ordinary skill in the art to try to implement/substitute a LLM (such as the LLM described by Baeumi) instead of/with the NLU model of Rajbhandari with the suggestion/motivation of reducing latency while performing various natural language processing (NLP) tasks.
Per claim 10, Baeumi in view of Krishnan discloses the method of claim 1,
Krishnan discloses collecting user feedback on how the second speech input was handled (para. [0306]; fig.15F)
Baeumi in view of Krishnan does not explicitly disclose retraining or fine-tuning the LLM based on the collected user feedback
However, these features are taught/suggested by Rajbhandari (the server(s) 120 may determine the subsequent user input corresponds to explicit user feedback (e.g., may determine the subsequent user input explicitly indicates the action is or was not a correct response to the initial user input). For further example, the server(s) 120 may determine the subsequent user input corresponds to implicit user feedback (e.g., may determine the subsequent user input corresponds to a rephrasing of the initial user input). The server(s) 120 may at least partially train (140) at least one machine learning model, using the original user input and the subsequent user input, to detect when future user inputs should be rewritten, para. [0038]; para. [0041]; para. [0104]; para. [0118]; para. [0106]; para. [0132]-[0133], NLU model as suggesting LLM)
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Rajbhandari with the method of Baeumi in arriving at the missing features of Baeumi, because such combination would have resulted in automatically improving a systems' understanding of user inputs (Rajbhandari, para. [0025]). It would have been obvious to one of ordinary skill in the art to try to implement/substitute a LLM (such as the LLM described by Baeumi) instead of/with the NLU model of Rajbhandari with the suggestion/motivation of reducing latency while performing various natural language processing (NLP) tasks.
Per claim 18, Baeumi in view of Krishnan discloses the computer program product of claim 11,
Product claim 18 and method claim 8 are related as product and the method of using same, with each claimed element's function corresponding to the claimed method step. Accordingly claim 18 is similarly rejected under the same rationale as applied above with respect to claim 8.
Per claim 19, Baeumi in view of Krishnan discloses the method of claim 11,
Krishnan disclose collecting data associated with a current conversation between the user and the LLM and the second speech input (fig.15F; para. [0146]; para. [0306])
collecting user feedback on how the second speech input was handled (para. [0306]; fig.15F);
Baeumi in view of Krishnan does not explicitly disclose retraining or fine-tuning the LLM based on the collected data or collected user feedback
However, this feature is taught by Rajbhandari (para. [0040]-[0041]; The behavioral data 525 may include barge-in data. Barge-in data may represent instances when the system detects a wakeword while the system is performing an action believed responsive to a user input (e.g., the user interrupts or “barges in” with a subsequent user input while the system is performing an action) …, para. [0104]; para. [0118]; para. [0106]; para. [0132]-[0133], NLU model as suggesting LLM)
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Rajbhandari with the method of Baeumi in arriving at the missing features of Baeumi, because such combination would have resulted in automatically improving a systems' understanding of user inputs (Rajbhandari, para. [0025]). It would have been obvious to one of ordinary skill in the art to try to implement/substitute a LLM (such as the LLM described by Baeumi) instead of/with the NLU model of Rajbhandari with the suggestion/motivation of reducing latency while performing various natural language processing (NLP) tasks.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See PTO 892 form.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OLUJIMI A ADESANYA whose telephone number is (571)270-3307. The examiner can normally be reached Monday-Friday 8:30-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richemond Dorvil can be reached at 571-272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OLUJIMI A ADESANYA/Primary Examiner, Art Unit 2658
Prosecution Insights