DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 59-61, 66, 70, 73, 75, 77 is/are rejected under 35 U.S.C. 103 as being unpatentable over Silvia Terragni et al. (hereinafter Terragni) (“In-Context Learning User Simulators for Task-Oriented Dialog Systems,” 06/01/2023) in view of Asaf Aharoni et al. (hereinafter Aharoni) (US 20220255885 A1, 08/11/2022).
Regarding claim 59, Terragni teaches;
prompting an artificial intelligence (AI) to assume a persona ([pg. 11] the prompt consists of a task description which grounds the LLM with a persona) and to seek to fill a service request ([pg. 11] prompt consists of … requirements sentence derived from the user goals... [pg. 2] a user goal could be to make a reservation for a specific date for two people in an Italian restaurant… A US (the LLM) should fulfill all the user goal requirements by the end of the dialog) by a [interactive assistant] (dialogue system) ([pg. 3] The simulator (LLM) interacts directly with the dialog system)
NOTE: In each conversation / dialog between the LLM and the dialogue system, the LLM generates a sequence of utterances requesting a specific service based on the goal ([pg. 3] looking for a train that departs from peterborough and arrives by 19:30) from a dialogue system which acts as an interactive assistant to help provide the specific service ([pg. 3] There’s a train that arrives at 19:09).
receiving, from the AI (LLM), a query (utterance) for the (dialogue system) (utterance) to the (dialogue system) ([pg. 3] The aim of the US (LLM) is to generate user’s utterances … utterance serves as input for the dialog system) receiving, from the (dialogue system) ([pg. 4] The dialog system … returns a natural language utterance per turn)
continuing to permit the AI (LLM) to query the (dialogue system) (LLM) is satisfied that the (dialogue system) (dialogue system) ([pg. 4] US (LLM) generates a new utterance … and the interaction continues until the conversation ends… [pg. 13] The user simulator (LLM) gives up and ends the dialog)
logging the queries (utterances from the Customer/LLM, see below) and responses (utterances from the virtual assistant/dialogue system, see below) between the AI (customer/LLM) and the (assistant/dialogue system)
[pg. 3]
PNG
media_image1.png
75
579
media_image1.png
Greyscale
Terragni fails to teach but Aharoni teaches;
A computer-implemented method of training ([Abstract] train, … the trained voice bot) an interactive voice assistant (IVA) ([0067] the given voice bot can be trained to conduct conversations … as an automated assistant), comprising:
a service provider that operates the IVA; ([0003] the trained voice bot has conducted a plurality of conversations on behalf of Hypothetical Café ... for at least restaurant reservations)
and based on the logging ([0004] conversations conducted by the trained voice bot, ... can be stored as voice bot activity in a voice bot activity database.), debugging or improving the IVA. ([0004] The voice bot activity stored in the voice bot activity can be processed, ... to identify a given behavioral error of the trained voice bot… [0007] perform one or more of the actions that are directed to correcting the given behavioral error of the trained voice bot.)
OBVIOUSNESS TO COMBINE AHARONI:
Aharoni is analogous art to the present disclosure as it pertains to training an interactive voice assistant.
Terragni already teaches a simulated conversation system between an LLM and an interactive assistant (dialogue system) and logging their conversations, while Aharoni teaches logging conversations conducted by a voice-based interactive assistant operated by a service provider (i.e., some third party entity), and using the logged conversations to identify and correct behavioral errors of the interactive voice assistant, as taught above.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use Aharoni’s voice-bot development architecture to extend Terragni’s automated dialogue system to customer service voice assistants and to utilize Aharoni’s method of identifying and correcting behavioral errors of the interactive assistant based on logged conversations to improve the behavior of the interactive voice assistant.
Regarding claim 60, Terragni teaches;
wherein the AI is a large language model (LLM) ([pg. 2] leveraging LLMs for user simulation)
Regarding claim 61, Terragni fails to teach but Aharoni teaches;
wherein the IVA is a less capable machine learning (ML) model than an LLM. ([0041] The voice bot can ... utilize a plurality of machine learning (ML) layers of one or more ML models … layers may correspond to those of … RNN models … and/or other ML layers of other ML models)
NOTE: An RNN is less capable than an LLM.
Regarding claim 66, Terragni teaches;
wherein the persona comprises being terse or concise or speaking with poor grammar. ([pg. 11] We want the LLM to enact a role of the customer, with a certain persona attributes … "Complete the conversation as a CUSTOMER", "Be precise with the REQUIREMENTS, clear and concise".)
Regarding claim 70, Terragni teaches;
further comprising causing the AI (LLM) to generate a large number of ([pg. 12] running 200 identical conversations) service requests to the [interactive assistant] ([pg. 3] US (LLM) generate user’s utterances given a target user goal gt … target dialog dt, i.e. the conversation to be generated by the interaction between US (LLM) and (dialogue) system.)
NOTE: For each conversation / dialog dt, and based on the goal gt, the LLM generates a sequence of utterances specifying a request for a specific service (e.g., booking a reservation or appointment, see page 2) which is provided to the dialogue system.
using a plurality of personas.
([pg. 11] the LLM … with a certain persona attributes that make them more likely to successfully book appointments… persona modifiers include: "You are a picky tourist", "try to rephrase your request until you are sure the ASSISTANT understood you")
Regarding claim 73, Terragni teaches;
service requests
Using the same reasoning from claim 59
Terragni fails to teach but Aharoni teaches;
wherein debugging comprises identifying service requests ([0003] the trained voice bot has conducted a plurality of conversations on behalf of Hypothetical Café ... for at least restaurant reservations) that failed ([0005] One or more embeddings associated with the corresponding conversation for which the given behavioral error is identified), and debugging based on the failed service requests ([0007] the voice bot development platform can automatically perform one or more of the actions that are directed to correcting the given behavioral error of the trained voice bot)
OBVIOUSNESS:
Using the same reasoning from claim 59, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use Aharoni’s voice-bot development architecture to extend Terragni’s automated dialogue system to customer service voice assistants and to utilize Aharoni’s method of identifying and correcting behavioral errors of the interactive assistant based on logged conversations to improve the behavior of the interactive assistant.
Regarding claim 75,
Claim 75 is substantially similar to claim 59, with the addition of the following limitations, where Terragni teaches;
to generate tests for an interactive ([pg. 8] our study demonstrated that in-context learning user simulation generates diverse language valuable for testing a dialog system)
and Terragni fails to teach but Aharonit teaches;
One or more tangible, nontransitory computer-readable storage media having stored thereon executable instructions ([0162] implementations also include one or more non-transitory computer readable storage media storing computer instructions … to perform any of the aforementioned methods.)
interactive voice assistant (IVA)
(using the same reasoning from claim 59)
OBVIOUSNESS:
Using the same reasoning from claim 59, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use Aharoni’s voice-bot development architecture to extend Terragni’s automated dialogue system to customer service voice assistants and to utilize Aharoni’s method of identifying and correcting behavioral errors of the interactive assistant based on logged conversations to improve the behavior of the interactive assistant.
The remaining limitations are taught using the same reasoning as in claim 59.
Regarding claim 77, Terragni teaches;
a first data connection to communicatively couple to a large language model (LLM); (see below, the orange arrow transmitting the generated text from the LLM)
a second data connection to communicatively couple to an (dialog system) (see below, the blue and orange arrows transmitting utt to and from the dialog system)
[pg. 4]
PNG
media_image2.png
351
470
media_image2.png
Greyscale
prompt the LLM to assume a persona ([pg. 11] the prompt consists of a task description which grounds the LLM with a persona) and to seek to fill a service request ([pg. 11] prompt consists of … requirements sentence derived from the user goals... [pg. 2] a user goal could be to make a reservation for a specific date for two people in an Italian restaurant… A US (the LLM) should fulfill all the user goal requirements by the end of the dialog) by a (dialogue system) ([pg. 3] The simulator (LLM) interacts directly with the dialog system)
receive, via the first data connection (see [1] below), a query for the IVA (utterance utt_u,i+1 derived from the LLM generated text); provide the query (utt_u,i+1) to the (dialog system) via the second data connection (see [2] below); receive, via the second data connection (see [2] below), a response to the query (utt_s,i);
[pg. 4]
[AltContent: textbox ([2])][AltContent: textbox ([2])][AltContent: textbox ([1])]
PNG
media_image2.png
351
470
media_image2.png
Greyscale
continue to permit the LLM to query the (dialogue system) ([pg. 3] The simulator (LLM) interacts directly with the dialog system) until the LLM is satisfied that the (dialogue system) (dialogue system) ([pg. 4] US (LLM) generates a new utterance … and the interaction continues until the conversation ends… [pg. 13] The user simulator (LLM) gives up and ends the dialog)
and log the queries (logged utterances from the Customer/LLM, see below) and responses (logged utterances from the virtual assistant/dialogue system, see below) between the LLM (customer/LLM) and the (assistant/dialogue system)
[pg. 3]
PNG
media_image1.png
75
579
media_image1.png
Greyscale
Terragni fails to teach but Aharoni teaches;
An orchestrator ([Abstract] development system that enables the third-party developer to train, update, validate, and monitor performance of the trained voice bot), comprising: a hardware platform comprising a processor circuit and a memory; a user interface;
[fig. 7]
PNG
media_image3.png
263
453
media_image3.png
Greyscale
interactive voice assistant (IVA) ([0067] the given voice bot can be trained to conduct conversations … as an automated assistant)
and instructions encoded within the memory to instruct the processor circuit to: ([0162] processors are operable to execute instructions stored in associated memory, … to cause performance of any of the aforementioned methods.) according to instructions from an operator via the user interface, ([0070] third-party developer may … add training instances for training the voice bot … on the user interface)
a service provider that operates the IVA; ([0003] the trained voice bot has conducted a plurality of conversations on behalf of Hypothetical Café ... for at least restaurant reservations)
OBVIOUSNESS:
Using the same reasoning from claim 59, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use Aharoni’s voice-bot development hardware architecture to extend Terragni’s automated dialogue system to customer service voice assistants and to utilize Aharoni’s method of identifying and correcting behavioral errors of the interactive assistant based on logged conversations to improve the behavior of the interactive assistant.
Claim(s) 63-64 is/are rejected under 35 U.S.C. 103 as being unpatentable over Terragni (“In-Context Learning User Simulators for Task-Oriented Dialog Systems,” 06/01/2023) in view of Aharoni (US 20220255885 A1, 08/11/2022) as applied to claim 59 above, further in view of Vidya Rajagopal et al. (hereinafter Rajagopal) (US 20210064826 A1, 03/04/2021).
Regarding claim 63, Terragni and Aharoni fail to teach but Rajagopal teaches;
further comprising converting the query to speech via a text-to-speech engine before sending the query to the IVA. ([0060] the conversation simulator 330 may include a voice based simulator for TTS, so that test conversations in voice format may be sent to the Bot for training)
OBVIOUSNESS TO COMBINE RAJAGOPAL:
Rajagopal is analogous art to the present disclosure as it pertains to training a voice-based interactive assistant.
Terragni teaches simulating conversations between an AI (LLM) and an interactive assistant (dialogue system), Aharoni teaches training an interactive voice assistant operated by a service provider and improving the assistant based on logged conversations, and Rajagopal teaches simulating conversations between a text based AI and a voice based interactive assistant (bot).
Rajagopal further teaches utilizing text-to-speech and speech-to-text to convert inputs to a suitable format for each model;
([0060] the conversation simulator 330 may include a voice based simulator for TTS, so that test conversations in voice format may be sent to the Bot for training … [0101] when the Bot's response includes a voice, recording voice response and converting the voice response to text response … [0039] AI trainer … analyzing Bot's responses)
A machine learning model can only read inputs that are in the format that the model is configured to receive.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use the methods of Rajagopal to convert inputs for the AI and interactive voice assistant of the system of Terragni as modified by Aharoni into a allowable format before providing the inputs to their respective model to enable each model to read their received inputs.
Regarding claim 64, Terragni and Aharoni fail to teach but Rajagopal teaches;
further comprising converting the response to text via a speech-to-text engine before providing the response to the Al. ([0101] when the Bot's response includes a voice, recording voice response and converting the voice response to text response … [0039] AI trainer … analyzing Bot's responses)
OBVIOUSNESS:
Using the same reasoning from claim 63, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use the methods of Rajagopal to convert inputs for the AI and interactive voice assistant of the system of Terragni as modified by Aharoni into a allowable format before providing the inputs to their respective model to allow each model to read their received inputs.
Claim(s) 62, 65, 67, 76, 78 is/are rejected under 35 U.S.C. 103 as being unpatentable over Terragni (“In-Context Learning User Simulators for Task-Oriented Dialog Systems,” 06/01/2023) in view of Aharoni (US 20220255885 A1, 08/11/2022) as applied to claim 59, 75, 77 above, further in view of Holgar Quast et al. (hereinafter Quast) (US 20150172463 A1, 06/18/2015)
Regarding claim 62,
Terragni teaches;
LLM
([0011] natural language which can be understood by the LLM)
Terragni and Aharoni fail to teach but Quast teaches;
wherein the IVA comprises a domain-specific
([0216] The virtual assistant may recognize the received voice input using a topic-dependent language model for recognizing speech input relating to sports and/or any other suitable language model.)
OBVIOUSNESS TO COMBINE QUAST:
Quast is analogous art to the present disclosure as it pertains to providing an interactive voice assistant.
Terragni already teaches that LLMs can be used for understanding language input, Aharoni teaches an interactive voice assistant, and Quast teaches an interactive voice assistant using a domain specific LM (where an LLM is a type of LM).
From this, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to configure the IVA of the system of Terragni as modified by Aharonit to be implemented using a domain specific language model as taught by Quast (where the language model can be an LLM, as taught by Terragni), predictably improving performance of the IVA for it’s specific domain.
Regarding claim 65, Terragni and Aharoni fail to teach but Quast teaches;
wherein the persona comprises being verbose.
([0074] another virtual assistant persona (e.g., specified in the profile of user 102b to cause the virtual assistant to talk with a male voice in a verbose manner))
OBVIOUSNESS:
Terragni already teaches configuring an AI with personas, and specifies that certain personas make the AI more likely to successfully fulfill its service request (e.g. book an appointment);
([pg. 11] LLM … with a certain persona attributes that make them more likely to successfully book appointments.)
Quast teaches an AI (the virtual assistant LM) being configured with various other personas that are not explicitly mentioned in Terragni (e.g., being verbose)
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use the additional personas taught by Quast in the system of Terragni as modified by Aharoni to provide additional personas which may make the AI more likely to successfully fulfill its service request.
Regarding claim 67, Terragni and Aharoni fail to teach but Quast teaches;
wherein the persona comprises speaking unusually fast or slow.
([0074] virtual assistant 105b may adopt one virtual assistant persona (e.g., ... talk with a slow speaking rate…))
OBVIOUSNESS:
Using the same reasoning from claim 65, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use the additional personas taught by Quast in the system of Terragni as modified by Aharoni to provide additional personas which may make the AI more likely to successfully fulfill its service request.
Regarding claims 76 and 78,
Claims 76 and 78 directly correspond to claim 62 and are rejected using the same reasoning.
Claim(s) 68 is/are rejected under 35 U.S.C. 103 as being unpatentable over Terragni (“In-Context Learning User Simulators for Task-Oriented Dialog Systems,” 06/01/2023) in view of Aharoni (US 20220255885 A1, 08/11/2022) as applied to claim 59 above, further in view of Srinivasan Janarthanam et al. (hereinafter Janarthanam) (“Adaptive Generation in Dialogue Systems Using Dynamic User Modeling,” 2014)
Regarding claim 68, Terragni and Aharoni fail to explicitly teach but Janarthanam teaches;
wherein the persona is a person unfamiliar with a service ([pg. 12] We built a corpus-based user simulation model that simulates the dialogue behavior of a real human user… [pg. 13] The models ranged from novices to experts … A novice user knew only power adaptor) provided by the IVA. ([pg. 5] we built a ... technical support dialogue system that helps users to set up a home broadband connection... The dialogue system consists of a … speech synthesizer)
NOTE: The novice user simulator models are configured to be unfamiliar with setting up home broadband connection, which is the service provided by the interactive voice assistant (the dialogue system).
OBVIOUSNESS TO COMBINE JANARTHANAM:
Jarnathanam is analogous art to the present disclosure as it pertains to facilitating conversations between simulated users and an interactive voice assistant.
Terragni already teaches configuring a user simulation AI with personas, and specifies that certain personas make the AI more likely to successfully fulfill its service request (e.g. book an appointment);
([pg. 11] LLM … with a certain persona attributes that make them more likely to successfully book appointments.)
As shown above, Jarnathanam teaches configuring user simulation models with a ‘novice’ persona, making them behave as a person that is less familiar with the service (setting up home broadband) provided by the IVA (the dialogue system).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use the additional personas taught by Jarnathanam in the system of Terragni as modified by Aharoni to provide additional personas which may make the AI more likely to successfully fulfill its service request.
Claim(s) 69 is/are rejected under 35 U.S.C. 103 as being unpatentable over Terragni (“In-Context Learning User Simulators for Task-Oriented Dialog Systems,” 06/01/2023) in view of Aharoni (US 20220255885 A1, 08/11/2022) as applied to claim 59 above, further in view of Pasquale Demaio et al. (hereinafter Demaio) (US 9961192 B1, 05/01/2018)
Regarding claim 69, Terragni fails to teach but Aharoni teaches;
wherein providing the query to the IVA comprises contacting the IVA via
([0001] these bots can initiate telephone calls or answer incoming telephone calls, and conduct conversations with humans to perform action(s) on behalf of a third-party.)
OBVIOUSNESS:
Using the same reasoning from claim 59, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use Aharoni’s voice-bot development architecture to extend Terragni’s automated dialogue system to customer service voice assistants and to utilize Aharoni’s method of identifying and correcting behavioral errors of the interactive assistant based on logged conversations to improve the behavior of the interactive assistant.
Terragni and Aharoni fail to teach but Demiao teaches;
a virtual dialer that simulates a phone interface.
PNG
media_image4.png
573
664
media_image4.png
Greyscale
OBVIOUSNESS TO COMBINE DEMIAO WITH TERRAGNI:
Demiao is analogous art to the present disclosure as it pertains to facilitating communication between two endpoints using a virtual dialer that simulates a phone interface.
Terragni teaches simulated customers / users conversing with a virtual assistant, Aharoni teaches an interactive voice assistant deployed to answer incoming telephone calls and conduct conversations with customers on behalf of a third-party entity, while Demiao teaches a software communication interface that simulates a telephone connection and includes a keypad through which a user may provide voice or keypad responses without using a conventional telephone line.
Demiao further explains;
([col. 2, pg. 44-55] Once execution of a contact workflow is initiated, the contact run-time engine generates a communication interface (e.g., a softphone interface, … ) that connects the executing contact workflow with the GUI so that the user may provide responses to prompts generated by the contact workflow without requiring the use of a dedicated conventional communication channel … Thus, security for development and/or testing of contact workflows is dramatically improved because the risk of exposure of a contact workflow to a publically available communication channel (e.g., a publically available telephone number) is eliminated)
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use Demiao’s telephone interface as an input interface to Aharoni’s incoming-call voice bot in the system of Terragni as modified by Aharoni, allowing a developer to simulate a telephone caller contacting and interacting with the voice bot without posing security risks.
Claim(s) 71 is/are rejected under 35 U.S.C. 103 as being unpatentable over Terragni (“In-Context Learning User Simulators for Task-Oriented Dialog Systems,” 06/01/2023) in view of Aharoni (US 20220255885 A1, 08/11/2022) as applied to claim 70 above, further in view of Terragni [github] (“In-Context Learning User Simulators for Task-Oriented Dialog Systems (Github),” 06/02/2023)
Regarding claim 71, Terragni teaches;
cause the Al to generate the large number of service requests.
Using the same reasoning from claim 70
Terragni and Aharoni fail to explicitly teach but Terragni [github] teaches;
further comprising using automated scripting to cause the Al to generate the large number of service requests.
([pg. 2] python scripts/user_simulator_script.py … The script will run a specific number of dialogs with the defined dialog system and user simulator models. The results are stored in the specified folder.)
NOTE: As previously taught by Terragni, in each dialog/conversation the user simulator/LLM generates a sequence of utterances requesting a service. Thus, this automated python script causes the user simulator/LLM to generate a plurality of service requests (plurality of sequence of utterances requesting a service for each conversation/dialog).
OBVIOUSNESS TO COMBINE TERRAGNI [GITHUB]
Terragni [github] is analogous art to the present disclosure as it pertains to simulating conversation between a user simulator and dialogue system.
Terragni explicitly cites Terragni [Github] as the implementation used for their experiements;
([Terragni, pg. 1] Our implementation is available at https://github.com/telepathylabsai/prompt-based-user-simulator.)
Using the implementation in the repository disclosed by Terragni [github] would not alter Terragni’s principle of operation. Rather, it would implement Terragni’s same user simulation process using the software connections and execution mechanisms expressly supplied for that purpose. A person of ordinary skill, before the effective filing date, would have reasonably expected the implementation using Terragni [github] to succeed because it was provided by the authors as the executable implementation used in the paper of Terragni.
Claim(s) 72 is/are rejected under 35 U.S.C. 103 as being unpatentable over Terragni (“In-Context Learning User Simulators for Task-Oriented Dialog Systems,” 06/01/2023) in view of Aharoni (US 20220255885 A1, 08/11/2022) as applied to claim 70 above, further in view of Pegah Jandaghi et al. (hereinafter Jandaghi) (“Faithful Persona-based Conversational Dataset Generation with Large Language Models,” 12/15/2023)
Regarding claim 72, Terragni teaches;
the large number of service requests
Using the same reasoning from claim 70
Terragni and Aharoni fail to teach but Jandaghi teaches;
identifying, within the large number of (conversations) (conversations) (conversations) ([pg. 5] evaluates the candidate conversations based on the predetermined policies, and selects the best candidate conversations. Step 3 The best candidate conversations are added to the dataset for the next iteration of generation.)
NOTE: Conversations of the plurality of conversations that are identified to not be the best (i.e., bad conversations) are not included (dropped) in the next iteration.
OBVIOUSNESS TO COMBINE JANDAGHI:
Jandaghi is analogous art to the present disclosure as it pertains to facilitating conversations between large language models.
Terragni already teaches generating a large number of service requests from an LLM based user simulator where each service request corresponds to a sequence of utterances from the LLM in a conversation, and Jandaghi teaches filtering out bad conversations from a conversation training dataset.
Jandaghi additionally indicates that their method of filtering out bad conversations from the training dataset improves resulting conversational models;
([Abstract] Training Natural Language Processing (NLP) models on a diverse and comprehensive persona-based dataset can lead to conversational models that create a deeper connection with the user, and maintain their engagement. In this paper, we leverage the power of Large Language Models (LLMs) to create a large, high-quality conver sational dataset)
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to utilize Jandaghi’s method of dropping bad conversations (i.e., only including the best conversations) from a training dataset, to generate a high-quality conversation dataset to train the natural language processing models of the system of Terragni as modified by Aharoni, thereby producing improved conversational models capable of creating a deeper connection with their conversational partner, and maintaining engagement.
Claim(s) 74 is/are rejected under 35 U.S.C. 103 as being unpatentable over Terragni (“In-Context Learning User Simulators for Task-Oriented Dialog Systems,” 06/01/2023) in view of Aharoni (US 20220255885 A1, 08/11/2022) as applied to claim 70 above, further in view of J.D. Zamfirescu-Pereira et al (hereinafter Pereira) (“Conversation Regression Testing: A Design Technique for Prototyping Generalizable Prompt Strategies for Pre-trained Language Models,” 02/06/2023)
Regarding claim 74, Terragni teaches;
service requests
Using the same reasoning from claim 59
Terragni and Aharoni fail to teach but Pereira teaches;
Identifying (conversational contexts) (conversational contexts) (successful contexts used for regression testing).
([pg. 2] Conversation Regression Testing uses the conversational contexts where a baseline LM ... notably succeeded ... as reusable test cases)
OBVIOUSNESS TO COMBINE PEREIRA:
Pereira is analogous art to the present disclosure as it pertains to regression testing for language models.
Terragni teaches an LLM that generates service requests (a sequence of utterances for booking a restaurant reservation, for example) over a conversation with an interactive assistant, Aharoni teaches an interactive voice assistant, and Pereira teaches identifying successful conversational contexts for language models (LLMs, for example) which are then used for regression testing.
Pereira further provides the benefit of this process;
([pg. 2] Conversation Regression Testing uses the conversational contexts where a baseline LM has … notably succeeded … as reusable test cases and helps designers track the effects of prompt strategy updates on these test cases. This approach allows designers to freely experiment with many prompt strategies to address a particular error in context, while ensuring the system’s overall stability and a trajectory of continuous improvements.)
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to preserve successful generated service requests from the conversations of Terragni as modified by Aharoni as Pereira’s reusable regression test cases, to allow the developer to verify that changes intended to improve the interactive voice assistant do not break service request flows that previously worked correctly.
CONCLUSION
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Matthew Alan Cady whose telephone number is (571) 272-7229. The examiner can normally be reached Monday - Friday, 7:30 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached on (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC)
at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW ALAN CADY/ Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights