DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The information disclosure statements (IDS) submitted on 05/15/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claims 1-7 and 11-23 are pending and claims 1, 19 and 20 are independent claims.
Response to Arguments
Applicant’s arguments with respect to claims 1-5, 7, and 11-20, filed on 05/15/2026, have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 - 6, 19 and 20-23 are rejected under 35 U.S.C. 103 as being unpatentable over Feuz et al. Pat App No. US 20200050788 A1 (Feuz) in view of Siefken et al. Pat App No. US 20200273089 A1 (Siefken), further in view of Kesanupalli et al. Pat App. No. US 6603837 B1 (Kesanupalli), and further in view of Rasipuram et al. at App No. US 20230134970 A1 (Rasipuram).
Regarding Claim 1. Feuz discloses a method implemented by one or more processors (Feuz, para 0020, a method performed by one or more processors), the method comprising:
identifying an entity for an automated assistant to engage with during an automated telephone call (Feuz, para 0065, annotate references to an entity at a high level of granularity (e.g., to enable identification of all references to an entity class such as people) and/or a lower level of granularity (e.g., to enable identification of all references to a particular entity such as a particular person). The entity tagger may rely on content of the natural language input to resolve a particular entity and/or may optionally communicate with a knowledge graph or other entity database to resolve a particular entity; [“communicate” may include “automated telephone call”]);
selecting an initial voice to be utilized by the automated assistant and during the automated telephone call with the entity, the initial voice to be utilized by the automated assistant in generating one or more corresponding instances of synthesized speech to be rendered during the automated telephone call with the entity (Feuz, para 0016, Users often customize various aspects of their automated assistant experiences, e.g., by selecting different voice synthesizers. Techniques described herein may leverage these customizations in order to strengthen the appearance of speaking with another user's automated assistant. For instance, Dave may set his automated assistant client to a male voice, and Alice may set her automated assistant client to a female voice. Consequently, when Dave invokes an instance of an automated assistant (and assuming his identity is ascertained), he hears a male voice. Likewise, when Alice invokes an instance of an automated assistant, she hears a female voice. In various implementations, techniques described herein may be employed such that when Dave invokes seeks to communicate with Alice's automated assistant, the female voice synthesizer employed by Alice may be activated for the automated instance invoked by Dave. Consequently, even though Dave may still be interacting with his own automated assistant client executing on his own computing device, he nevertheless hears the voice of Alice's assistant, effectively providing Dave with the experience of speaking with Alice's assistant);
initiating the automated telephone call with the entity (Feuz, para 0041, to determine the second user's availability, the first user might call…the second user; [i.e., the first user initiating the call; “the second user” as “the entity”]); and
during the automated telephone call with the entity (Feuz, para 0041, However, the second user may not be available to answer the first user's question, or might prefer not to be interrupted. In such a situation an automated assistant could step in and provide information to the first user about the second user's availability; [i.e., the automated assistant could step in “during the … telephone call with the entity”]).
in response to determining to select the alternative voice to be utilized by the automated assistant and during the automated telephone call with the entity (Feuz, para 0058-0059, using one or more voice synthesizers that may be, for instance, selected by a user, selected automated based on a user's region and/or demographics, etc. And as noted elsewhere herein, all or parts of modules 116, 117, and 112 may be implemented on client device 106, in addition to or instead of on the cloud. In some implementations, automated assistant 120 generates responsive content in response to various inputs generated by a user of one of the client devices 106 during a human-to-computer dialog session with automated assistant 120. Automated assistant 120 may provide the responsive content (e.g., over one or more networks when separate from a client device of a user) for presentation to the user as part of the dialog session).
causing the automated assistant to utilize the alternative voice in continuing with the automated telephone call (Feuz, para 0132, in some implementations when a first user attempts to engage with an automated assistant that serves a second user, a voice synthesizer selected by the second user when they engage with automated assistant 120 may be used to communicate with the first user, thereby giving the illusion to the first user of speaking with the second user's automated assistant).
Feuz does not specifically disclose receiving audio data capturing speech provided by a representative associated with the entity, the audio data being received during the automated telephone call, and the representative associated with the entity being one of: a human representative associated with the entity, a voice bot associated with the entity, or an interactive voice response system associated with the entity, and determining , based on the initial voice that was selected to be utilized by the automated assistant during the automated telephone call and based on a language or an accent of the speech provided by the representative associated with the entity during the automated telephone call, whether to select an alternative voice to be utilized, and in lieu of the initial voice, by the automated assistant and during the automated telephone call with the entity.
However, Siefken, in the same field of endeavor, discloses:
receiving audio data capturing speech provided by a representative associated with the entity, the audio data being received during the automated telephone call, and the representative associated with the entity being one of: a human representative associated with the entity, a voice bot associated with the entity, or an interactive voice response system associated with the entity (Siefken, para 0086, As shown in FIG. 3, the process S100 is started (S101) when a customer calls the store 109. The voice API 113 facilitates interaction with the customer 101 via an automated assistant. The assistant provides a greeting (S103), such as “Hi, welcome to Voice Ordering, what would you like to order today?” The system 100 causes the API 113; Siefken, para 0065-0066, The system 100 is configured to perform such functions by interacting with a number of other systems including a telephony provider 111 …The call 103 is placed through a telephony provider … capable of sending and receiving audio telephone calls; [i.e., this is mapped to the part of the limitation “an interactive voice response system associated with the entity”] ).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Siefken in the method of Feuz because this would enable restaurants or other businesses to implement an interactive voice response system (Siefken, para 0062).
Feuz in view of Siefken do not specifically disclose determining , based on the initial voice that was selected to be utilized by the automated assistant during the automated telephone call and based on a language or an accent of the speech provided by the representative associated with the entity during the automated telephone call, whether to select an alternative voice to be utilized, and in lieu of the initial voice, by the automated assistant and during the automated telephone call with the entity
However, Kesanupalli, in the same field of endeavor, discloses determining , based on the initial voice that was selected to be utilized by the automated assistant during the automated telephone call and based on a language or an accent of the speech provided by the representative associated with the entity during the automated telephone call, whether to select an alternative voice to be utilized, and in lieu of the initial voice, by the automated assistant and during the automated telephone call with the entity (Kesanupalli, col 18, ln 49-60, In a presently preferred embodiment, when a user 114 calls the telephony server 404 via the telephone 430, the initial voice prompts are played in the default language of the telephony server 404 until the time that the user 114 actually signs on to the telephony server 404. For example, in the United States, the default locale would generally be English/U.S. with English being the default language, while in Japan, the default locale would generally be Japanese/Japan with Japanese being the default language. Once the user 114 identifies himself/herself by signing on, the subsequent voice prompts are played in the user's 114 language of choice selected at the time of registration ).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Kesanupalli in the method of Feuz in view of Siefken because this has the advantage of accessing information services in the traveling user's language, independent of the country in which the user may be situated at the time of access (Kesanupalli, col 2, ln 21-24).
Feuz in view of Siefken and Kesanupalli do not specifically disclose selecting , based on the language or the accent of the speech provided by the representative associated with the entity during the automated telephone call, the alternative voice to be utilized by the automated assistant and during the automated telephone call with the entity, the alternative voice to be utilized by the automated assistant in generating the one or more corresponding instances of synthesized speech to be rendered during the automated telephone call with the entity.
However, Rasipuram, in the same field of endeavor discloses selecting, based on the language or the accent of the speech provided by the representative associated with the entity during the automated telephone call, the alternative voice to be utilized by the automated assistant and during the automated telephone call with the entity, the alternative voice to be utilized by the automated assistant in generating the one or more corresponding instances of synthesized speech to be rendered during the automated telephone call with the entity (Rasipuram, para 0013, Determining, based on the text, a tag corresponding to a speaker of the text; and determining, based on the tag, a prosody for the speaker allows for automatic and efficient creation of realistic sounding audio books. In particular, by determining a tag corresponding to a speaker of the text voices for different speakers of the text may be determined and used to generate speech which may be combined into a book. This audio file may be created with an efficient use of processing power to create multiple voices for a more engaging and interesting interaction for the listener; Rasipuram, para 0035-0036, FIG. 1 illustrates a block diagram of system 100 according to various examples. In some examples, system 100 implements a digital assistant. The terms “digital assistant,” “virtual assistant,” “intelligent automated assistant,” or “automatic digital assistant” refer to any information processing system that interprets natural language input in spoken and/or textual form to infer user intent, and performs actions based on the inferred user intent. For example, to act on an inferred user intent, the system performs one or more of the following: identifying a task flow with steps and parameters designed to accomplish the inferred user intent, inputting specific requirements from the inferred user intent into the task flow; executing the task flow by invoking programs, methods, services, APIs, or the like; and generating output responses to the user in an audible (e.g., speech) and/or visual form. Specifically, a digital assistant is capable of accepting a user request at least partially in the form of a natural language command, request, statement, narrative, and/or inquiry. Typically, the user request seeks either an informational answer or performance of a task by the digital assistant; [i.e., automatic response by the digital assistant which is capable of accepting a user request at least partially in the form of a natural language command, request, statement, narrative, and/or inquiry and perform as intended; para 0239, task flow processing module 736 employs the assistance of service processing module 738 (“service processing module”) to complete a task requested in the user input or to provide an informational answer requested in the user input. For example, service processing module 738 acts on behalf of task flow processing module 736 to make a phone call; Rasipuram, para 0271, prosody includes one or more of a rate of speech, a volume level, a level of excitedness, an accent of speech, a pitch of a voice, timbre, frequency, or any other characteristic that affects the way that speech output 814 sounds; [i.e., meaning the digital assistant automatically responds the call based on the requested language, for instance.] ); and
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Rasipuram in the method of Feuz in view of Siefken and Kesanupalli because this would enable an appropriate voice for reading of a children's book and thus a voice that is cheerful and light can be used instead of, for example, a voice that is suspenseful or angry which would be inappropriate (Rasipuram, para 0250).
Regarding Claim 2. Feuz in view of Siefken, Kesanupalli and Rasipuram disclose the method of claim 1.
Furthermore, Rasipuram teaches:
wherein the initial voice is associated with a first set of prosodic properties (Rasipuram, para 0249-0259, the present system determines an appropriate expressivity and/or prosody based on the text and further processes a body of text in a manner that ensures the continuity of the text is maintained. Further, by considering the context of the text, the expressivity or style of a selected voice can be adjusted to create a realistic voice that users enjoy listening to. Moreover, the style of the selected voice can be adjusted based on a genre and/or themes of text to create a voice that provides a consistent experience with the desired genre or theme. For example, a voice that is suspenseful or angry would be inappropriate for reading of a children's book and thus a voice that is cheerful and light can be used instead… By ensuring the continuity of the larger text is preserved with this overlapping of portions of the smaller text inputs the overall tone and genre of the large text (e.g., book) is maintained and conveyed in the audio output. Further, when multiple voices may be used in the audio output (as discussed further below) maintaining the continuity in this manner allows for system 800 to track the conversation and generate the appropriate voice and audio output for each person in the conversation. This provides for a more enjoyable user experience and a higher quality audio output automatically based on the processing of system 800; [“expressivity and/or prosody or style” and “tone” as “set of prosodic properties”]),
wherein the alternative voice is associated with a second set of prosodic properties (Rasipuram, para 0271, In some examples, prosody includes one or more of a rate of speech, a volume level, a level of excitedness, an accent of speech, a pitch of a voice, timbre, frequency, or any other characteristic that affects the way that speech output 814 sounds. Thus, prosody predictor 804 may determine one or more of these characteristics for text 802 representative of the determined genre. For example, when the determined genre is horror, prosody predictor 804 may determine a rate of speech that is slow and a volume level that is quiet to create fear or suspense. As another example, when the determined genre is children's, prosody predictor 804 may determine a pitch that is high and vibrant and a level of excitedness that is high to get children interested).
and
wherein the second set of prosodic properties differs from the first set of prosodic properties (Rasipuram, para 0272-0273, In some examples, the determined prosody is indicative of a particular voice or voice profile. For example, prosody predictor 804 and system 800 may determine that for a children's book the speaker of audio output 810 should be a bright individual who is excited to read and thus determine a prosody with characteristics that match that type of voice profile. In some examples, prosody is represented with a global style token or other data structure. Thus prosody predictor 804 may determine the prosody and generate a global style token that may be provided to other components of system 800 to reference. In this way, the determined prosody can be referenced simply through the token or data structure. For example, prosody predictor 804 may provide a global style token which indicates a high volume, quick rate of speech, and British accent).
Regarding Claim 3. Feuz in view of Siefken, Raissyan and Rasipuram disclose the method of claim 2.
Furthermore, Feuz teaches:
when the initial voice is being utilized by the automated assistant and during the automated telephone call with the entity (Feuz, para 0016, In various implementations, techniques described herein may be employed such that when Dave invokes seeks to communicate with Alice's automated assistant, the female voice synthesizer employed by Alice may be activated for the automated instance invoked by Dave; [“employ” or “invoke” as “utilized”; “communicate” as “automated telephone call”]).
when the alternative voice is being utilized by the automated assistant and during the automated telephone call with the entity (Feuz, para 0058-0059, In some implementations, TTS module 116 may be configured to convert text to computer-synthesized speech, e.g., using one or more voice synthesizers that may be, for instance, selected by a user, selected automated based on a user's region and/or demographics, etc. And as noted elsewhere herein, all or parts of modules 116, 117, and 112 may be implemented on client device 106, in addition to or instead of on the cloud. In some implementations, automated assistant 120 generates responsive content in response to various inputs generated by a user of one of the client devices 106 during a human-to-computer dialog session with automated assistant 120. Automated assistant 120 may provide the responsive content (e.g., over one or more networks when separate from a client device of a user) for presentation to the user as part of the dialog session. For example, automated assistant 120 may generate responsive content in response to free-form natural language input provided via client device 106; [i.e., “during a human-to-computer dialog session with automated assistant” as “during the automated telephone call” ]):
Furthermore, Rasipuram teaches:
processing, using a text-to-speech (TTS) model, textual content to be provided for presentation to the representative associated with the entity and the first set of prosodic properties to generate one or more of the corresponding instances of synthesized speech audio data (Rasipuram, para 0306, system 800 generates speech output 814 of text 802 with audio generator 812. Similarly, after determining a prosody based on one or more tags, system 800 generates speech output 814 of text 802 with audio generator 812. Speech output 814 can be generated with various text to speech technologies including text to speech techniques discussed herein. Speech output 814 is generated using the prosody or speech characteristics determined to create a voice that converts (e.g., reads) the text as speech and thus can be provided as speech output 814); and
processing, using the TTS model, the textual content to be provided for presentation to the representative associated with the entity and the second set of prosodic properties to generate one or more of the corresponding instances of synthesized speech audio data (Rasipuram, para 0266-0268, In some examples, the machine learning model (e.g., neural network) performs global style token (GST) embedding, Bidirectional Encoder Representations from Transformers (BERT) embedding, natural language processing, and/or text to speech processing… In some examples, system 800 determines a theme of text 802. Exemplary themes include love, redemption, courage, revenge, coming of age, chaos, order, good and evil, struggle for power, family, faith, hubris, identity, justice, corruption, survival, war, and other similar concepts that are often conveyed through text such as novels or short stories. System 800 processes text 802 to determine a theme of the text or associated with the text. For example, system 800 may determine that based on the sentences “‘Do you want to come with me?’” and “‘No I'm supposed to hang out with my sister,’ Jack replied” a theme of the text is family. In some examples, system 800 determines the genre of text 802 based on the determined theme. For example, when the determined theme is family, system 800 may determine that a genre of the text could be children's books... Thus, the determined genre can be indicative of the determined theme and can allow system 800 and prosody predictor 804 to better determine the appropriate prosody or voice to apply when creating output 810 from text 802).
Regarding Claim 4. Feuz in view of Siefken, Kesanupalli and Rasipuram disclose the method of claim 1.
Furthermore, Rasipuram teaches:
wherein the initial voice is associated with a first text-to-speech (TTS) model (Rasipuram. para 0300, In some examples, the prosody is representative of a genre of the text. For example, as discussed above, the text may be analyzed to determine a theme of the text and/or to determine specific words which indicate a genre of the text. Thus, system 800 may determine a prosody that fits the determined genre and adjust and/or determine prosody for the speakers that is consistent with the determined genre. Accordingly, if the determined genre is children's a happy prosody may be generated for Christina, while if the genre is horror a scared or concerned prosody may be generated), wherein the alternative voice is associated with a second TTS model (Rasipuram. para 0301-0303, When text 802 is associated with multiple tags for different speakers of text 802, tag detector 810 detects each of the tags and determines (e.g., assigns) the appropriate prosody to each of the tags. Thus, tag detector 810 may determine multiple different prosodies for the different tags, even if the tags include one or more attributes that are the same. In particular, each of the different prosodies can include at least one different prosodic quality (e.g., frequency, pitch, cadence, spectral tilt, etc.) to differentiate the speakers from each other. In some examples, tag detector 810 includes a machine learning model (e.g., neural network) trained to determine a prosody based on a recognized tag. The machine learning model is trained in a similar manner to the machine learning models discussed above with respect to prosody predictor 804, semantic encoder 806, and tag encoder 808. Thus, tags and their associated prosodies may be provided to the machine learning model iteratively so that the machine learning model learns connections between the tags and the prosodies (e.g., characteristics of the tags correspond to the determined prosody) and can efficiently determine and assign a prosody based on those connections. In some examples, tag detector 810 includes a machine learning model (e.g., neural network) trained to determine a prosody based on the text. In some examples, tag detector 810 is at least partially combined with prosody predictor 804. For example, prosody predictor 804 may determine a genre for the text as discussed above, while tag detector 810 determines tags corresponding to the text. Thus, prosody predictor 804 and tag detector 810 may work together to determine the speech characteristics for the character Christina based on both a genre of the text and characteristics of the tag associated with Christina), and wherein the second TTS model differs from the first TTS model (Rasipuram. para 0305-0306, Thus, system 800 may be implemented as a number of different machine learning models or as a single machine learning model depending on the type of processing needed for the current application, the training data available, and other factors such as the amount of processing available, the structures of the machine learning models, etc. After determining the semantic meaning of text 802 and a prosody, system 800 generates speech output 814 of text 802 with audio generator 812. Similarly, after determining a prosody based on one or more tags, system 800 generates speech output 814 of text 802 with audio generator 812. Speech output 814 can be generated with various text to speech technologies including text to speech techniques discussed herein. Speech output 814 is generated using the prosody or speech characteristics determined to create a voice that converts (e.g., reads) the text as speech and thus can be provided as speech output 814).
Regarding Claim 5. Feuz in view of Siefken, Kesanupalli and Rasipuram disclose the method of claim 4
Furthermore, Feuz teaches:
when the initial voice is being utilized by the automated assistant and during the automated telephone call with the entity (Feuz, para 0016, In various implementations, techniques described herein may be employed such that when Dave invokes seeks to communicate with Alice's automated assistant, the female voice synthesizer employed by Alice may be activated for the automated instance invoked by Dave; [“employ” or “invoke” as “utilized”; “communicate” as “automated telephone call”]).
processing, using the first TTS model, textual content to be provided for presentation to a representative associated with the entity to generate one or more of the corresponding instances of synthesized speech audio data (Feuz, para 0058, In some implementations, TTS module 116 may be configured to convert text to computer-synthesized speech, e.g., using one or more voice synthesizers that may be, for instance, selected by a user, selected automated based on a user's region and/or demographics, etc.); and
when the alternative voice is being utilized by the automated assistant and during the automated telephone call with the entity (Feuz, para 0058-0059, In some implementations, TTS module 116 may be configured to convert text to computer-synthesized speech, e.g., using one or more voice synthesizers that may be, for instance, selected by a user, selected automated based on a user's region and/or demographics, etc. And as noted elsewhere herein, all or parts of modules 116, 117, and 112 may be implemented on client device 106, in addition to or instead of on the cloud. In some implementations, automated assistant 120 generates responsive content in response to various inputs generated by a user of one of the client devices 106 during a human-to-computer dialog session with automated assistant 120. Automated assistant 120 may provide the responsive content (e.g., over one or more networks when separate from a client device of a user) for presentation to the user as part of the dialog session. For example, automated assistant 120 may generate responsive content in response to free-form natural language input provided via client device 106; [i.e., “during a human-to-computer dialog session with automated assistant” as “during the automated telephone call” ]).
processing, using the second TTS model, the textual content to be provided for presentation to the representative associated with the entity to generate one or more of the corresponding instances of synthesized speech audio data (Feuz, para 0098, The digital component can be configured for a parametrically driven text to speech technique. The digital component can be configured for text-to-speech (TTS) implementations that convert normal language text into speech. For example, the digital component can include an image that is displayed to the user and, via TTS, text related to the displayed image is presented to the user. The digital component can be input to an application programming interface that utilizes a speech-synthesis capability to synthesize text into natural-sounding speech in a variety of languages, accents, and voices).
Regarding Claim 6. Feuz in view of Siefken, Kesanupalli and Rasipuram disclose the method of claim 1, wherein selecting the initial voice to be utilized by the automated assistant and during the automated telephone call with the entity is based on one or more of: a type of the entity (Feuz, para 0016, selecting different voice synthesizers. Techniques described herein may leverage these customizations in order to strengthen the appearance of speaking with another user's automated assistant. For instance, Dave may set his automated assistant client to a male voice, and Alice may set her automated assistant client to a female voice; [“male” or “female” as “a type of the entity”]), a particular location associated with the entity (Feuz, para 0058, TTS module 116 may be configured to convert text to computer-synthesized speech, e.g., using one or more voice synthesizers that may be, for instance, selected by a user, selected automated based on a user's region and/or demographics, etc.; [i.e., “demographics” as “type of the entity”; “user's region” as “a particular location”]), or whether a phone number associated with the entity is a landline or non-landline.
Regarding Claim 19. Feuz discloses a system (Feuz, para 0028, The system can include one or more server computing devices that implement a server portion of one or more automated assistants. The system can include, implement or execute a first automated assistant and a second automated assistant) comprising:
at least one processor (Feuz, para 0020, … performed by one or more processors); and
memory storing instructions that, when executed by the at least one processor, cause the at least one processor (Feuz, para 0027, the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the aforementioned methods):
identify an entity for an automated assistant to engage with during an automated telephone call (Feuz, para 0065, annotate references to an entity at a high level of granularity (e.g., to enable identification of all references to an entity class such as people) and/or a lower level of granularity (e.g., to enable identification of all references to a particular entity such as a particular person). The entity tagger may rely on content of the natural language input to resolve a particular entity and/or may optionally communicate with a knowledge graph or other entity database to resolve a particular entity; [“communicate” may include “automated telephone call”]);
select an initial voice to be utilized by the automated assistant and during the automated telephone call with the entity, the initial voice to be utilized by the automated assistant in generating one or more corresponding instances of synthesized speech to be rendered during the automated telephone call with the entity (Feuz, para 0016, Users often customize various aspects of their automated assistant experiences, e.g., by selecting different voice synthesizers. Techniques described herein may leverage these customizations in order to strengthen the appearance of speaking with another user's automated assistant. For instance, Dave may set his automated assistant client to a male voice, and Alice may set her automated assistant client to a female voice. Consequently, when Dave invokes an instance of an automated assistant (and assuming his identity is ascertained), he hears a male voice. Likewise, when Alice invokes an instance of an automated assistant, she hears a female voice. In various implementations, techniques described herein may be employed such that when Dave invokes seeks to communicate with Alice's automated assistant, the female voice synthesizer employed by Alice may be activated for the automated instance invoked by Dave. Consequently, even though Dave may still be interacting with his own automated assistant client executing on his own computing device, he nevertheless hears the voice of Alice's assistant, effectively providing Dave with the experience of speaking with Alice's assistant);
initiate the automated telephone call with the entity (Feuz, para 0041, to determine the second user's availability, the first user might call…the second user; [i.e., the first user initiating the call; “the second user” as “the entity”]); and
during the automated telephone call with the entity (Feuz, para 0041, However, the second user may not be available to answer the first user's question, or might prefer not to be interrupted. In such a situation an automated assistant could step in and provide information to the first user about the second user's availability; [i.e., the automated assistant could step in “during the … telephone call with the entity”]).
in response to determining to select the alternative voice to be utilized by the automated assistant and during the automated telephone call with the entity (Feuz, para 0058-0059, using one or more voice synthesizers that may be, for instance, selected by a user, selected automated based on a user's region and/or demographics, etc. And as noted elsewhere herein, all or parts of modules 116, 117, and 112 may be implemented on client device 106, in addition to or instead of on the cloud. In some implementations, automated assistant 120 generates responsive content in response to various inputs generated by a user of one of the client devices 106 during a human-to-computer dialog session with automated assistant 120. Automated assistant 120 may provide the responsive content (e.g., over one or more networks when separate from a client device of a user) for presentation to the user as part of the dialog session).
cause the automated assistant to utilize the alternative voice in continuing with the automated telephone call (Feuz, para 0132, in some implementations when a first user attempts to engage with an automated assistant that serves a second user, a voice synthesizer selected by the second user when they engage with automated assistant 120 may be used to communicate with the first user, thereby giving the illusion to the first user of speaking with the second user's automated assistant).
Feuz does not specifically disclose receiving audio data capturing speech provided by a representative associated with the entity, the audio data being received during the automated telephone call, and the representative associated with the entity being one of: a human representative associated with the entity, a voice bot associated with the entity, or an interactive voice response system associated with the entity, and determining , based on the initial voice that was selected to be utilized by the automated assistant during the automated telephone call and based on a language or an accent of the speech provided by the representative associated with the entity during the automated telephone call, whether to select an alternative voice to be utilized, and in lieu of the initial voice, by the automated assistant and during the automated telephone call with the entity.
However, Siefken, in the same field of endeavor, discloses:
receive audio data capturing speech provided by a representative associated with the entity, the audio data being received during the automated telephone call, and the representative associated with the entity being one of: a human representative associated with the entity, a voice bot associated with the entity, or an interactive voice response system associated with the entity (Siefken, para 0123, telephony provider 111 receives call 103 and notifies the voice recognition system 125 via a pre-configured automated message (e.g., via a webhook URL, such as ‘AnswerURL’ shown in FIG. 13). Upon receiving notification of the call 103, the voice recognition system 125 generates a response (e.g., a provider call control object PCCO), which may include a unique URL for the telephony provider 111 to create a web socket connection back to the voice recognition system 125. The telephony provider 111 may then open the connection to the unique URL and associate the call 103 (e.g., a live phone call) with the original notification from the telephony provider 111. These processes may be performed at the beginning of the interaction of the customer 101 with the voice recognition system 125 to facilitate communication and collection of audio data).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Siefken in the method of Feuz because this would enable restaurants or other businesses to accept new orders from customers via telephone, or other audio input methods, in an automated way, without intervention by the business, or with more limited intervention by the business (Siefken, para 0062).
Feuz in view of Siefken do not specifically disclose determine, based on the initial voice that was selected to be utilized by the automated assistant during the automated telephone call and based on a language or an accent of the speech provided by the representative associated with the entity during the automated telephone call, whether to select an alternative voice to be utilized, and in lieu of the initial voice, by the automated assistant and during the automated telephone call with the entity.
However, Kesanupalli , in the same field of endeavor, discloses determine, based on the initial voice that was selected to be utilized by the automated assistant during the automated telephone call and based on a language or an accent of the speech provided by the representative associated with the entity during the automated telephone call, whether to select an alternative voice to be utilized, and in lieu of the initial voice, by the automated assistant and during the automated telephone call with the entity (Kesanupalli, col 18, ln 49-60, In a presently preferred embodiment, when a user 114 calls the telephony server 404 via the telephone 430, the initial voice prompts are played in the default language of the telephony server 404 until the time that the user 114 actually signs on to the telephony server 404. For example, in the United States, the default locale would generally be English/U.S. with English being the default language, while in Japan, the default locale would generally be Japanese/Japan with Japanese being the default language. Once the user 114 identifies himself/herself by signing on, the subsequent voice prompts are played in the user's 114 language of choice selected at the time of registration).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Kesanupalli in the method of Feuz in view of Siefken because this has the advantage of accessing information services in the traveling user's language, independent of the country in which the user may be situated at the time of access (Kesanupalli, col 2, ln 21-24).
Feuz in view of Siefken and Raissyan do not specifically disclose select, based on the language or the accent of the speech provided by the representative associated with the entity during the automated telephone call, the alternative voice to be utilized by the automated assistant and during the automated telephone call with the entity, the alternative voice to be utilized by the automated assistant in generating the one or more corresponding instances of synthesized speech to be rendered during the automated telephone call with the entity.
However, Rasipuram, in the same field of endeavor discloses select the alternative voice to be utilized by the automated assistant and during the automated telephone call with the entity, the alternative voice to be utilized by the automated assistant in generating the one or more corresponding instances of synthesized speech to be rendered during the automated telephone call with the entity (Rasipuram, para 0013, Determining, based on the text, a tag corresponding to a speaker of the text; and determining, based on the tag, a prosody for the speaker allows for automatic and efficient creation of realistic sounding audio books. In particular, by determining a tag corresponding to a speaker of the text voices for different speakers of the text may be determined and used to generate speech which may be combined into a book. This audio file may be created with an efficient use of processing power to create multiple voices for a more engaging and interesting interaction for the listener; ALTERNATIVE, Rasipuram, para 0239-0250, In some examples, task flow processing module 736 employs the assistance of service processing module 738 (“service processing module”) to complete a task requested in the user input or to provide an informational answer requested in the user input. For example, service processing module 738 acts on behalf of task flow processing module 736 to make a phone call…Additional details on digital assistants can be found in the U.S. Utility application Ser. No. 12/987,982, entitled “Intelligent Automated Assistant” … Further, by considering the context of the text, the expressivity or style of a selected voice can be adjusted to create a realistic voice that users enjoy listening to. Moreover, the style of the selected voice can be adjusted based on a genre and/or themes of text to create a voice that provides a consistent experience with the desired genre or theme; para 0239, task flow processing module 736 employs the assistance of service processing module 738 (“service processing module”) to complete a task requested in the user input or to provide an informational answer requested in the user input. For example, service processing module 738 acts on behalf of task flow processing module 736 to make a phone call; Rasipuram, para 0271, prosody includes one or more of a rate of speech, a volume level, a level of excitedness, an accent of speech, a pitch of a voice, timbre, frequency, or any other characteristic that affects the way that speech output 814 sounds; [i.e., meaning the digital assistant automatically responds the call based on the requested language, for instance.]); and
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Rasipuram in the method of Feuz in view of Siefken and Kesanupalli because this would enable an appropriate voice for reading of a children's book and thus a voice that is cheerful and light can be used instead of, for example, a voice that is suspenseful or angry which would be inappropriate (Rasipuram, para 0250).
Regarding Claim 20. Feuz discloses a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to be operable (Feuz, para 0027, one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the aforementioned methods) to:
identifying an entity for an automated assistant to engage with during an automated telephone call (Feuz, para 0065, annotate references to an entity at a high level of granularity (e.g., to enable identification of all references to an entity class such as people) and/or a lower level of granularity (e.g., to enable identification of all references to a particular entity such as a particular person). The entity tagger may rely on content of the natural language input to resolve a particular entity and/or may optionally communicate with a knowledge graph or other entity database to resolve a particular entity; [“communicate” may include “automated telephone call”]);
selecting an initial voice to be utilized by the automated assistant and during the automated telephone call with the entity, the initial voice to be utilized by the automated assistant in generating one or more corresponding instances of synthesized speech to be rendered during the automated telephone call with the entity (Feuz, para 0016, Users often customize various aspects of their automated assistant experiences, e.g., by selecting different voice synthesizers. Techniques described herein may leverage these customizations in order to strengthen the appearance of speaking with another user's automated assistant. For instance, Dave may set his automated assistant client to a male voice, and Alice may set her automated assistant client to a female voice. Consequently, when Dave invokes an instance of an automated assistant (and assuming his identity is ascertained), he hears a male voice. Likewise, when Alice invokes an instance of an automated assistant, she hears a female voice. In various implementations, techniques described herein may be employed such that when Dave invokes seeks to communicate with Alice's automated assistant, the female voice synthesizer employed by Alice may be activated for the automated instance invoked by Dave. Consequently, even though Dave may still be interacting with his own automated assistant client executing on his own computing device, he nevertheless hears the voice of Alice's assistant, effectively providing Dave with the experience of speaking with Alice's assistant);
initiating the automated telephone call with the entity (Feuz, para 0041, to determine the second user's availability, the first user might call…the second user; [i.e., the first user initiating the call; “the second user” as “the entity”]); and
during the automated telephone call with the entity (Feuz, para 0041, However, the second user may not be available to answer the first user's question, or might prefer not to be interrupted. In such a situation an automated assistant could step in and provide information to the first user about the second user's availability; [i.e., the automated assistant could step in “during the … telephone call with the entity”]).
in response to determining to select the alternative voice to be utilized by the automated assistant and during the automated telephone call with the entity (Feuz, para 0058-0059, using one or more voice synthesizers that may be, for instance, selected by a user, selected automated based on a user's region and/or demographics, etc. And as noted elsewhere herein, all or parts of modules 116, 117, and 112 may be implemented on client device 106, in addition to or instead of on the cloud. In some implementations, automated assistant 120 generates responsive content in response to various inputs generated by a user of one of the client devices 106 during a human-to-computer dialog session with automated assistant 120. Automated assistant 120 may provide the responsive content (e.g., over one or more networks when separate from a client device of a user) for presentation to the user as part of the dialog session).
causing the automated assistant to utilize the alternative voice in continuing with the automated telephone call (Feuz, para 0132, in some implementations when a first user attempts to engage with an automated assistant that serves a second user, a voice synthesizer selected by the second user when they engage with automated assistant 120 may be used to communicate with the first user, thereby giving the illusion to the first user of speaking with the second user's automated assistant).
Feuz does not specifically disclose receiving audio data capturing speech provided by a representative associated with the entity, the audio data being received during the automated telephone call, and the representative associated with the entity being one of: a human representative associated with the entity, a voice bot associated with the entity, or an interactive voice response system associated with the entity, and determining , based on the initial voice that was selected to be utilized by the automated assistant during the automated telephone call and based on a language or an accent of the speech provided by the representative associated with the entity during the automated telephone call, whether to select an alternative voice to be utilized, and in lieu of the initial voice, by the automated assistant and during the automated telephone call with the entity.
However, Siefken, in the same field of endeavor, discloses:receiving audio data capturing speech provided by a representative associated with the entity, the audio data being received during the automated telephone call, and the representative associated with the entity being one of: a human representative associated with the entity, a voice bot associated with the entity, or an interactive voice response system associated with the entity (Siefken, para 0123, telephony provider 111 receives call 103 and notifies the voice recognition system 125 via a pre-configured automated message (e.g., via a webhook URL, such as ‘AnswerURL’ shown in FIG. 13). Upon receiving notification of the call 103, the voice recognition system 125 generates a response (e.g., a provider call control object PCCO), which may include a unique URL for the telephony provider 111 to create a web socket connection back to the voice recognition system 125. The telephony provider 111 may then open the connection to the unique URL and associate the call 103 (e.g., a live phone call) with the original notification from the telephony provider 111. These processes may be performed at the beginning of the interaction of the customer 101 with the voice recognition system 125 to facilitate communication and collection of audio data).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Siefken in the method of Feuz because this would enable restaurants or other businesses to accept new orders from customers via telephone, or other audio input methods, in an automated way, without intervention by the business, or with more limited intervention by the business (Siefken, para 0062).
Feuz in view of Siefken do not specifically disclose determining , based on the initial voice that was selected to be utilized by the automated assistant during the automated telephone call and based on a language or an accent of the speech provided by the representative associated with the entity during the automated telephone call, whether to select an alternative voice to be utilized, and in lieu of the initial voice, by the automated assistant and during the automated telephone call with the entity.
However, Kesanupalli, in the same field of endeavor, discloses determining , based on the initial voice that was selected to be utilized by the automated assistant during the automated telephone call and based on a language or an accent of the speech provided by the representative associated with the entity during the automated telephone call, whether to select an alternative voice to be utilized, and in lieu of the initial voice, by the automated assistant and during the automated telephone call with the entity (Kesanupalli, col 18, ln 49-60, In a presently preferred embodiment, when a user 114 calls the telephony server 404 via the telephone 430, the initial voice prompts are played in the default language of the telephony server 404 until the time that the user 114 actually signs on to the telephony server 404. For example, in the United States, the default locale would generally be English/U.S. with English being the default language, while in Japan, the default locale would generally be Japanese/Japan with Japanese being the default language. Once the user 114 identifies himself/herself by signing on, the subsequent voice prompts are played in the user's 114 language of choice selected at the time of registration).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Kesanupalli in the method of Feuz in view of Siefken because this has the advantage of accessing information services in the traveling user's language, independent of the country in which the user may be situated at the time of access (Kesanupalli, col 2, ln 21-24).
Feuz in view of Siefken and Kesanupalli do not specifically disclose selecting, based on the language or the accent of the speech provided by the representative associated with the entity during the automated telephone call, the alternative voice to be utilized by the automated assistant and during the automated telephone call with the entity, the alternative voice to be utilized by the automated assistant in generating the one or more corresponding instances of synthesized speech to be rendered during the automated telephone call with the entity.
However, Rasipuram, in the same field of endeavor discloses selecting, based on the language or the accent of the speech provided by the representative associated with the entity during the automated telephone call, the alternative voice to be utilized by the automated assistant and during the automated telephone call with the entity, the alternative voice to be utilized by the automated assistant in generating the one or more corresponding instances of synthesized speech to be rendered during the automated telephone call with the entity (Rasipuram, para 0013, Determining, based on the text, a tag corresponding to a speaker of the text; and determining, based on the tag, a prosody for the speaker allows for automatic and efficient creation of realistic sounding audio books. In particular, by determining a tag corresponding to a speaker of the text voices for different speakers of the text may be determined and used to generate speech which may be combined into a book. This audio file may be created with an efficient use of processing power to create multiple voices for a more engaging and interesting interaction for the listener; Rasipuram, para 0035-0036, FIG. 1 illustrates a block diagram of system 100 according to various examples. In some examples, system 100 implements a digital assistant. The terms “digital assistant,” “virtual assistant,” “intelligent automated assistant,” or “automatic digital assistant” refer to any information processing system that interprets natural language input in spoken and/or textual form to infer user intent, and performs actions based on the inferred user intent. For example, to act on an inferred user intent, the system performs one or more of the following: identifying a task flow with steps and parameters designed to accomplish the inferred user intent, inputting specific requirements from the inferred user intent into the task flow; executing the task flow by invoking programs, methods, services, APIs, or the like; and generating output responses to the user in an audible (e.g., speech) and/or visual form. Specifically, a digital assistant is capable of accepting a user request at least partially in the form of a natural language command, request, statement, narrative, and/or inquiry. Typically, the user request seeks either an informational answer or performance of a task by the digital assistant; [i.e., automatic response by the digital assistant which is capable of accepting a user request at least partially in the form of a natural language command, request, statement, narrative, and/or inquiry and perform as intended; para 0239, task flow processing module 736 employs the assistance of service processing module 738 (“service processing module”) to complete a task requested in the user input or to provide an informational answer requested in the user input. For example, service processing module 738 acts on behalf of task flow processing module 736 to make a phone call; Rasipuram, para 0271, prosody includes one or more of a rate of speech, a volume level, a level of excitedness, an accent of speech, a pitch of a voice, timbre, frequency, or any other characteristic that affects the way that speech output 814 sounds; [i.e., meaning the digital assistant automatically responds the call based on the requested language, for instance.] ); and
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Rasipuram in the method of Feuz in view of Siefken and Kesanupalli because this would enable an appropriate voice for reading of a children's book and thus a voice that is cheerful and light can be used instead of, for example, a voice that is suspenseful or angry which would be inappropriate (Rasipuram, para 0250).
Regarding Claim 21. Feuz in view of Siefken, Raissyan and Rasipuram disclose the system of claim 19.
Furthermore, Rasipuram teaches:
wherein the initial voice is associated with a first set of prosodic properties, wherein the alternative voice is associated with a second set of prosodic properties, and wherein the second set of prosodic properties differs from the first set of prosodic properties (Rasipuram, para 0260, system 800 produces a complete audio output (e.g., book, script, article, paper, etc.) of the input text using the appropriate prosody or prosodies (e.g., voices); para 0272-0273, In some examples, the determined prosody is indicative of a particular voice or voice profile. For example, prosody predictor 804 and system 800 may determine that for a children's book the speaker of audio output 810 should be a bright individual who is excited to read and thus determine a prosody with characteristics that match that type of voice profile. In some examples, prosody is represented with a global style token or other data structure. Thus prosody predictor 804 may determine the prosody and generate a global style token that may be provided to other components of system 800 to reference. In this way, the determined prosody can be referenced simply through the token or data structure. For example, prosody predictor 804 may provide a global style token which indicates a high volume, quick rate of speech, and British accent); [i.e., various prosodies are used for creating various voices, including accents]).
Regarding Claim 22. Feuz in view of Siefken, Raissyan and Rasipuram disclose the system of claim 19.
Furthermore, Rasipuram teaches:
wherein the initial voice is associated with a first text-to-speech (TTS) model, wherein the alternative voice is associated with a second TTS model, and wherein the second TTS model differs from the first TTS model (Rasipuram, para 0260, system 800 produces a complete audio output (e.g., book, script, article, paper, etc.) of the input text using the appropriate prosody or prosodies (e.g., voices) Rasipuram, para 0305-0306, Thus, system 800 may be implemented as a number of different machine learning models or as a single machine learning model depending on the type of processing needed for the current application, the training data available, and other factors such as the amount of processing available, the structures of the machine learning models, etc. After determining the semantic meaning of text 802 and a prosody, system 800 generates speech output 814 of text 802 with audio generator 812. Similarly, after determining a prosody based on one or more tags, system 800 generates speech output 814 of text 802 with audio generator 812. Speech output 814 can be generated with various text to speech technologies including text to speech techniques discussed herein. Speech output 814 is generated using the prosody or speech characteristics determined to create a voice that converts (e.g., reads) the text as speech and thus can be provided as speech output 814).
Regarding Claim 23. Feuz in view of Siefken, Raissyan and Rasipuram disclose the method of claim 1, wherein determining to select the alternative voice to be utilized by the automated assistant and during the automated telephone call with the entity occurs prior to causing any of the one or more corresponding instances of synthesized speech being rendered during the automated telephone call with the entity (Feuz, para 0016, Users often customize various aspects of their automated assistant experiences, e.g., by selecting different voice synthesizers. Techniques described herein may leverage these customizations in order to strengthen the appearance of speaking with another user's automated assistant. For instance, Dave may set his automated assistant client to a male voice, and Alice may set her automated assistant client to a female voice. Consequently, when Dave invokes an instance of an automated assistant (and assuming his identity is ascertained), he hears a male voice. Likewise, when Alice invokes an instance of an automated assistant, she hears a female voice. In various implementations, techniques described herein may be employed such that when Dave invokes seeks to communicate with Alice's automated assistant, the female voice synthesizer employed by Alice may be activated for the automated instance invoked by Dave. Consequently, even though Dave may still be interacting with his own automated assistant client executing on his own computing device, he nevertheless hears the voice of Alice's assistant, effectively providing Dave with the experience of speaking with Alice's assistant).
Claims 7, 11-18 are rejected under 35 U.S.C. 103 as being unpatentable over Feuz in view of Siefken, further in view of Raissyan, further in view of Rasipuram, and further in view of Yuval et al. Pat App No. US 20220006900 A1 (Yuval).
Regarding Claim 7. Feuz in view of Siefken, Raissyan and Rasipuram disclose the method of claim 1.
Feuz in view of Siefken, Raissyan and Rasipuram does not specifically disclose wherein determining whether to select the alternative voice to be utilized, and in lieu of the initial voice, by the automated assistant and during the automated telephone call with the entity is based on analyzing content received upon initiating the automated telephone call with the entity.
However, Yuval, in the same field of endeavor, discloses wherein determining whether to select the alternative voice to be utilized, and in lieu of the initial voice, by the automated assistant and during the automated telephone call with the entity is further based on analyzing content received upon initiating the automated telephone call with the entity (Yuval, para 0004, Implementations are directed to using an automated assistant to initiate and perform automated telephone call(s). In some implementations, the automated telephone call(s) can be initiated and performed in response to a request to initiate and perform the automated telephone call(s). The automated assistant can identify a group of entities associated with the request, and can initiate a corresponding automated telephone call with one or more of the entities of the group to perform an action associated with the request. Further, the automated assistant can perform the action associated with the request through rendering instance(s) of synthesized speech related to the request. At least some of the instance(s) of synthesized speech can be generated based on the request and based on processing response(s), during the automated telephone call, that are from a representative of the entity that is participating in the telephone call).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Yuval in the method of Feuz in view of Siefken, Raissyan and Rasipuram because this would enable performing requests through rendering the instance(s) of the synthesized speech the automated assistant and continue initiating the automated telephone call(s) with the entities of the group until the request is satisfied (Yuval, para 0009).
Regarding Claim 11. Feuz in view of Siefken, Raissyan, Rasipuram disclose the method of claim 1.
Feuz in view of Siefken, Raissyan and Rasipuram does not specifically disclose wherein identifying the entity for the automated assistant to engage with during the automated telephone call is based on user input that is received at a client device of a user, and wherein the automated assistant initiates and conducts the automated telephone call on behalf of the user.
However, Yuval, in the same field of endeavor, discloses wherein identifying the entity for the automated assistant to engage with during the automated telephone call is based on user input that is received at a client device of a user, and wherein the automated assistant initiates and conducts the automated telephone call on behalf of the user (from a computing device of a user, a request to initiate performance of an automated telephone call on behalf of the user, identifying, based on the request, an action to be performed during the automated telephone call, identifying, based on the action to be performed during the automated call, a group of entities that are each capable of satisfying the request, and initiating, by an automated assistant, performance of the automated telephone call with a given entity, of the entities, of the group).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Yuval in the method of Feuz in view of Siefken, Raissyan and Rasipuram because this would enable performing requests through rendering the instance(s) of the synthesized speech the automated assistant and continue initiating the automated telephone call(s) with the entities of the group until the request is satisfied (Yuval, para 0009).
Regarding Claim 12. Feuz in view of Siefken, Raissyan, Rasipuram and Yuval disclose the method of claim 11.
Furthermore, Yuval discloses:
subsequent to the automated assistant completing the automated telephone call (Yuval, para 0087, the automated assistant can initiate and perform a subsequent corresponding automated telephone call with the corresponding entity to request a corresponding further status update regarding the item):
generating, based on a result of the automated telephone call, a notification (Yuval, para 0010, the automated assistant can generate notification(s) based on a result of one or more of the automated telephone call(s));
and
causing the notification to be rendered for presentation to the user via the client device (Yuval, para 0079, At block 368, the system causes the notification(s) to be rendered visually and/or audibly via the computing device of the user).
Regarding Claim 13. Feuz in view of Siefken, Raissyan, Rasipuram and Yuval disclose the method of claim 11.
Furthermore, Yuval discloses:
wherein the automated assistant is executed locally at the client device of the user (para 0021-0022, the automated telephone calls described herein can be initiated and performed using an automated assistant 115 that is executed locally at the client device 110 (e.g., a local assistant), remotely at one or more servers (e.g., a cloud-based assistant), and/or a combination thereof (e.g., as indicated by dashed lines in FIG. 1). In various implementations, the automated assistant 115 can initiate and perform the automated telephone calls using automated request system 180. In implementations where the automated assistant 115 is executed locally at the client device 110, the automated request system 180 can be executed locally on the client device 110 such that the automated telephone calls are initiated and performed using only resources of the client device 110).
Regarding Claim 14. Feuz in view of Siefken, Raissyan, Rasipuram and Yuval disclose the method of claim 11, wherein the automated assistant is executed remotely from the client device of the user (Feuz, para 0019-0024, Formulating responses to relatively open-ended and/or broad requests enables automated assistants to “hide” individual data points in relatively broad answers, preserving a user's privacy in specific data points…users are able to determine information about other users without actually establishing communications (e.g., telephone calls, electronic correspondence) with the other users, or without having to repeatedly attempt to establish communications with the other users…The request may be a request for information and causing the client device to respond to the request for information may comprise causing the client device to output a natural language response to the request for information. Alternatively, the request may be a request to control one or more devices, and causing the device to respond to the request may comprise transmitting instructions to the one or more devices. … the second data source is available on one or more servers that are remote from any client computing device operated by the second user).
Regarding Claim 15. Feuz in view of Siefken, Raissyan, Rasipuram disclose the method of claim 1.
Feuz in view of Siefken, Raissyan and Rasipuram does not specifically disclose wherein identifying the entity for the automated assistant to engage with during the automated telephone call is based on a spike in query activity across a population of client devices in a certain geographical area, and wherein the automated assistant initiates and conducts the automated telephone call on behalf of the population of client devices.
However, Yuval, in the same field of endeavor, discloses:
wherein identifying the entity for the automated assistant to engage with during the automated telephone call is based on a spike in query activity across a population of client devices in a certain geographical area (Yuval, para 0025 – para 0028, the recognized text to determine at least an intent of finding toilet paper proximate to a current location of the given user as an action included in the request to initiate and perform the automated telephone calls… In some implementations, the query analysis engine 151 can analyze query activity stored in query activity database 151A, and can transmit a request to initiate and perform the automated telephone calls in response to determining one or more conditions are satisfied. The query activity analyzed by the query analysis engine 151 can include queries submitted to a search engine (e.g., via a search interface or automated assistant interface) by a plurality of users using respective client devices), and wherein the automated assistant initiates and conducts the automated telephone call on behalf of the population of client devices (Yuval, para 0028, the automated assistant 115 can initiate and perform the automated telephone calls, using the automated request system 180, without detecting any user input via the user interface input engine 111).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Yuval in the method of Feuz in view of Siefken, Raissyan and Rasipuram because this would enable performing requests through rendering the instance(s) of the synthesized speech the automated assistant and continue initiating the automated telephone call(s) with the entities of the group until the request is satisfied (Yuval, para 0009).
Regarding Claim 16. Feuz in view of Siefken, Raissyan, Rasipuram and Yuval disclose F the method of claim 15.
Furthermore, Yuval discloses:
subsequent to the automated assistant completing the automated telephone call (Yuval, para 0007, the automated assistant will initiate a subsequent corresponding automated telephone call with the given entity):
updating, based on a result of the automated telephone call, one or more databases (The automated assistant can then notify the user of a result of the automated telephone call and/or search results associated with the particular entity can be updated based on the result).
Regarding Claim 17. Feuz in view of Siefken, Raissyan, Rasipuram and Yuval disclose the method of claim 16, wherein the one or more databases are associated with a web browser software application or a maps software application (Feuz, para 0045-0053, In some implementations, a given user may communicate with automated assistant 120 utilizing a plurality of client computing devices 106 that collectively from a coordinated “ecosystem” of computing devices… A first automated assistant 120A can receive a text segment via a client device 106.sub.1 (or automated assistant client 118.sub.1 of the client device 106.sub.1)… The plurality of data sources can include a calendar database (e.g., online calendar 142), location service (e.g., position coordinate service 150), or a corpus of communication associated with the second electronic account (e.g., one or more of email 144, social media 146, or cloud storage 148)… Each of the client computing devices 106.sub.1-N may operate a variety of different applications, such as a corresponding one of a plurality of message exchange clients 107.sub.1-N. Message exchange clients 107.sub.1-N may come in various forms and the forms may vary across the client computing devices 106.sub.1-N and/or multiple forms may be operated on a single one of the client computing devices 106.sub.1-N. In some implementations, one or more of the message exchange clients 107.sub.1-N may come in the form of a short messaging service (“SMS”) and/or multimedia messaging service (“MMS”) client, an online chat client (e.g., instant messenger, Internet relay chat, or “IRC,” etc.), a messaging application associated with a social network, a personal assistant messaging service dedicated to conversations with automated assistant 120, and so forth. In some implementations, one or more of the message exchange clients 107.sub.1-N may be implemented via a webpage or other resources rendered by a web browser (not depicted) or other application of client computing device 106).
Regarding Claim 18. Feuz in view of Siefken, Raissyan, Rasipuram and Yuval disclose the method of claim 15, wherein the automated assistant is a cloud-based automated assistant (Feuz, para 0042, Now turning to FIG. 1, an example environment in which techniques disclosed herein may be implemented is illustrated. The example environment includes a plurality of client computing devices 106.sub.1-N. Each client device 106 may execute a respective instance of an automated assistant client 118. One or more cloud-based automated assistant components 119 may be implemented on one or more computing systems (collectively referred to as a “cloud” computing system) that are communicatively coupled to client devices 106.sub.1-N via one or more local and/or wide area networks (e.g., the Internet) indicated generally at 110.sub.1).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MULUGETA T. DUGDA whose telephone number is (703)756-1106. The examiner can normally be reached Mon - Fri, 4:30am - 7:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras D. Shah can be reached at 571-270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MULUGETA TUJI DUGDA/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
06/24/2026
Prosecution Insights