Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 2023-06-07, 2023-12-12, 2024-06-06, 2025-01-14, and 2025-02-18 have been considered by the examiner and made of record in the application file. However, examiner has only considered that which is in English. If applicant wishes examiner to consider foreign patent documents in their entirety, applicant is requested to submit English translations for consideration.
Specification
The abstract of the disclosure is objected to because it contains phrases such as “This document describes,” etc. Applicant is reminded of the proper language and format for an abstract of the disclosure. The language of the abstract should avoid using phrases which can be implied, such as, “The disclosure concerns,” “The disclosure defined by this invention,” “The disclosure describes,” etc.
A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP §608.01(b) for guidelines for the preparation of patent abstracts.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-4, 7, and 9-15 are rejected under 35 U.S.C. 102(a)(1) and 102(a)(2) as being anticipated by U.S. Patent Publication No. 2017/0339275 to Charignon (“Charignon”).
As to claim 1 (and similarly applied to claims 14 and 15), Charignon discloses a method (Fig. 6, ¶0051-53) comprising: obtaining, by a computing device, audio data output from a communication application executing on the computing device, the audio data comprising audible parts of a voice call or a video call between a user of the computing device and a third party (Figs. 1-3; ¶0039-40, the speech recognition module 306 analyzes audio from the third-party automated answering system/module); determining, by the computing device and using the audible parts, whether the audio data includes two or more selectable options, the two or more selectable options audibly provided by the third party during the voice call or the video call (¶0039-40, the speech recognition module 306 generates information about the menu options based on the incoming audio; Figs. 5A-D and ¶0045-49 disclose two or more selectable options audibly provided by the third party during the call); responsive to determining that the audio data includes the two or more selectable options, determining, by the computing device, a text description of the two or more selectable options, the text description providing a transcription of at least a portion of the two or more selectable options (¶0039-40, the speech recognition module 306 determines what options are available and converts the speech to text); and displaying two or more selectable controls on a display of the computing device, the two or more selectable controls configured to be selectable by the user to provide an indication to the third party of a selected option of the two or more selectable options, each of the two or more selectable controls providing the text description of a respective selectable option (¶0039-40, "the speech recognition module 306 can apply techniques to convert the speech to text so that such information can be provided for display to the user's computing device"; Figs. 5A-D and ¶0045-49 disclose displaying two or more user-selectable controls corresponding to the options audibly provided by the third party during the call).
Examiner notes that Fig. 8 and ¶0083-84 disclose a computing device comprising at least one processor and computer-readable storage medium comprising instructions that, when executed, configure a processor to perform the method as described above. Thus, claims 14 and 15 are similarly rejected.
As to claim 2, Charignon discloses the method of claim 1, the method further comprising: receiving a selection of one selectable control of the two or more selectable controls associated with the selected option, the selection made by the user during the voice call or the video call; and responsive to receiving the selection of the one selectable control, communicating, by the computing device, the selected option to the third party (Figs. 5A-D and ¶0045-49 disclose user selecting one of the two or more selectable controls, e.g., selecting one of two language options, the selected option then being communicated to the third party automated answering system).
As to claim 3, Charignon discloses the method of claim 2, wherein communicating the selected option to the third party comprises sending, by the computing device, an audio response or a dual-tone multi- frequency (DTMF) tone to the third party (¶0047, the user-selected option is mapped to a numerical key and corresponding DTMF tone to be recognized by the third party automated answering system) without the user audibly communicating the selected option (¶0047, the user selects the option via a gesture, e.g. a touch gesture).
As to claim 4, Charignon discloses the method of claim 2, the method further comprising: responsive to communicating the selected option to the third party, obtaining, by the computing device, additional audio data output from the communication application, the additional audio data including two or more additional selectable options audibly provided by the third party during the voice call or the video call in response to the selected option (Figs. 5A-D and ¶0045-49 disclose navigating through different automated answering system menus, with additional audio data conveying additional selectable options (e.g., Fig. 5C, "Scheduling", "Cancellation", "Billing Department", etc.) during the call).
As to claim 7, Charignon discloses the method of claim 1, wherein determining the text description of the two or more selectable options comprises executing, by the computing device, a machine-learned model to determine the text description of the two or more selectable options, the machine-learned model trained to determine text descriptions from the audio data, the audio data received from an audio mixer of the computing device (¶0039, "The speech recognition module 306 can utilize any generally known technique for performing speech recognition including Hidden Markov Models, neural networks, deep feedforward neural networks, to name some examples").
As to claim 9, Charignon discloses the method of claim 1, wherein the two or more selectable options are a menu representing options of an interactive voice response (IVR) system or a voice response unit (VRU) system (Figs 5A-D and ¶0045-49 disclose the two or more selectable options as menu(s) representing options of the IVR or VRU system, i.e. the third-party automated answering system (also see ¶0034)), the IVR system or VRU system configured to interact with the user and direct the user to at least one of another menu of the IVR system or VRU system, personnel associated with the third party, departments associated with the third party, services associated with the third party, or information associated with the third party (Figs. 5A-D and ¶0045-49 disclose multiple menus of the third-party automated answering system as well as departments, services, and information associated with the third party).
As to claim 10, Charignon discloses the method of claim 1, wherein the two or more selectable controls comprise at least one of buttons, toggles, selectable text, sliders, checkboxes, or icons and are included in a user interface of the communication application (Figs 5A-D and ¶0045-49 disclose selectable text/buttons/icons in a user interface).
As to claim 11, Charignon discloses the method of claim 1, wherein the text description includes a number associated with each of the two or more selectable options and wherein each of the selectable controls includes a visual representation of the number associated with each of the two or more selectable options (Figs 5A-D and ¶0045-49 disclose numbers associated with each of the two or more selectable options, including a visual representation of the number).
As to claim 12, Charignon discloses the method of claim 1, wherein the display of the computing device comprises a touch-sensitive screen (¶0047, the user can select the option with a "touch gesture") and wherein the selectable controls are presented on the touch-sensitive screen (Figs 5A-D and ¶0045-49 disclose the selectable controls presented on the touch-sensitive screen).
As to claim 13, Charignon discloses the method of claim 1, wherein the computing device comprises a smartphone, a computerized watch, a tablet device, a wearable device, or a laptop computer (Figs. 5A-D depict the computing device 502 as a smartphone or tablet; additionally, ¶0055 discloses that "the user device 710 can be a computing device or a device having computer functionality, such as a smart-phone, a tablet, a personal digital assistant (PDA), a mobile telephone, a laptop computer, a wearable device (e.g., a pair of glasses, a watch, a bracelet, etc.), a camera, an appliance, etc").
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Charignon in view of U.S. Patent No. 11,606,463 to Yeracaris et al. (“Yeracaris”).
As to claim 5, Charignon discloses the method of claim 1, the method further comprising: determining, by the computing device and using the audible parts, whether the audio data includes a request for user information, the request for user information audibly provided by the third party during the voice call or the video call (Charignon, ¶0049, the automated answering system may request information, e.g. social security number, from the user); and displaying, by the computing device, the user data on the display or providing, by the computing device, the user data to the third party during the voice call or the video call (Charignon, ¶0049, the computing device displays the type of user information requested, and "a virtual keypad may be presented on the display screen through which the user can input the requested information").
Charignon does not disclose: identifying, by the computing device and using the audible parts, user data responsive to the request for user information.
However, Yeracaris discloses: identifying, by the computing device and using the audible parts, user data responsive to the request for user information (Col 10: lines 8-11 and 27-47 describe automatic identification of personal information, e.g. a user's credit card number, based on a live conversation and using automatic speech recognition and/or natural language processing).
Charignon and Yeracaris are considered to be similar to the claimed invention because they are in one or more of the same fields of: interactive information services, e.g. directory enquiries, and arrangements therefore, e.g. interactive voice response (IVR) systems or voice portals; visual mode comprising meus; using speech recognition; computing arrangements based on biological models, e.g. recurrent networks; and/or providing customer assistance, e.g. assisting a customer via helpdesk and/or phone tree. As such, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Charignon to incorporate the teachings of Yeracaris to include: identifying, by the computing device and using the audible parts, user data responsive to the request for user information. Doing so would allow sensitive personal information to be redacted, censored, and/or broken up such that it could not be obtained by any singular actor, "so as to ensure that privacy is maintained" (Yeracaris, Col. 10: line 38).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Charignon in view of U.S. Patent Publication No. 2018/0032997 to Gordon et al. (“Gordon”).
As to claim 6, Charignon discloses the method of claim 1.
Charignon does not disclose: the method further comprising: determining, by the computing device and using the audible parts, whether the audio data includes communicated information, the communicated information related to a context of the voice call or the video call and audibly provided by the third party or the user during the voice call or the video call; responsive to determining that the audio data includes the communicated information, determining, by the computing device, a text description of the communicated information, the text description of the communicated information providing a transcription of at least a portion of the communicated information; and displaying another selectable control on the display, the other selectable control providing the text description of the communicated information and configured to be selectable by the user to save the communicated information in at least one of the computing device, the application, or another application on the computing device.
However, Gordon discloses: the method further comprising: determining, by the computing device and using the audible parts, whether the audio data includes communicated information, the communicated information related to a context of the voice call or the video call and audibly provided by the third party or the user during the voice call or the video call (Fig. 49-4; triggers cause an instruction to be executed [¶1730-1], and the triggers may include speech-to-text input for a note and may include context awareness features, i.e., can tell whether the audio data includes communicated information, e.g., "time, place, and/or other information associated with the…audio" [¶1733]); responsive to determining that the audio data includes the communicated information, determining, by the computing device, a text description of the communicated information, the text description of the communicated information providing a transcription of at least a portion of the communicated information (¶1733, the system uses speech-to-text); and displaying another selectable control on the display, the other selectable control providing the text description of the communicated information and configured to be selectable by the user (Fig. 49-5, the instruction may be displayed "by text (e.g. textual description of the one or more triggers and actions, etc.)" [¶1811] and modified, saved, and/or executed [¶1812-18]) to save the communicated information in at least one of the computing device, the application, or another application on the computing device (the note may be associated with an app, and may provide "the ability to track the note (e.g. note sent from user to contact…)" [¶1733]; additionally, the input may trigger "an action, including initiating a phone call (e.g. based on a contact listed in a note, etc.), sending a message (e.g. email, SMS, etc.), setting a reminder (e.g. calendar reminder, etc.), creating an event detail (e.g. calendar item, etc.), uploading information to an online server (e.g. social networking site, blog, etc.), and/or taking any other action in response to a note. In one embodiment, the note may be spoken as a voice command (e.g. “NOTE: remind me to clean the bathroom,” etc.)" [¶1734] ).
Charignon and Gordon are considered to be similar to the claimed invention because they are in one or more of the same fields of: interactive information services, e.g. directory enquiries, and arrangements therefore, e.g. interactive voice response (IVR) systems or voice portals; visual mode comprising meus; using speech recognition; computing arrangements based on biological models, e.g. recurrent networks; and/or providing customer assistance, e.g. assisting a customer via helpdesk and/or phone tree. As such, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Charignon to incorporate the teachings of Gordon to include: the method further comprising: determining, by the computing device and using the audible parts, whether the audio data includes communicated information, the communicated information related to a context of the voice call or the video call and audibly provided by the third party or the user during the voice call or the video call; responsive to determining that the audio data includes the communicated information, determining, by the computing device, a text description of the communicated information, the text description of the communicated information providing a transcription of at least a portion of the communicated information; and displaying another selectable control on the display, the other selectable control providing the text description of the communicated information and configured to be selectable by the user to save the communicated information in at least one of the computing device, the application, or another application on the computing device. Doing so would allow for improved user experience during a phone call, for example: automatic generation of suggested responses; automatic generation of important information to save; automatic generation of reminders to set; and/or improved interaction with other applications on the device (Gordon, ¶1731-33). Additionally, it would provide for hands-free or nearly-hands-free operation during a call, which would be useful for users who are multitasking and/or have physical/visual/hearing impairments.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Charignon in view of U.S. Patent Publication No. 2022/0139380 to Meng et al. (“Meng”).
As to claim 8, Charignon discloses the method of claim 7.
Charignon does not disclose: wherein the machine-learned model comprises an end-to-end Recurrent-Neural-Network-Transducer Automatic Speech Recognition Model.
However, Meng discloses: wherein the machine-learned model comprises an end-to-end Recurrent-Neural-Network-Transducer Automatic Speech Recognition Model (Fig. 8 and ¶0115-16).
Charignon and Meng are considered to be similar to the claimed invention because they are in one or more of the same fields of: interactive information services, e.g. directory enquiries, and arrangements therefore, e.g. interactive voice response (IVR) systems or voice portals; visual mode comprising meus; using speech recognition; computing arrangements based on biological models, e.g. recurrent networks; and/or providing customer assistance, e.g. assisting a customer via helpdesk and/or phone tree. As such, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Charignon to incorporate the teachings of Meng to include: wherein the machine-learned model comprises an end-to-end Recurrent-Neural-Network-Transducer Automatic Speech Recognition Model. Doing so would provide benefits such as lower word error rates, improved performance, and/or reduced complexity (Meng, ¶0115-18).
References Cited
Charignon, Laurent Gabriel Stephane (2017). Systems and methods for interacting with answering systems (US 2017/0339275 A1). Filed 2016-05-17.
Gordon, George A. et al. (2018). System, method, and computer program product for determining whether to prompt an action by a platform in connection with a mobile device (US 2018/0032997 A1). Filed 2013-10-09.
Meng, Zhong et al. (2022). Internal language model for e2e models (US 2022/0139380 A1). Filed 2021-01-21.
Yeracaris, Yoryos et al. (2023). Virtual assistant architecture for natural language understanding in a customer service system (US 11,606,463 B1). Filed 2020-03-31.
Other Pertinent References
The following prior art made of record and not relied upon is considered pertinent to applicant’s disclosure:
Adibi, Jafar et al. (2021). Methods and systems for socially aware virtual agents (US 2021/0133763 A1). Filed 2019-10-30.
Andrews, Peter J. et al. (2011). Voice response systems browsing (US 2011/0103559 A1). Filed 2009-11-02.
Atlas, Samuel R. (2016). Visual options for audio menu (US 9,270,811 B1). Filed 2014-09-23.
Basir, O. A. et al. (2013). Communication system with personal information management and remote vehicle monitoring and control features (US 8,577,543 B2). Filed 2010-05-28.
Berger, Shai et al. (2009). System and method for externally mapping an interactive voice response menu (US 2009/0207980 A1). Filed 2009-03-10.
Bladon, Anthony et al. (2016). Methods and systems for managing telecommunications and for translating voice messages to text messages (US 9,277,043 B1). Filed 2015-03-03.
Cochinwala, Munir et al. (2011). Automatic extraction of information from ongoing voice communication system and methods (US 2011/0228913 A1). Filed 2010-03-16.
Cutler, Ross G. et al. (2018). Artificially generated speech for a communication session (US 2018/0218727 A1). Filed 2017-02-02.
Dimaria, Matthew et al. (2023). System and methods for intent - based active callback management using enhanced callback objects (US 2023/0300249 A1). Filed 2023-03-24.
Fu, Li (2022). Speech recognition method and apparatus, and computer-readable storage medium (US 2022/0343898 A1). Filed 2020-04-30.
Furuta, Satoru (2020). Acoustic signal processing device, acoustic signal processing method, and hands-free communication device (US 2020/0045166 A1). Filed 2017-03-08.
Faaborg, Alexander et al. (2016). Visual indication of a recognized voice-initiated action (US 9,430,186 B2). Filed 2014-04-01.
Grigsby, Travis M. et al. (2010). Voice response unit mapping (US 2010/0057456 A1). Filed 2008-09-02.
Jia, Xin (2017). Call processing method and device (US 2017/0339263 A1). Filed 2015-01-26.
Karani, VIJAY (2019). Method and system for automatically transcribing a call and updating a record based on transcribed voice data (US 2019/0297186 A1). Filed 2018-03-22.
Klemm, Reinhard et al. (2020). System and method for efficient port and bandwidth utilization in setting up communication sessions (US 10,659,501 B2). Filed 2014-06-30.
Kulkarni, Alok et al. (2016). Interactive voice response system crawler (US 2016/0227034 A1). Filed 2016-04-05.
Lavian, Tal et al. (2014). Systems and methods for visual presentation and selection of ivr menu (US 8,687,777 B1). Filed 2011-10-18.
Leeds, Richard et al. (2014). Computer control of online social interactions based on conversation processing (US 2014/0177813 A1). Filed 2013-12-13.
Montero, A. et al. (2022). Presentation of communications (US 2022/0124193 A1). Filed 2020-10-19.
Pandian, Gnanaprakasam et al. (2021). Presenting, at a graphical user interface (gui), a constellation view of communications associated with node groups in a network (US 10,928,987 B2). Filed 2020-04-13.
Peterson, Patrick et al. (2022). Presenting multiple customer contact channels in a browseable interface (US 11,418,647 B1). Filed 2020-10-16.
Skinner, Cory Robert (2011). Method and apparatus for providing ivr system requested data (US 2011/0188644 A1). Filed 2010-01-29.
Thomson, David (2014). Apparatus and method for providing a virtual assistant (US 2014/0164476 A1). Filed 2012-12-06.
Thomson, David (2021). Hearing accommodation (US 2021/0289070 A1). Filed 2020-03-10.
Way, Brendan et al. (2021). Performing a custom action during call screening based on a purpose of a voice call (US 10,958,784 B1). Filed 2020-03-11.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAMUEL H LEONARD whose telephone number is (571)272-5720. The examiner can normally be reached Monday – Friday, 7am – 4pm (PT).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Yuwen (Kevin) Pan can be reached at (571)272-7855. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAMUEL H. LEONARD/Examiner, Art Unit 2649 /YUWEN PAN/Supervisory Patent Examiner, Art Unit 2649