DETAILED ACTION
This office action is in response to Applicant’s Request for Continued Examination (RCE), received on 03/04/2026. Claims 1, 17, 21, 27-28, and 38 have been amended. Claims 1, 3-28, 30-33, 35, 37-39 are pending and have been considered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/04/2026 has been entered.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119
(a)-(d). The certified copy has been filed for the parent Application No. CN-202310566940.6, filed on 05/18/2023.
Response to Arguments
Applicant's arguments filed 03/04/2026, see pgs. 19-24, with respect to “Claim Rejections under 35 U.S.C. 103” have been fully considered but they are not persuasive.
Applicant’s representative asserts, “Regarding amended claim 1, Applicant respectfully submits that amended claim 1 recites limitations that have not been disclosed, taught, or even suggested by Gross, Kang, and Gross2. For example, amended claim 1 recites in part:
‘...the smart glasses further comprise a data acquisition device and a wireless communication component electrically connected to the processor; wherein the data acquisition device comprises at least one component of: a position sensor, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera and a pedometer; and wherein the one or more computer programs further comprise... instructions for:
obtaining, through the data acquisition device, device data of the smart glasses, and converting, through the speech-to-text engine, the first speech into the first text;
sending, through the wireless communication component, the first text to the model server, and receiving description information of required data from the model server, wherein the model server, through the GAILLMs, according to the first text, determines whether the reply needs to be based on data collected by the data acquisition device on the smart glasses and/or a data acquisition device on a smart mobile terminal, wherein when the reply corresponding to the question needs to be based on the data, the model server generates the description information of the required data and selects to send the description information to the smart glasses or the smart mobile terminal, and wherein, when the reply corresponding to the question does not need to be based on the data, the model server obtains a second text comprising the reply through the GAILLMs based on the first text;
determining the required data from the device data of the smart glasses according to the description information, and sending, through the wireless communication component, the required data to the model server, wherein the model server inputs the required data to the GAILLMs to obtain the second text after receiving the required data, and wherein the GAILLMs associate the required data with the first text, and obtain the second text using the required data and the first text...’ (emphasis added).
That is, in amended claim 1, the GAILLMs on the model server are used to determine whether the reply needs to be based on data collected by the data acquisition device on the smart glasses and/or a data acquisition device on a smart mobile terminal according to the first text sent by the smart glasses.
Further, when the reply corresponding to the question needs to be based on the data, the model server of amended claim 1 is further used to generate the description information of the required data and select to send the description information to the smart glasses or the smart mobile terminal, and the smart glasses of amended claim 1 are used to, after receiving the description information, determine the required data from the device data of the smart glasses, which is obtained by the data acquisition device (including at least one component of: a position sensor, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera and a pedometer) on the smart glasses.
Furthermore, when the reply corresponding to the question needs to be based on the data, the second text including the reply in amended claim 1 is obtained by the GAILLMs using the first text and the required data determined by the smart glasses from the device data of the smart glasses.
Regarding Gross and Kang
On page 39 of the Final Office Action, Examiner states ‘Gross in view of Kang does not disclose: ...receiving description information of required data from the model server, wherein the model server, through the GAILLMs, according to the first text, determines whether the reply needs to be based on data collected by the data acquisition device on the smart glasses, and generates the description information of the required data when the reply corresponding to the question needs to be based on the data; determining the required data from the device data of the smart wearable device according to the description information, and sending, through the built-in wireless communication component, the required data to the model server, wherein the model server obtains a second text comprising the reply through the GAILLMs based on the required data and the first text...’ Therefore, Gross and Kang fail to disclose the aforementioned limitations of amended claim 1.
Regarding Gross2
The para. [0007] of Gross2 discloses ‘[t]he at least one smart device is configured to record a first audio data or signal associated with at least one voice query, and
provide the first audio data or signal associated with the at least one voice query to the chatbot application...’ (emphasis added). The para. [0032] of Gross2 discloses ‘Glasses 310 may include a microphone and/or a speaker for receiving, recording, emitting, and /or playing sound’ (emphasis added). The paras. [0042] to [0054] of Gross2 disclose ‘[i]f system 300 is configured to enable image queries at steps 410 and 415, process 400 may proceed to step 425... user 305 (e.g., and/or users 305a, 305b, and/or 305c) may provide multiple voice queries to system 300 via glasses 310... The first query may be similar to as described above regarding step 420 and may be a query for an audio and/or text response from system 300 (e.g., may not include a query for an image to be provided). The user may also provide a second query that may be a query for one or more images... At step 445, audio assistant 330 may receive voice data and/or signals and convert the voice data and/or signals to text data and/or signals at step 450 as shown at 340 and 345 of FIG. 4... At step 455 and as shown at 350 of FIG. 4, chatbot platform 335 may receive the text data and/or signals. At step 460, chatbot platform 335 may select one or more chatbots for responding to the received text data and/or signals of the one or more user queries... If image data is queried, any suitable image-based artificial intelligence such as, for example, DALL-E and/or any other suitable source may be used... At step 475, system 300 may finalize text and/or image response data and/or signals to be provided to a user in response to one or more user inquiries received by system 300 at step 420 or 425... At step 480, system 300 may determine whether or not image data... is to be provided. If such response data is to be provided..., process 400 may proceed to step 485. If such response data is not to be provided, process 400 may proceed to step 490... In at least some exemplary embodiments at step 485, an email and/or SMS number that may be provided at step 410 may be utilized. Textual response data may be automatically exported to a specified email or as a text message to an indicated phone (e.g., including any images added by an image generation feature for example as described herein)... System 300 may operate to generate graphs, charts, detail images, and/or any other desired image data based on a user's voice queries and commands (e.g., and or artificial intelligence operations), which may add visual information to textual responses provided by chatbot platform 335... If a user describes one or more images (e.g., as described at step 425), audio assistant 330 may transcribe and communicate this vocal image query (e.g., the exemplary disclosed second query) separately from the linguistic query (e.g., the exemplary disclosed first query) to the exemplary disclosed one or more modules of system 300 for processing (which may in turn submit the transcribed text of the vocal image query to an AI image generator)... Following step 485, process 400 may proceed to step 490. At step 490 and as shown at 355 of FIG. 4, audio assistant 330 may receive the exemplary disclosed finalized text data and/or signals from chatbot platform 335...’ (emphasis added).
That is, the audio data or signal of Gross2 associated with at least one voice query is the audio data or the audio signal obtained through the microphone of the Glasses 310. The audio assistant 330 of Gross2 converts the audio or sound data and/or the audio or sound signals to text data and/or text signals. When image queries are enabled, the user 305 of Gross2 may provide multiple voice queries to system 300 via glasses 310, the chatbot platform 335 of Gross2 generates the text reply only based on the text data and/or text signals converted from the voice queries of the user 305, and the image data is generated by an independent AI image generator based on the transcribed text of the user's vocal image query submitted by the system module. The chatbot platform 335 of Gross2 may select one or more chatbots (rather than select the Glasses 310 or the user device 320 (e.g., the phone)) for responding to the received text data and/or signals of the one or more user queries (rather than obtaining device data from the Glasses 310 or the user device 320). Additionally, in Gross, the user provides a second voice query for one or more images (e.g., as described at step 425), and the second voice query may describe one or more images (rather than that the chatbot platform 335 generates a description of the images that the Glasses 310 need to provide). Furthermore, ‘Receive Text and/or Image Data/Signals 490’ in FIG. 11 of Gross2 means that the audio assistant 330 receives the finalized ‘Text and/or Image Data/Signals’ from chatbot platform 335 (rather than that the chatbot platform 335 receives the ‘Text and/or Image Data/Signals’ from audio assistant 330) (see para. [0049] to para. [0054] of Gross2).
Accordingly, the chatbot platform 335 of Gross2 does not determine whether the reply needs to be based on data collected by the data acquisition device on the smart glasses and/or a data acquisition device on a smart mobile terminal; and the chatbot platform 335 of Gross2 neither generates the description information of the required data nor selects to send the description information to the smart glasses or the smart mobile terminal. Additionally, the Glasses 310 of Gross2 does not determine the required data from its own device data according to the description information, as the Glasses 310 only obtains audio data or audio signals through its built-in microphone. The reply of Gross2 is generated without using the required data derived from the smart glasses' device data (collected through at least one of a position sensor, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera, and a pedometer), as such device data and data acquisition components are not present in Gross2's Glasses 310.
Therefore, Gross2 also fails to disclose the aforementioned limitations of amended claim 1.”
In response, the examiner would like to refer to the broadest reasonable interpretation (BRI) of the claims as currently constructed in view of the cited combination of art. Specifically, the examiner respectfully disagrees with Applicant’s assertions that “the GAILLMs on the model server are used to determine whether the reply needs to be based on data collected by the data acquisition device on the smart glasses and/or a data acquisition device on a smart mobile terminal” is something which is not taught by the combination of cited art. A claim element similar to this was previously disclosed to be a part of dependent claim 17. Gross in view of Gross2 was relied upon to reject this claim. The devices and functionalities of Gross will be considered in view of Gross2. Gross discloses a process for receiving a speech request via smart glasses ([Fig. 7, S710]), wherein that request or command is then communicated to a mobile device ([Fig. 7, S715]), further wherein “[0106] At step 720, the mobile application controlling the paired or connected devices the devices used for particular tasks or applications will access lookup tables and present or display information related to the requested task or application using the paired or connected devices (smartglasses, smartphone, or smartwatch) preset to view or hear that information”. Presentation of information gathered from lookup tables, wherein the lookup tables are accessed via mobile device via a request sent from smart glasses, indicates a situation in which there is a determination that the reply needs to be based on data collected by a data acquisition device on a smart mobile terminal which is then selected to be sent to the glasses for presentation. The “and/or” included in the amendments to the claims does not require data acquisition to be performed by both the smart glasses and smart mobile terminal. Only one of these two optional steps are required to satisfy the claim mapping. Taking the plurality of devices of Gross in view of the plurality of chatbots of Gross2, the examiner respectfully asserts that each device of Gross (having its own mobile application) would have its own chatbot as disclosed in Gross2 as a chatbot is a functionality reasonably understood to be contained in mobile applications in the context of chatting (see [0210]-[0235] of Gross). A chatbot is not a physical component. Regardless, the examiner has updated their rejection of the claims focused around an interpretation of Gross with less reliance on Gross2.
Regarding Applicant’s arguments against the obtaining of the device data and/or data acquisition device, the examiner would like to refer to the functionality of the cameras on Gross in view of the definition of the device data as currently claimed. Device data is currently claimed to be nothing more than something obtained from the smart glasses. [0116] of Gross discloses “smartglasses 200 or smartwatch 300 contains an embedded camera capable of transmitting visual information to a connected smartphone 400 AI interface 460, then the smartglasses mobile application 210 could assist the user in rapidly reading a QR code containing a cryptocurrency public address (or product information) via one of their wearable devices”. The image being processed on the smartglasses (with associated data being sent to the smartphone) necessarily requires the image data to be obtained by the smartglasses, at that point becoming device data, to be sent to the smartphone.
Regarding Applicant’s arguments against the generation of description information of the required data and sending the required data, there is no definition as to what the required data and/or description information must be. In the context of an image (QR code as disclosed in Gross), the examiner asserts that all of the code is required data; therefore, an entire image is required data when the image is of a QR code. The pixels comprising that image is description information. With this interpretation in mind, the examiner would like to refer to the previously disclosed operational system of Gross which disclosed sending information gathered from a smart glasses to a smart mobile device, i.e. smartphone, wherein the information sent can be an image comprised of required data and description information.
With regard to Applicant’s arguments against determining the required data from its own device, the examiner respectfully asserts that the voice and/or image information gathered by the glasses of Gross is required data gathered from the glasses. The examiner agrees with Applicant’s assertions that Gross2 does not teach this elements, but Gross is being depended upon to incorporate the required inputs.
With regard to Applicant’s arguments against Gross2 generating the reply without using the required data derived from the smart glasses device data, the examiner agrees with Applicant’s assertions, but respectfully asserts that Gross discloses obtaining a second text based on required data and first text. Specifically, Fig. 7, S740 of Gross discloses smart glasses which communicate an override to a device mobile application controlling paired devices for presentation, wherein presentation is necessarily of chat responses as discussed. Sending an override request, i.e. selecting a different device for presentation, necessarily requires device data (which is required for presentation) being sent from the smart glasses to the smart mobile application in order to appropriately display the output generated via the voice assistants of Gross. See updated rejections below.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: "speech-to-text engine" and "text-to-speech engine" used for “converting” in claims 11, 14, 19, 22, 23, 34, 36.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-5, 7, 11-15, 17, 28, 30-31, 35, 38 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gross (US-20210304743-A1), in view of Kang et al. (US-20200020334-A1), hereinafter Kang, further in view of Gross et al. (US-20240321279-A1), hereinafter Gross2.
Regarding claim 1, Gross discloses: smart glasses ([0042] FIG. 2 illustrates smartglasses) based on generative artificial intelligence large language models (GAILLMs) ([0042] smartglasses that can be used in accordance with the methods and systems for using artificial intelligence to control mobile devices used for presentation and display of mobile device tasks [In view of the context of Gross being for voice conversation/response generation using smart glasses (see abstract), there is an indication that the artificial intelligence is based on generating dialog responses, i.e. presentation of language, indicating the artificial intelligence is a generative artificial intelligence large language model, as would be required for response generation based on input speech]), comprising:
a front frame ([Fig. 2, Bridge 210, Rims 220], [The combination of which forms a front frame as disclosed in the figure]), a temple ([Fig. 2, temples 255]), a microphone ([Fig. 2, microphone 225]), a speaker ([Fig. 2, bone conduction speakers 235]), a processor and a memory ([Fig. 2, hardware 260], [In view of Gross disclosing that the user can active the voice assistant using Bluetooth glasses for message recording, see [0202], indicating processing of the input, i.e. to determine the speech, and storage, i.e. recording, using the hardware disclosed in Fig. 2]);
wherein the temple is coupled to the front frame ([Fig. 2], [0060] The smartglasses 200 can have a bridge 210 with end pieces 205 connected to temples 255 and corrective lenses 215 in rims 220), and the processor is electrically connected to the microphone, the speaker and the memory ([Fig. 2, printed circuit board 245…connectivity module and hardware 260], [Inclusion of a printed circuit board and connectivity module with hardware indicates the hardware, i.e. microphone 225, speaker 235, and the memory associated with the voice recording (see above claim element for rationale) are all connected]);
one or more computer programs executable on the processor are stored in the memory ([0258] software or code can be embodied in any non-transitory computer-readable medium for use by or in connection with an instruction execution system such as, for example, a processor in a computer system) and the one or more computer programs comprise instructions for:
activating a chat function of the smart glasses in response to a first control instruction for activating the chat function ([0104] the microphone 225 of smartglasses 200 and smartglasses AI interface 280 receive an oral request or command audibly proceeded by language, such as “tell me” or “show me,” that is communicated to glasses' AI interface 280… [0197] an embodiment Vyrb implementation may permit a user to create a verbal “hotkey” configured to trigger on-demand activation of a customizable range of data retrieval and user action patterns [“tell me” or “show me” represent first control instructions, i.e. verbal hotkeys, to activate a chat function, i.e. for data retrieval operations, in view of [0037] keyword activation of the instant application.]);
obtaining, through the microphone, a first speech of a user ([Fig. 7, 710]), wherein the first speech comprises a question asked by the user ([0104] Examples of questions can include questions designed to perform tasks, open applications, including map or navigation, music, wallet, application store, calculator, watch, stopwatch, banking or money transfer, video, games, stocks, weather, etc., [Taking the received oral request by the smartglasses of step 710 in view of the question asking of [0104] indicates the oral requests are first speeches comprising questions]); and,
obtaining, through the GAILLMs, a second speech comprising a reply corresponding to the question ([0115] The user could then initiate an on-blockchain token transfer, check a token balance, and more via their smartglasses 200 alone. The smartglasses could communicate confirmation prompts, balance inquiries etc., back to the user. [Communicating information back to a user based on input indicates that communicated information is a response]), and playing, through the speaker ([Fig. 2, speaker 235]), the second speech ([Fig. 8, 895], [0139] At 1220, based on predefined user settings for different types of content and suitable response actions thereof, a voice prompt will trigger from the voice assistant after the content finishes playback, and can ask the user to respond “like,” or take another action on the content via the user's voice [Triggering a voice prompt for the end of content, in view of the questions/responses of [0115], indicates the content, i.e. second speech, is a response]);
wherein the smart glasses further comprise a Bluetooth component electrically connected to the processor ([Fig. 2], [0061] connectivity module and hardware 260 may use WiFi, Bluetooth [In view of the previously disclosed processor of Gross, indicating the hardware consists of at least a processor]), and the one or more computer programs further comprise instructions for:
receiving, through the Bluetooth component, the first control instruction from a smart mobile terminal ([0202] The user can activate their voice assistant via the phone or watch, or connected Bluetooth glasses…The user can also dictate a vocal post to their profiles on other platforms and direct messages to users on other platforms, without posting to the Vyrb app. For example, “Hey Siri, use Vyrb to send a message to John Q on Facebook.” [The first control instruction represents “Hey Siri…”, in view of [0037] of the instant application, the smart phone and/or watch represent a smart mobile terminal in view of Fig. 3 of the instant application]), wherein the first control instruction is generated by a virtual assistant program of the mobile smart terminal after a voice wake-up instruction is obtained by the virtual assistant of the mobile smart terminal ([0104] the microphone 225 of smartglasses 200 and smartglasses AI interface 280 receive an oral request or command audibly proceeded by language, such as “tell me” or “show me,” that is communicated to glasses' AI interface 280… [0123] the device level voice assistant manager process may be programmed to activate the device level voice assistant to perform an operation requested by the user 805, based on configuring the device level voice assistant with voice commands determined by the device level voice assistant manager process executing on the user's web-enabled device, in response to the operation 815 requested by the user 805 [Using a voice assistant manager to perform operations requested by the user and activate a device level voice assistant manager, i.e. virtual assistant, in view of the previously disclosed wake-up word(s) “Hey Siri”, “tell me”, “show me”, etc., indicates the control instruction, i.e. operation request, is generated by the voice assistant manager (on a device, i.e. smart terminal, level), i.e. comprising virtual assistant operations, based on text following a wake-up instruction. Wherein Siri is a known virtual assistant on mobile devices, i.e. mobile smart terminals. The ability to perform/respond to tasks indicates a generation of control instructions]); or,
the first control instruction is generated after an operation for wake-up is detected by a user interface of the mobile smart terminal ([0059] The device level voice assistant manager process may be programmed to activate a device level voice assistant to perform an operation requested by a user, based on configuring the device level voice assistant with voice commands determined by the device level voice assistant manager process executing on the user's mobile device [Performing operations based on received voice input indicates generation of control instructions after a wake-up is detected, in view of the “Hey Siri” wake-up of [0202], a well-known wake word to anyone familiar with Apple products. A device level voice assistant indicates the voice assistant is on a mobile smart terminal, i.e. device, which can include an interface in the form of a cell phone, see smart phone of Gross (Fig. 3)]); and,
waking up the microphone while activating the chat function ([0197] an embodiment Vyrb implementation may permit a user to create a verbal “hotkey” configured to trigger on-demand activation of a customizable range of data retrieval and user action patterns [“tell me” or “show me” represent first control instructions, i.e. verbal hotkeys, to activate a chat function, i.e. waking a microphone for data, i.e. input speech, retrieval operations, in view of [0037] keyword activation of the instant application. Further in view of “Hey Siri” ([0202]) which is a clear wake term for activating a microphone for voice analysis]);
receiving, through the bluetooth component, a second control instruction for deactivating the chat function from the smart mobile terminal ([Figs. 10A-C], [0133] The Vyrb App begins recording user actions on the device, including taps and keystrokes. When the desired action(s) have been performed, the user can hit an overlaid check button to complete the recording [Hitting a button to stop recording, i.e. chat function, on a smartphone, i.e. smart mobile terminal, represents a second control instruction, wherein the smartglasses of Gross contain a bluetooth component (see [0061]), and Gross discloses smartglasses capable of recording (see [0104]), indicating a system to receive through bluetooth a control instruction for deactivation of a chat function on smartglasses from a smart mobile terminal]).
Gross does not disclose:
deactivating the chat function and the microphone in response to the second control instruction.
Kang discloses:
deactivating the chat function and the microphone in response to the second control instruction ([0150] if the hardware button is pressed two times more, the electronic device 101 may deactivate the microphone. Or, in a case where the user utterance received in operation 870 indicates continuous command mode termination, the electronic device 101 (e.g., the processor 120 or 210) may terminate the continuous command mode [Termination of continuous command mode, wherein command mode tracks to processing of user utterances (see Fig. 8), indicates termination of a chat function in view of the chat function of Gross. Further in view of Fig. 34 indicating the second control instruction for deactivating a chat function is clearly originating from a smart mobile terminal, i.e. smartphone]).
Gross and Kang are considered analogous art within speech command processing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross to incorporate the teachings of Kang, because of the novel way to select any one of a number of various user utterance processing modes which consider device state based on received input for improved conversation services (Kang, [0005]).
Gross further discloses:
wherein the GAILLMs are configured on a model server ([0077] AI software such as Siri, Bixby or Alexa, [Wherein the three previous software assistants are all based on GAILLM architecture as required for question responses. The model server is the mobile device containing mobile application which contains a voice assistant, i.e. GAILLMs ([0059])]), the smart glasses further comprise a data acquisition device ([Fig. 2, Camera 265], [In view of [0166] of the instant app which defined a data acquisition device to be a camera]) and a wireless communication component electrically connected to the process ([Fig. 2], [0061] connectivity module and hardware 260 may use WiFi, Bluetooth [In view of the previously disclosed processor of Gross, indicating the hardware consists of at least a processor]);
wherein the data acquisition device comprises at least one component of: a position sensor, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera ([0116] smartglasses 200 or smartwatch 300 contains an embedded camera capable of transmitting visual information to a connected smartphone 400 AI interface 460) and a pedometer, and the least one component is electrically connected to the processor ([In view of the previously disclosed processor on the smart glasses of Gross, indicating a camera on the same glasses will also be electrically connected to the same processor. The examiner would like to note that due to the disjunctive structure of the claim, not all sensors require a mapping]);
wherein the one or more computer programs further comprise:
a speech-to-text engine ([0077] Speech to text utility of AI software such as Siri, Bixby or Alexa can be used);
a text-to-speech engine ([0129] the device's text to speech engine); and
instructions for:
obtaining, through the data acquisition device, device data of the smart glasses ([0116] smartglasses 200 or smartwatch 300 contains an embedded camera capable of transmitting visual information to a connected smartphone 400 AI interface 460, then the smartglasses mobile application 210 could assist the user in rapidly reading a QR code containing a cryptocurrency public address, [As the data acquisition device is a sensor, it is reasonable to assume that device data gathered by the data acquisition device is sensor data. Therefore, images gathered by the smart glasses track to device data]), and
converting, through the speech-to-text engine, the first speech into the first text ([0237] 1. Speech to text functionality);
sending, through the wireless communication component ([In view of the previously disclosed wireless communication component]), the first text to the model server ([Fig. 7, S710 “Smartglasses receive an oral request or command”, S715 “Request or command…to perform a particular…task is communicated to the mobile application”], [In view of the previously disclosed speech-to-text of Gross]);
receiving description information of required data from the model server ([Fig. 7, S720 “Access lookup tables and present any information related to the requested device…for display and presentation”], [If the task/command is presentation, then the requested device information is describing how the required data should be presented. The mobile device is the model server]),
wherein the model server, through the GAILLMs, according to the first text ([In view of the previously disclosed sending of first text to mobile device]), determines whether the reply needs to be based on data collected by the data acquisition device on the smart glasses and/or a data acquisition device on a smart mobile terminal ([Fig. 7, S730 “Smartglasses receive optional override or switch command or request to change the paired or connected device used for presentation or display”], [Choosing a presentation device based on the type of task at a mobile device indicates a determination that, based on the first text, the connected device is chosen, wherein each device will necessarily have data to be collected (as disclosed in S720)]),
wherein when the reply corresponding to the question needs to be based on the data, the model server generates the description information of the required data ([Considering the previously cited S720 of Gross, accessing information related to a requested device by a mobile application, i.e. model server, wherein the request comes from smart glasses, indicates a determination that the accessed description information is relevant to presenting the requested data, i.e. reply, in order to appropriately present the requested information in the form of a reply]); and
selects to send the description information to the smart glasses or the smart mobile terminal ([As previously disclosed in S730, the smartglasses receiving the request to change presentation device will necessarily be comprised of description information to be read by the smart glasses which describe the override action in a way that the glasses can understand]), and
wherein when the reply corresponding to the question does not need to be based on the data, the model server obtains a second text comprising the reply through the GAILLMS based on the first text ([As previously disclosed in [0115], the ability of the smartglasses to return answers to a user based on an input indicates a reply generated through GAILLMs (as previously disclosed) without the need for additional information about where the information is to be presented. Consider a situation in which the optional override is not received by the smart glasses]);
determining the required data from the device data of the smart glasses according to the description information ([There is no limitation requiring the required data to be less than the complete device data; therefore, as Gross previously discloses device data and description information, it is reasonable to assert that Gross also discloses the required data as claimed, i.e. in QR image collection (a described task via description information of an input request), the image gathered by the device, e.g. device data, is all relevant, and therefore, required]), and
sending, through the wireless communication component, the required data to the model server ([As previously disclosed in S715, smart glasses send request/task data, i.e. the required data, to the mobile device, i.e. model server]),
wherein the model server inputs the required data to the GAILLMs to obtain the second text after receiving the required data ([0140] The user requests their five newest inbound Vyrb posts through the device-level voice assistant. After each post is read out, the voice assistant prompts the user if they would like to leave a voice comment. If the user says yes, they record a comment, [A voice assistant is reasonably understood to be any of the AI software (GAILLMs) previously cited. Further, wherein the voice request from the user can be gathered from the smart glasses and sent to the smart mobile device as disclosed in Fig. 7, indicating a model server inputting the required data into the GAILLMs, i.e. inputting the post data into the voice assistant, wherein the voice assistant asking whether or not the user would like to leave a voice comment is second text vocalized by the voice assistant based on the required data, i.e. the posts]), and
wherein the GAILLMS associate the required data with the first text ([0059] mobile application 401 may include a device level voice assistant manager process, [Wherein the required data is the device data as previously disclosed. Further, as previously disclosed, sending a request or command to a mobile application, wherein that mobile application is used for controlling presentation of tasks (S715) indicates a situation in which the GAILLMs, i.e. the voice assistant on mobile application, associate the device data with the first text through the sending operation. As the command is sent to the mobile application to be answered/presented (S715), device data is gathered (S720) in relation to the request, indicating a situation in which the mobile application (containing the voice assistant) associates the device, i.e. required, data with the received first text]), and
obtain the second text using the required data and the first text ([Fig. 7, S740 “smartglasses communicate the override…controlling the paired or connected devices used for display or presentation of tasks or applications”[As previously disclosed in [0140] of Gross, vocalizing a question/comment asked by the voice assistant indicates a received text, i.e. prompt, to be vocalized, wherein the vocalization is performed based on the device connected to display and/or presentation of tasks (S740 of Fig. 7) indicating the second text to be received by the user using the required device data and first text]).
Gross in view of Kang is not relied upon to disclose:
receiving, through the wireless communication component, the second text comprising the reply from the model server, and
converting, through the text-to-speech engine, the second text comprising the reply into the second speech.
Gross2 is relied upon to disclose:
receiving, through the wireless communication component, the second text comprising the reply from the model server ([Fig. 11, “Receive text and/or image data/signals 490”], [0054] At step 490 and as shown at 355 of FIG. 4, audio assistant 330 may receive the exemplary disclosed finalized text data and/or signals from chatbot platform 335), and
converting, through the text-to-speech engine, the second text comprising the reply into the second speech ([Fig. 11, “Convert Text to Voice Data/Signals”], [0054] At step 495, audio assistant 330 may convert the finalized text data and/or signals to voice data and/or signals and transfer the voice data and/or signals to the exemplary disclosed device (e.g., glasses 310).
Gross, Kang, and Gross2 are considered analogous art within smart device chatbot communication. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang to incorporate the teachings of Gross2, because of the novel way to introduce an intermediate audio assistant before sending received audio to a chatbot, allowing for a voice interface for more powerful text-based chatbots traditionally lacking voice interfaces (Gross2, [0005]).
Regarding claim 3, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 1.
Gross further discloses: wherein the one or more computer programs further comprise a virtual assistant program ([0134] For example, a voice command created in this way might look like: “Siri, Vyrb CNN”. [Siri is a well-known virtual assistant program]), and the first control instruction is a first voice instruction ([Fig. 7, 710], [Receiving an oral request is equivalent to a first voice instruction]); and,
wherein the one or more computer programs further comprise:
instructions for obtaining, through the virtual assistant program of the smart glasses, the first voice instruction ([Fig. 15, 1520], [0146] Upon determining a spoken user voice command is detected, at 1515 the process creates a custom voice command based on combining command parameters determined from user voice input captured with the detected user voice command. At 1520, the process activates the voice assistant with the custom voice command to perform the user requested operation [Activating a voice assistant, i.e. virtual assistant, with a custom voice command for performing operations indicates the voice command is an obtained first instruction]), wherein the first voice instruction comprises a preset first keyword for activating the chat function ([0204] For example, “Hey Siri, use Vyrb to send a message to John Q on Facebook.” The app would then prompt the user to record the message, which would be sent as a voice message (if permitted by the other platform) or transcribed into text and sent [“Hey Siri” is a well-known voice instruction for activating a chat function, using “Hey Siri” and then following with a command indicates “Hey Siri” as a keyword for activation]).
Kang further discloses:
instructions for obtaining, through the virtual assistant program of the smart glasses ([0231] a voice command, such as “keyboard” or “Bixby, keyboard,” [Bixby is a well-known virtual assistant program]), a second voice instruction comprising a preset second keyword for deactivating the chat function ([Fig. 8, 880], [Checking for a termination event, wherein the inputs to be checked are user utterances 830/870, indicates the termination event is comprised of one of more keywords from within the user utterance to know to deactivate the continuous command mode, i.e. chat function]); and,
deactivating the chat function in response to the second voice instruction ([0217] the electronic device 101 may obtain a user utterance 3160 with the hardware button 3113 pressed. The electronic device 101 may send data about the obtained user utterance to the external server and receive and display a text 3161 corresponding thereto. In a case where a text designated to terminate the dialog mode, such as “Chat mode Off,” is received, the electronic device 101 may terminate the dialog mode corresponding to releasing the press of the hardware button 3113 [Wherein “chat mode off” is a second voice instruction used for deactivation the chat function, i.e. chat mode, based on a spoken utterance in view of the speech-to-text conversions of Gross])
Regarding claim 4, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 1.
Kang further discloses:
wherein the smart glasses further comprise a button electrically connected to the processor ([Fig. 2, Input Device 250 connected to Processor 210], [0074] The key 256 may include e.g., a hardware button [Key tracks to a predetermined input, i.e. type of button press/keyword/etc. Further, Kang defines smartglasses as capable of being the electronic device containing the previously disclosed elements, see [0062]]);
the button comprises a physical button and/or a touch sensor based virtual button ([0078] A hardware button (e.g., the key 256) [Hardware button indicates a physical button]); and,
the first control instruction is triggered based on a first preset operation on the button performed by the user ([0159] For example, if a hardware button is pressed or pressed-and-held while the virtual keyboard 1021 is displayed, the electronic device 101 may obtain a user utterance through the microphone and receive a text corresponding thereto from the external server [Hardware button being pressed represents a first preset operation on the button, obtaining an utterance represents the first control instruction based on the button being pressed]).
Regarding claim 5, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 4.
Gross further discloses:
wherein the one or more computer programs further comprise:
instructions for obtaining, through the microphone, the first speech after a user voice comprising a third preset keyword is obtained through the microphone ([0134] For example, a voice command created in this way might look like: “Siri, Vyrb CNN” … [0104] At step 710, the microphone 225 of smartglasses 200 and smartglasses AI interface 280 receive an oral request or command audibly proceeded by language, such as “tell me” or “show me,” that is communicated to glasses' AI interface 280. Examples of questions can include questions designed to perform tasks [In view of the “Siri” first keyword, the “tell me” or “show me” can represent third keywords]), wherein the third preset keyword is configured to indicate that the user is beginning to ask the question ([In view of the above excerpt from [0104], “tell me” and “show me” are clear indications of beginnings of questions to be answered, i.e. “Hey Siri, tell me what the weather is like in Las Vegas today?”, after a first keyword “Siri” or “Hey Siri” which would activate the chat function]).
Kang further discloses:
instructions for obtaining, through the microphone, the first speech in response to a second preset operation on the button performed by the user ([0086] receiving a second-type user input through the button [Compared to the regular press or press and hold of [0159] of Kang]), wherein the second preset operation comprises:
any one of long pressing the virtual button, short pressing the virtual button, touching the virtual button, tapping the virtual button, sliding on the virtual button installed on the temple, and pressing and holding the physical buttons ([0088] the user inputs may be selected from one of a single press on the button, a double press on the button, a triple press on the button, a single press-and-hold on the button, or a double press-and-hold on the button [The examiner would like to note that due to the disjunctive nature of the claim, not all elements require a mapping; therefore, the “pressing and holding” of a physical button has been mapped]), and wherein a duration of the short pressing is shorter than a duration of the long pressing ([Defining single press and single press-and-hold for the same button indicates that the press-and-hold is of a longer duration than just press, otherwise there would be no need to define both operations]).
Regarding claim 7, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 5.
Gross further discloses:
wherein the one or more computer programs further comprise:
instructions for, before obtaining, through the microphone, the first speech, waking up the microphone and outputting, through the speaker, a first prompt sound to prompt the user to start asking the question ([0139] a voice prompt will trigger from the voice assistant after the content finishes playback, and can ask the user to respond… [0204] The app would then prompt the user to record the message The app would then prompt the user to record the message [In view of the previously disclosed questions of Gross ([0104]), indicating that the voice prompt trigger used for a response could also be used as the prompt for an initial recording, wherein the initial recording represents a response to “Would you like to begin speaking?”. Recording a message based on a prompt indicates waking up a microphone for recording based on the prompt]).
Regarding claim 11, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 1.
Gross2 further discloses:
wherein the one or more computer programs further comprise instructions for:
sending, through the wireless communication component, the first speech to a conversion server, so as to convert the first speech into a first text and send the first text to the model server through the conversion server ([Fig. 4, 345>330>350], [0028] user 305 may provide a voice (e.g., speech) query at 340. Audio or sound data and/or signals (e.g., of the speech or voice query provided at 340) may be transferred via the exemplary disclosed communication techniques from the exemplary disclosed device (e.g., glasses 310, headphones 315, and/or user device 320) to audio assistant 330 as shown at 345. Audio assistant 330 may convert the audio or sound data and/or signals to text data and/or signals [An audio assistant tracks to a conversion server. Sending text information from the audio assistant to a chatbot platform 335 indicates the chatbot platform represents a model server, i.e. that configured using GAILLMs, in view of the plurality of GAILLM trained chatbots disclosed in [0034]]);
wherein the model server obtains a second text through the GAILLMs based on the first text ([Fig. 4, 335], [0047] Predetermined algorithms, machine learning operations, user input and/or criteria (e.g., user input regarding desired length of responses and desired chatbots), and/or any other suitable criteria may be used for selecting, prioritizing, analyzing, integrating, and/or utilizing a plurality of chatbots to provide a response to each user inquiry [Using chatbots, i.e. model servers, to obtain responses, wherein those chatbots consist of GAILLMs, indicates the chatbot obtains a response, i.e. second text, before sending back as a response based on an input audio/text 345]);
sends the second text back to the conversion server to convert the second text to the second speech through the conversion server ([Fig. 4, 355], [0056] The exemplary disclosed application may respond with a text-to-speech vocalization of a textual response of chatbot platform 335); and,
receiving, through the wireless communication component, the second speech from the conversion server ([Fig. 4, 360], [0032] For example, glasses 310 may include a microphone and/or a speaker for receiving a speech or voice query at 340 and emitting a sound (e.g., voice or speech) response at 365).
Regarding claim 12, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 1.
Kang further discloses:
wherein the one or more computer programs further comprise instructions for:
generating chat logs ([0145] the DM 757 may manage the user's chat history… [0258] the memory 4440 may include a log database capable of storing log information [Disclosing chat history and logs indicates that the chat history is stored in the form of a log, indicating generation]), and sending, through the wireless communication component ([In view of the wireless communication component of Gross]), the chat logs and the first text to the model server to generate the second text through the GAILLMs based on the chat logs and the first text ([0145] The DM 757 may manage the dialog history (dialog flow) and manage the dialog status of the current chat window. The DM 757 may determine whether to utilize information about the previous dialog or to make a new inquiry depending on the dialog status [Determining to utilize information from previous dialog indicates using a chat log and/or first text (which would be a previous dialog statement at the current time of operation) for generating a second text, i.e. a new inquiry or one based on the history. In view of the GAILLM system of Gross/Gross2 which could be used as the dialog manager (DM) of Kang without a change in functionality]).
Regarding claim 13, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 1.
Gross further discloses:
wherein the one or more computer programs further comprise instructions for:
obtaining sensing data of the data acquisition device ([0116] For example, scanning a QR code with the smartwatch could be set to open an AI-compatible cryptocurrency token wallet application (or traditional banking application) on the smartphone, preparing a transaction to this address [Scanning a QR code indicates the camera obtaining sensing data, i.e. reading the QR code, wherein though applied to a smartwatch in this example, in view of the camera being disclosed to be on smart glasses, this operation could similarly be performed with a camera on smart glasses without a change in functionality]); and,
sending, through the wireless communication component ([In view of the previously disclosed wireless communication component of Gross]), the sensing data of the data acquisition device and the first text to the model server to generate the second text through the GAILLMs based on the sensing data of the data acquisition device and the first text ([0116] The user could then speak the desired transaction amount into their smartglasses 200, hear a confirmation prompt, speak their consent, and see a visual receipt on the smartwatch 300. In this way the smartglasses mobile application 201 can assist the user in communication with public blockchains. Extrapolating from this function, the smartglasses mobile application 210 could be used to read QR codes in one device, and manage actions on it in another [Generating a receipt, i.e. second text, based on the image of a QR code and input transaction amount, i.e. first text, indicates generation of a second text through GAILLMs (in view of the previously disclosed GAILLMs of Gross/Gross2) based on sensing data, i.e. image data, and a first text]).
Regarding claim 14, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 1.
Gross further discloses:
wherein the one or more computer programs further comprise instructions for:
converting, through the speech-to-text engine, the first speech into a first text ([0202] Based on user selected settings, the voice assistant then plays back the recorded message or a speech-to-text transcription of the message, whereupon the user selects to send or rerecord the message by voice command [Generation of a transcription of a spoken message indicated a speech-to-text conversion]);
sending, through the Bluetooth component ([In view of the previously disclosed Bluetooth component of Gross]), the first text to the smart mobile terminal ([Fig. 8, 830], [0124] The web-enabled device 100 audibly presents the custom voice operation command 820 via audible commands 830 to the VA 835, [Wherein VA (voice assistant) 835 is clearly a smart mobile terminal, see [0003] for example voice assistants which are traditionally located on smart mobile terminals (physical devices), i.e. Amazon Echo featuring Alexa, and the information being sent is transcribed before sending, see mapping in “converting” step]);
receiving the second text comprising the reply from the smart mobile terminal ([0193] The voice assistant will then interface with the Vyrb app to collect the designated data in real time, and use the voice assistant's text-to-speech engine to vocalize it for the user [Collecting data to be translated into speech indicates the data is originally in a textual format]); and,
converting, through the text-to-speech engine, the second text into the second speech ([0129] a new command with the chosen name, which can be spoken to the device-level voice assistant to retrieve the text in the selected area and vocalize with the device's text to speech engine [Vocalizing retrieved text, wherein the retrieved text can be in response to a question, indicates converting a second text into a second speech]).
Gross2 further discloses:
wherein the smart mobile terminal sends the first text to the model server ([0028] The text data and/or signals may be transferred via the exemplary disclosed communication techniques from audio assistant 330 to chatbot platform 335 as shown at 350 [Wherein the chatbot platform represents the model server and an audio assistant represents a smart mobile terminal in view of the smart mobile terminals of Gross]); and,
the second text is generated through the model server based on the first text and the GAILLMs ([0047] At step 460, chatbot platform 335 may select one or more chatbots for responding to the received text data and/or signals of the one or more user queries and/or prioritize responses from the one or more chatbots. For example, chatbot platform 335 may select, prioritize, combine, analyze, integrate, process, and/or utilize response text data… [0054] audio assistant 330 may receive the exemplary disclosed finalized text data and/or signals from chatbot platform 335. At step 495, audio assistant 330 may convert the finalized text data and/or signals to voice data and/or signals and transfer the voice data and/or signals to the exemplary disclosed device (e.g., glasses 310) [Wherein a chatbot represents a model server, the chatbots are comprised of GAILLMs (see [0034]), and a second text, i.e. response, is generated based on the input text (Fig. 4, 350) and GAILLMs. Further, consider Fig. 11, 455 which receives text before selecting a chatbot for response, indicating second text generated based on a first text]).
Regarding claim 15, Gross discloses: the smart glasses of claim 1.
Gross further discloses:
wherein the one or more computer programs further comprise instructions for:
sending, through the Bluetooth component, the first speech to the smart mobile terminal ([0124] The web-enabled device 100 audibly presents the custom voice operation command 820 via audible commands 830 to the VA 835 [A voice command operation tracks to a first speech, the VA (voice assistant) tracks to a smart mobile terminal see [0003] for example voice assistants which are traditionally located on smart mobile terminals (physical devices), i.e. Amazon Echo featuring Alexa]); and,
receiving the second speech from the smart mobile terminal ([0125] providing the user 805 with the follower 875 audio response 895), wherein the smart mobile terminal converts the first speech into a first text ([0202] a speech-to-text transcription of the message [Wherein the message is audio recorded by a user, indicating a first speech]).
Gross2 further discloses:
sends the first text to the model server ([0046] audio assistant 330 may receive voice data and/or signals and convert the voice data and/or signals to text data and/or signals at step 450 as shown at 340 and 345 of FIG. 4 (e.g., and/or system 300 may directly receive and convert data into text data and/or signals for example as described regarding step 440). At step 455 and as shown at 350 of FIG. 4, chatbot platform 335 may receive the text data [Wherein a chatbot platform represents a model server]), the model server obtains the second text comprising the reply by inputting the first text into the GAILLMs ([0047] chatbot platform 335 may select one or more chatbots for responding to the received text data and/or signals of the one or more user queries and/or prioritize responses from the one or more chatbots. For example, chatbot platform 335 may select, prioritize, combine, analyze, integrate, process, and/or utilize response text data and/or signals from any desired number of suitable chatbots (e.g., chatbot types) for example as described herein. In at least some exemplary embodiments, multiple chatbots (e.g., including ChatGPT and/or any other suitable chatbots for example as described herein) may be integrated and queried at one time by chatbot platform 335 and results may be presented by voice or text [Using chatbots to generate response text data indicates the model server, i.e. chatbot platform, obtains a second text based on input, i.e. received, first text into a GAILLM (GPT)]); and,
sends the second text to the smart mobile terminal ([0054] audio assistant 330 may receive the exemplary disclosed finalized text data and/or signals from chatbot platform 335 [Wherein an audio assistant represents a smart mobile terminal]), and the smart mobile converts the second text into the second speech and sends the second speech to the smart glasses ([0054] audio assistant 330 may convert the finalized text data and/or signals to voice data and/or signals and transfer the voice data and/or signals to the exemplary disclosed device (e.g., glasses 310)).
Regarding claim 17, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 1.
Gross further discloses:
wherein the one or more computer programs further comprise instructions for:
sending, through the Bluetooth component, the device data of the smart glasses and the first text to the smart mobile terminal ([Fig. 8, 830], [0124] The web-enabled device 100 audibly presents the custom voice operation command 820 via audible commands 830 to the VA 835, [Wherein VA (voice assistant) 835 is clearly a smart mobile terminal, see [0003] for example voice assistants which are traditionally located on smart mobile terminals (physical devices), i.e. Amazon Echo featuring Alexa, and the information being sent is transcribed before sending, see mapping in “converting” step]); and
receiving, through the Bluetooth component ([In view of the previously disclosed Bluetooth component of Gross]), the second text comprising the reply from the smart mobile terminal ([0193] The voice assistant will then interface with the Vyrb app to collect the designated data in real time, and use the voice assistant's text-to-speech engine to vocalize it for the user [Collecting data to be translated into speech indicates the data is originally in a textual format]), and
wherein the data acquisition device on the smart mobile terminal comprises at least one component of: the position sensor, the inertial measurement unit sensor, the temperature sensor, the proximity sensor, the humidity sensor, the electronic compass, the timer, the camera and the pedometer ([0116] smartglasses 200 or smartwatch 300 contains an embedded camera capable of transmitting visual information to a connected smartphone 400 AI interface 460) and a pedometer, and the least one component is electrically connected to the processor ([In view of the previously disclosed processor on the smart glasses of Gross, indicating a camera on the same glasses will also be electrically connected to the same processor. The examiner would like to note that due to the disjunctive structure of the claim, not all sensors require a mapping]).
Gross2 further discloses:
converting, through the text-to-speech engine, the second text comprising the reply into the second speech ([Fig. 11, “Convert Text to Voice Data/Signals”], [0054] At step 495, audio assistant 330 may convert the finalized text data and/or signals to voice data and/or signals and transfer the voice data and/or signals to the exemplary disclosed device (e.g., glasses 310),
wherein the smart mobile terminal receives the device data of the smart glasses and the first text from the smart glasses ([0028] Text data and/or signals may be transferred via the exemplary disclosed communication techniques from chatbot platform 335 to audio assistant 330 as shown at 355, [Outputting text data and/or signals from the chatbot, i.e. model server, indicates a required reception of the output data at the audio assistant. Further, consider Fig. 11 of Gross2 which discloses reception of text and/or image queries (415) to then be sent to chatbot (455) and sent out of chatbot (475) as response data/signals indicating the image/text to be description information of required data for generating the response. Consider a received audio/image signal to be device data and a transcription, i.e. TTS conversion, to be the first text]),
obtains device and environmental data of the smart mobile terminal through the data acquisition device on the smart mobile terminal ([0028] Audio or sound data and/or signals (e.g., of the speech or voice query provided at 340) may be transferred via the exemplary disclosed communication techniques from the exemplary disclosed device (e.g., glasses 310, headphones 315, and/or user device 320) to audio assistant 330 as shown at 345, [Audio data received by a microphone in an environment on a device indicates the received audio signals to be either device and/or environmental data of the smart mobile terminal, i.e. the device receiving the sounds in the environment]), and
sends the first text to the model server ([0028] The text data and/or signals may be transferred via the exemplary disclosed communication techniques from audio assistant 330 to chatbot platform 335 as shown at 350 [Wherein the chatbot platform represents the model server]),
when the smart mobile terminal receives the description information, the smart mobile terminal determines the required data from the device data of the smart glasses and the device and environmental data of the smart mobile terminal according to the description information ([0028] Audio or sound data and/or signals (e.g., of the speech or voice query provided at 340) may be transferred via the exemplary disclosed communication techniques from the exemplary disclosed device (e.g., glasses 310, headphones 315, and/or user device 320) to audio assistant 330 as shown at 345, [Audio data received by a microphone in an environment on a device indicates the received audio signals to be either device and/or environmental data of the smart mobile terminal, i.e. the device receiving the sounds in the environment]), and
sends the required data to the model server ([In view of the previously disclosed first text being sent to the chatbot platform, i.e. model server, wherein the first text is required data for generating a response]), so that the model server obtains, through the GAILLMs, the second text comprising the reply based on the required data and the first text ([0115] The user could then initiate an on-blockchain token transfer, check a token balance, and more via their smartglasses 200 alone. The smartglasses could communicate confirmation prompts, balance inquiries etc., back to the user. [Communicating information back to a user based on input indicates that communicated information is a response]) and sends the second text comprising the reply to the smart mobile terminal ([In view of the previously disclosed information sending from chatbot, i.e. model server, 335 to audio assistant, i.e. smart mobile terminal, 330 through operation 355, wherein the information sent includes text data, as performed for the description information]).
Regarding claim 28, Gross discloses: a computer-implemented method for controlling a smart wearable device based on generative artificial intelligence large language models (GAILLMs) ([0091] AI of the smartglasses 200 [Wherein smartglasses track to a smart wearable device, AI in the context of response generation is clearly generative, i.e. generating responses, and based on large language models, in order to generate meaningful responses]), applied to the smart wearable device, wherein the method comprises:
activating a chat function of the smart glasses in response to a first control instruction for activating the chat function ([0104] the microphone 225 of smartglasses 200 and smartglasses AI interface 280 receive an oral request or command audibly proceeded by language, such as “tell me” or “show me,” that is communicated to glasses' AI interface 280… [0197] an embodiment Vyrb implementation may permit a user to create a verbal “hotkey” configured to trigger on-demand activation of a customizable range of data retrieval and user action patterns [“tell me” or “show me” represent first control instructions, i.e. verbal hotkeys, to activate a chat function, i.e. for data retrieval operations, in view of [0037] keyword activation of the instant application.]);
obtaining, through a built-in microphone ([Considering the previously disclosed microphone 225]), a first speech of a user ([Fig. 7, 710]), wherein the first speech comprises a question asked by the user ([0104] Examples of questions can include questions designed to perform tasks, open applications, including map or navigation, music, wallet, application store, calculator, watch, stopwatch, banking or money transfer, video, games, stocks, weather, etc., [Taking the received oral request by the smartglasses of step 710 in view of the question asking of [0104] indicates the oral requests are first speeches comprising questions]); and,
obtaining, through the GAILLMs, a second speech comprising a reply corresponding to the question ([0115] The user could then initiate an on-blockchain token transfer, check a token balance, and more via their smartglasses 200 alone. The smartglasses could communicate confirmation prompts, balance inquiries etc., back to the user. [Communicating information back to a user based on input indicates that communicated information is a response]), and playing, through a built-in speaker ([Fig. 2, speaker 235]), the second speech ([Fig. 8, 895], [0139] At 1220, based on predefined user settings for different types of content and suitable response actions thereof, a voice prompt will trigger from the voice assistant after the content finishes playback, and can ask the user to respond “like,” or take another action on the content via the user's voice [Triggering a voice prompt for the end of content, in view of the questions/responses of [0115], indicates the content, i.e. second speech, is a response]);
wherein the smart glasses further comprise a Bluetooth component electrically connected to the processor ([Fig. 2], [0061] connectivity module and hardware 260 may use WiFi, Bluetooth [In view of the previously disclosed processor of Gross, indicating the hardware consists of at least a processor]), and the one or more computer programs further comprise instructions for:
receiving, through a built-in Bluetooth component ([Fig. 2], [0061] connectivity module and hardware 260 may use WiFi, Bluetooth), the first control instruction from a smart mobile terminal ([0202] The user can activate their voice assistant via the phone or watch, or connected Bluetooth glasses…The user can also dictate a vocal post to their profiles on other platforms and direct messages to users on other platforms, without posting to the Vyrb app. For example, “Hey Siri, use Vyrb to send a message to John Q on Facebook.” [The first control instruction represents “Hey Siri…”, in view of [0037] of the instant application, the smart phone and/or watch represent a smart mobile terminal in view of Fig. 3 of the instant application])
activating the chat function ([0059] The device level voice assistant manager process may be programmed to activate a device level voice assistant to perform an operation requested by a user, based on configuring the device level voice assistant with voice commands determined by the device level voice assistant manager process executing on the user's mobile device [Performing operations based on received voice input indicates generation of control instructions after a wake-up is detected, in view of the “Hey Siri” wake-up of [0202], a well-known wake word to anyone familiar with Apple products. A device level voice assistant indicates the voice assistant is on a mobile smart terminal, i.e. device, which can include an interface in the form of a cell phone, see smart phone of Gross (Fig. 3)]) and waking up the microphone while activating the chat function ([0197] an embodiment Vyrb implementation may permit a user to create a verbal “hotkey” configured to trigger on-demand activation of a customizable range of data retrieval and user action patterns [“tell me” or “show me” represent first control instructions, i.e. verbal hotkeys, to activate a chat function, i.e. waking a microphone for data, i.e. input speech, retrieval operations, in view of [0037] keyword activation of the instant application. Further in view of “Hey Siri” ([0202]) which is a clear wake term for activating a microphone for voice analysis]);
receiving, through the built-in Bluetooth component, a second control instruction for deactivating the chat function from the smart mobile terminal ([Figs. 10A-C], [0133] The Vyrb App begins recording user actions on the device, including taps and keystrokes. When the desired action(s) have been performed, the user can hit an overlaid check button to complete the recording [Hitting a button to stop recording, i.e. chat function, on a smartphone, i.e. smart mobile terminal, represents a second control instruction, wherein the smartglasses of Gross contain a bluetooth component (see [0061]), and Gross discloses smartglasses capable of recording (see [0104]), indicating a system to receive through bluetooth a control instruction for deactivation of a chat function on smartglasses from a smart mobile terminal]).
Gross does not disclose:
deactivating the chat function and the built-in microphone in response to the first control instruction.
Kang discloses:
deactivating the chat function and the built-in microphone in response to the first control instruction ([0150] if the hardware button is pressed two times more, the electronic device 101 may deactivate the microphone. Or, in a case where the user utterance received in operation 870 indicates continuous command mode termination, the electronic device 101 (e.g., the processor 120 or 210) may terminate the continuous command mode [Termination of continuous command mode, wherein command mode tracks to processing of user utterances (see Fig. 8), indicates termination of a chat function in view of the chat function of Gross. Further in view of Fig. 34 indicating the second control instruction for deactivating a chat function is clearly originating from a smart mobile terminal, i.e. smartphone]).
Gross and Kang are considered analogous art within speech command processing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross to incorporate the teachings of Kang, because of the novel way to select any one of a number of various user utterance processing modes which consider device state based on received input for improved conversation services (Kang, [0005]).
Gross further discloses:
wherein the GAILLMs are configured on a model server ([0077] AI software such as Siri, Bixby or Alexa, [Wherein the three previous software assistants are all based on GAILLM architecture as required for question responses. The model server is the mobile device containing mobile application which contains a voice assistant, i.e. GAILLMs ([0059])]), and
the step of obtaining, through the GAILLMs, the second speech comprising the reply corresponding to the question comprises:
converting, through a built-in speech-to-text engine, the first speech into the first text ([0237] 1. Speech to text functionality);
sending, through the wireless communication component ([In view of the previously disclosed wireless communication component]), the first text to the model server ([Fig. 7, S710 “Smartglasses receive an oral request or command”, S715 “Request or command…to perform a particular…task is communicated to the mobile application”], [In view of the previously disclosed speech-to-text of Gross]);
receiving description information of required data from the model server ([Fig. 7, S720 “Access lookup tables and present any information related to the requested device…for display and presentation”], [If the task/command is presentation, then the requested device information is describing how the required data should be presented. The mobile device is the model server]),
wherein the model server, through the GAILLMs, according to the first text ([In view of the previously disclosed sending of first text to mobile device]), determines whether the reply needs to be based on data collected by the data acquisition device on the smart wearable device and/or a data acquisition device on a smart mobile terminal ([Fig. 7, S730 “Smartglasses receive optional override or switch command or request to change the paired or connected device used for presentation or display”], [Choosing a presentation device based on the type of task at a mobile device indicates a determination that, based on the first text, the connected device is chosen, wherein each device will necessarily have data to be collected (as disclosed in S720)]),
when the reply corresponding to the question needs to be based on the data, generates the description information of the required data ([As Gross is previously disclosed to be generating replies based on data, this necessarily indicates that the reply needs to be based on the data]); and
selects to send the description information to the smart wearable device or the smart mobile terminal ([As previously disclosed in S730, the smartglasses receiving the request to change presentation device will necessarily be comprised of description information to be read by the smart glasses which describe the override action in a way that the glasses can understand]), and
wherein when the reply corresponding to the question does not need to be based on the data, the model server obtains a second text comprising the reply through the GAILLMS based on the first text ([As previously disclosed in [0115], the ability of the smartglasses to return answers to a user based on an input indicates a reply generated through GAILLMs (as previously disclosed) without the need for additional information about where the information is to be presented. Consider a situation in which the optional override is not received by the smart glasses]); and
wherein the data acquisition device comprises at least one component of: a position sensor, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera ([0116] smartglasses 200 or smartwatch 300 contains an embedded camera capable of transmitting visual information to a connected smartphone 400 AI interface 460) and a pedometer;
when receiving the description information, determining the required data from the device data of the smart glasses according to the description information ([There is no limitation requiring the required data to be less than the complete device data; therefore, as Gross previously discloses device data and description information, it is reasonable to assert that Gross also discloses the required data as claimed, i.e. in QR image collection (a described task via description information of an input request), the image gathered by the device, e.g. device data, is all relevant, and therefore, required]), and
sending, through the built-in wireless communication component, the required data to the model server ([As previously disclosed in S715, smart glasses send request/task data, i.e. the required data, to the mobile device, i.e. model server]),
wherein the model server inputs the required data to the GAILLMs to obtain the second text after receiving the required data ([0140] The user requests their five newest inbound Vyrb posts through the device-level voice assistant. After each post is read out, the voice assistant prompts the user if they would like to leave a voice comment. If the user says yes, they record a comment, [A voice assistant is reasonably understood to be any of the AI software (GAILLMs) previously cited. Further, wherein the voice request from the user can be gathered from the smart glasses and sent to the smart mobile device as disclosed in Fig. 7, indicating a model server inputting the required data into the GAILLMs, i.e. inputting the post data into the voice assistant, wherein the voice assistant asking whether or not the user would like to leave a voice comment is second text vocalized by the voice assistant based on the required data, i.e. the posts]), and
wherein the GAILLMS associate the required data with the first text ([0059] mobile application 401 may include a device level voice assistant manager process, [Wherein the required data is the device data as previously disclosed. Further, as previously disclosed, sending a request or command to a mobile application, wherein that mobile application is used for controlling presentation of tasks (S715) indicates a situation in which the GAILLMs, i.e. the voice assistant on mobile application, associate the device data with the first text through the sending operation. As the command is sent to the mobile application to be answered/presented (S715), device data is gathered (S720) in relation to the request, indicating a situation in which the mobile application (containing the voice assistant) associates the device, i.e. required, data with the received first text]), and
obtain the second text using the required data and the first text ([Fig. 7, S740 “smartglasses communicate the override…controlling the paired or connected devices used for display or presentation of tasks or applications”[As previously disclosed in [0140] of Gross, vocalizing a question/comment asked by the voice assistant indicates a received text, i.e. prompt, to be vocalized, wherein the vocalization is performed based on the device connected to display and/or presentation of tasks (S740 of Fig. 7) indicating the second text to be received by the user using the required device data and first text]).
Gross in view of Kang is not relied upon to disclose:
receiving, through the wireless communication component, the second text comprising the reply from the model server, and
converting, through the text-to-speech engine, the second text comprising the reply into the second speech.
Gross2 discloses:
receiving, through the wireless communication component, the second text comprising the reply from the model server ([Fig. 11, “Receive text and/or image data/signals 490”], [0054] At step 490 and as shown at 355 of FIG. 4, audio assistant 330 may receive the exemplary disclosed finalized text data and/or signals from chatbot platform 335), and
converting, through the text-to-speech engine, the second text comprising the reply into the second speech ([Fig. 11, “Convert Text to Voice Data/Signals”], [0054] At step 495, audio assistant 330 may convert the finalized text data and/or signals to voice data and/or signals and transfer the voice data and/or signals to the exemplary disclosed device (e.g., glasses 310).
Gross, Kang, and Gross2 are considered analogous art within smart device chatbot communication. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang to incorporate the teachings of Gross2, because of the novel way to introduce an intermediate audio assistant before sending received audio to a chatbot, allowing for a voice interface for more powerful text-based chatbots traditionally lacking voice interfaces (Gross2, [0005]).
Regarding claim 30, Gross in view of Kang, further in view of Gross2 discloses: the method of claim 28.
Gross further discloses: wherein the first control instruction is a first voice instruction ([Fig. 7, 710], [Receiving an oral request is equivalent to a first voice instruction]); and,
the method further comprises:
obtaining, through a built-in virtual assistant program ([0050] a device-level voice assistant [Wherein a smart mobile terminal represents a device]), the first voice instruction ([Fig. 15, 1520], [0146] Upon determining a spoken user voice command is detected, at 1515 the process creates a custom voice command based on combining command parameters determined from user voice input captured with the detected user voice command. At 1520, the process activates the voice assistant with the custom voice command to perform the user requested operation [Activating a voice assistant, i.e. virtual assistant, with a custom voice command for performing operations indicates the voice command is an obtained first instruction]).
Kang further discloses:
obtaining, through the built-in virtual assistant program ([In view of the previously disclosed virtual assistant program of Gross]), a second voice instruction ([Fig. 8, 880], [Checking for a termination event, wherein the inputs to be checked are user utterances 830/870, indicates the termination event is comprised of one of more keywords from within the user utterance to know to deactivate the continuous command mode, i.e. chat function]); and,
deactivating the chat function in response to the second voice instruction ([0217] the electronic device 101 may obtain a user utterance 3160 with the hardware button 3113 pressed. The electronic device 101 may send data about the obtained user utterance to the external server and receive and display a text 3161 corresponding thereto. In a case where a text designated to terminate the dialog mode, such as “Chat mode Off,” is received, the electronic device 101 may terminate the dialog mode corresponding to releasing the press of the hardware button 3113 [Wherein “chat mode off” is a second voice instruction used for deactivation the chat function, i.e. chat mode, based on a spoken utterance])
Regarding claim 31, Gross in view of Kang, further in view of Gross2 discloses: the method of claim 30.
Gross further discloses:
obtaining, through the built-in virtual assistant program ([In view of the previously disclosed voice assistant of Gross]), a user voice ([0124] The web-enabled device 100 audibly presents the custom voice operation command 820 via audible commands 830 to the VA 835. The custom voice operation command 820 may include the user voice command 825); and,
determining that the first voice instruction is obtained, when the user voice comprises a preset wake-up word ([0204] For example, “Hey Siri, use Vyrb to send a message to John Q on Facebook.” The app would then prompt the user to record the message, which would be sent as a voice message (if permitted by the other platform) or transcribed into text and sent [“Hey Siri” is a well-known voice instruction for activating a chat function, using “Hey Siri” and then following with a command indicates “Hey Siri” as a first voice instruction]).
Regarding claim 35, Gross in view of Kang, further in view of Gross2 discloses: the method of claim 28.
Kang further discloses:
generating chat logs ([0145] the DM 757 may manage the user's chat history… [0258] the memory 4440 may include a log database capable of storing log information [Disclosing chat history and logs indicates that the chat history is stored in the form of a log, indicating generation]), and sending, through the built-in wireless communication component ([In view of the previously disclosed wireless communication components of Gross/Gross2]), the chat logs and the first text to the model server, so that the GAILLMs generates the second text based on the chat logs and the first text ([0145] The DM 757 may manage the dialog history (dialog flow) and manage the dialog status of the current chat window. The DM 757 may determine whether to utilize information about the previous dialog or to make a new inquiry depending on the dialog status [Determining to utilize information from previous dialog indicates using a chat log and/or first text (which would be a previous dialog statement at the current time of operation) for generating a second text, i.e. a new inquiry or one based on the history. In view of the GAILLM system of Gross/Gross2 which could be used as the dialog manager (DM) of Kang without a change in functionality]).
Regarding claim 38, Gross in view of Kang, further in view of Gross2 discloses: the method of claim 28.
Gross further discloses:
wherein after obtaining, through the built-in microphone, the first speech of the user, the method further comprises:
sending, through the built-in Bluetooth component, the device data of the smart wearable device and the first speech to the smart mobile terminal ([Fig. 8, 830], [0124] The web-enabled device 100 audibly presents the custom voice operation command 820 via audible commands 830 to the VA 835, [Wherein VA (voice assistant) 835 is clearly a smart mobile terminal, see [0003] for example voice assistants which are traditionally located on smart mobile terminals (physical devices), i.e. Amazon Echo featuring Alexa, and the information being sent is transcribed before sending, see mapping in “converting” step]); and
receiving, through the built-in Bluetooth component ([In view of the previously disclosed Bluetooth component of Gross]), the second speech comprising the reply from the smart mobile terminal ([0193] The voice assistant will then interface with the Vyrb app to collect the designated data in real time, and use the voice assistant's text-to-speech engine to vocalize it for the user),
wherein the smart mobile terminal converts the first speech into first text ([0077] Speech to text utility of AI software),
Gross2 further discloses:
obtains device and environmental data of the smart mobile terminal through the data acquisition device on the smart mobile terminal ([0028] Audio or sound data and/or signals (e.g., of the speech or voice query provided at 340) may be transferred via the exemplary disclosed communication techniques from the exemplary disclosed device (e.g., glasses 310, headphones 315, and/or user device 320) to audio assistant 330 as shown at 345, [Audio data received by a microphone in an environment on a device indicates the received audio signals to be either device and/or environmental data of the smart mobile terminal, i.e. the device receiving the sounds in the environment]), and
sends the first text to the model server ([0028] The text data and/or signals may be transferred via the exemplary disclosed communication techniques from audio assistant 330 to chatbot platform 335 as shown at 350 [Wherein the chatbot platform represents the model server]),
wherein when the smart mobile terminal receives the description information, the smart mobile terminal further determines the required data from the device data of the smart wearable device and the device and environmental data of the smart mobile terminal according to the description information ([0028] Audio or sound data and/or signals (e.g., of the speech or voice query provided at 340) may be transferred via the exemplary disclosed communication techniques from the exemplary disclosed device (e.g., glasses 310, headphones 315, and/or user device 320) to audio assistant 330 as shown at 345, [Audio data received by a microphone in an environment on a device indicates the received audio signals to be either device and/or environmental data of the smart mobile terminal, i.e. the device receiving the sounds in the environment]), and
sends the required data to the model server ([In view of the previously disclosed first text being sent to the chatbot platform, i.e. model server, wherein the first text is required data for generating a response]), so that the model server obtains the second text through the GAILLMs based on the required data and the first text ([0115] The user could then initiate an on-blockchain token transfer, check a token balance, and more via their smartglasses 200 alone. The smartglasses could communicate confirmation prompts, balance inquiries etc., back to the user. [Communicating information back to a user based on input indicates that communicated information is a response]) and sends the second text comprising the reply to the smart mobile terminal ([In view of the previously disclosed information sending from chatbot, i.e. model server, 335 to audio assistant, i.e. smart mobile terminal, 330 through operation 355, wherein the information sent includes text data, as performed for the description information. Considering the first and second queries of [Fig. 11, 425], it is reasonable to indicate that the previously cited text sending operation for a first text would be extended to the second query without a change in functionality to Gross2]); and
wherein the smart mobile terminal further converts the second text into the second speech and sends the second speech to the smart wearable device ([Considering the first and second queries of [Fig. 11, 425], it is reasonable to indicate that the previously cited text conversion operation for a first text would be extended to the second query without a change in functionality to Gross2]).
Claim(s) 6, 32 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gross in view of Kang, further in view of Gross2, further in view of Hansen et al. (US-12301634-B2), hereinafter Hansen.
Regarding claim 6, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 5.
Gross in view of Kang, further in view of Gross2 does not disclose:
wherein obtaining, through the microphone, the first speech after the user voice comprising the third preset keyword is obtained through the microphone comprises:
extracting a voice print in the user voice, when the user voice comprising the third preset keyword is obtained through the microphone; and,
when the user passes the identity authentication, obtaining, through the microphone, the first speech.
Hansen discloses:
wherein obtaining, through the microphone ([In view of the previously disclosed microphone of Gross]), the first speech after the user voice comprising the third preset keyword is obtained through the microphone comprises:
extracting a voice print in the user voice ([Col. 92, Lines 5-10] server 916 determines whether user 902 is a registered user of user device 910 based on user data associated with user device 910 (e.g., user voice profile information associated with a registered user of user device 910)) [Comparing voice profiles, i.e. prints, of an input utterance to registered users indicates the input utterances also needs to be in a voice profile form for comparison]), when the user voice comprising the third preset keyword is obtained through the microphone ([Col. 93, Line 7] “Hey Siri, what is Star Wars.”), which is received by communal device 1204.) [In view of the first keyword “Siri” or “Hey Siri” as previously disclosed in Gross, indicating “what” as the third keyword obtained]).
performing an identity authentication on the user according to the voice print ([Col. 84, Lines 30-35] device selection module 1110 may select the at least two electronic devices for multimodal task performance if device selection module 1110 determines that the plurality of tasks includes a task of user authentication [In view of the previously disclosed voiceprints of Hansen for determining if the user if registered]); and,
when the user passes the identity authentication, obtaining, through the microphone, the first speech ([Col. 89, Lines 44-47] After authenticating a user or receiving an indication that a user has been authenticated, command module transmits the commands corresponding to the remaining sets of tasks to the electronic device that provided the user voice input data [There is no limitation within Hansen that the remaining sets of tasks are limited to being received before authentication. The very first required task of Hansen could be authentication. For example, an input could be “Hey Siri, can…” representing audio to be used for identity authentication containing first and third keywords, wherein the remaining task follows the authentication, i.e. … you tell me the password to this account?”, indicating obtaining through a microphone, in view of the microphone of Gross, first speech if the user is authenticated]).
Gross, Kang, Gross2, and Hansen are considered analogous art within chatbot conversational analysis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang, further in view of Gross2 to incorporate the teachings of Hansen, because of the novel way to determine tasks and corresponding device identifiers based on input voice data and context information, improving the digital assistant's accuracy in determining/responding to user intent (Hansen, [Col. 11, Lines 40-67]).
Regarding claim 32, Gross in view of Kang, further in view of Gross2 discloses: the method of claim 28.
Gross in view of Kang, further in view of Gross2 does not disclose:
wherein the step of obtaining, through the built-in microphone, the first speech of the user comprises:
when a user voice comprising a preset keyword is obtained through the built-in microphone, extracting a voice print in the user voice; and,
performing an identity authentication on the user based on the voice print, and,
obtaining, through the built-in microphone, the first speech when the user passes the identity authentication, wherein the preset keyword is configured to indicate that the user is beginning to ask the question.
Hansen discloses:
wherein the step of obtaining, through the built-in microphone ([In view of the previously disclosed microphone of Gross]), the first speech of the user comprises:
when a user voice comprising a preset keyword is obtained through the built-in microphone, extracting a voice print in the user voice ([Col. 92, Lines 5-10] server 916 determines whether user 902 is a registered user of user device 910 based on user data associated with user device 910 (e.g., user voice profile information associated with a registered user of user device 910)) [Comparing voice profiles, i.e. prints, of an input utterance to registered users indicates the input utterances also needs to be in a voice profile form for comparison]);
performing an identity authentication on the user based on the voice print ([Col. 84, Lines 30-35] device selection module 1110 may select the at least two electronic devices for multimodal task performance if device selection module 1110 determines that the plurality of tasks includes a task of user authentication [In view of the previously disclosed voiceprints of Hansen for determining if the user if registered]); and,
obtaining, through the built-in microphone, the first speech when the user passes the identity authentication ([Col. 89, Lines 44-47] After authenticating a user or receiving an indication that a user has been authenticated, command module transmits the commands corresponding to the remaining sets of tasks to the electronic device that provided the user voice input data [There is no limitation within Hansen that the remaining sets of tasks are limited to being received before authentication. The very first required task of Hansen could be authentication. For example, an input could be “Hey Siri, can…” representing audio to be used for identity authentication containing first and third keywords, wherein the remaining task follows the authentication, i.e. … you tell me the password to this account?”, indicating obtaining through a microphone, in view of the microphone of Gross, first speech if the user is authenticated]);
wherein the preset keyword is configured to indicate that the user is beginning to ask the question ([Col. 93, Line 7] “Hey Siri, what is Star Wars.”), which is received by communal device 1204.) [In view of the first keyword “Siri” or “Hey Siri” as previously disclosed in Gross, indicating beginning of questions]).
Gross, Kang, are considered analogous art within chatbot conversational analysis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang, further in view of Gross2 to incorporate the teachings of Hansen, because of the novel way to determine tasks and corresponding device identifiers based on input voice data and context information, improving the digital assistant's accuracy in determining/responding to user intent (Hansen, [Col. 11, Lines 40-67]).
Claim(s) 8-9, 33 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gross in view of Kang, further in view of Gross2, further in view of Lee et al. ("Interaction Methods for Smart Glasses"), hereinafter Lee.
Regarding claim 8, Gross in view of Kang discloses: the smart glasses of claim 5.
Gross further discloses:
wherein the one or more computer programs further comprise:
instructions for, before obtaining, through the GAILLMs ([In view of the previously disclosed GAILLMs of Gross]), the second speech comprising the reply corresponding to the question,
terminating the operation of obtaining the first speech ([Fig. 10B, Check Button], [0133] When the desired action(s) have been performed, the user can hit an overlaid check button to complete the recording, [Completing a recording is equivalent to terminating obtaining of audio]);
outputting a second prompt sound to prompt the user that the question is asked ([0138] enabling a voice assistant response prompt once content finishes playing [wherein content can represent speech, i.e. verbal content as disclosed in Gross]), when any of following events is detected:
Kang further discloses:
the user completing the second preset operation ([0109] The electronic device 101 may activate the microphone while the button is pressed and remains pressed and process a user utterance obtained through the activated microphone… [0125] The electronic device 101 may determine that input of the user utterance is complete according to release of the press-and-hold of the hardware button , [Wherein the second preset operation represents a type of button press, see claim 5 rejection, indicating completion of this operation is lifting of the finger from the button, i.e. only recording while the finger is pressing]);
the user performing a third preset operation on the button after completion of the second preset operation ([0150] if the user presses a hardware button twice, the microphone of the electronic device 101 may be activated and wait to receive a speech so that all voice commands issued later may be transferred to the external server until a user input corresponding to the continuous command mode termination is identified. Further, if the hardware button is pressed two times more, the electronic device 101 may deactivate the microphone [A double-press represents a third preset operation, in view of the button press and hold second preset operation, wherein the third operation can be performed after completion of the second preset operation resulting in two distinct utterance recordings]); or,
occurring a silence of a first preset duration ([The examiner would like to note that this limitation does not need to be addressed due to the disjunctive nature of the claims. Further, consider automatic sending of text based on a certain time period passing since the text was entered of Kang [0089] which could be similarly applied to speech before transformation into text as disclosed in Kang]);
Gross in view of Kang, further in view of Gross2 does not disclose:
wherein, the third preset operation comprises any one of: touching the virtual button, tapping the virtual button, and sliding on the virtual button installed on the temple.
Lee discloses:
wherein, the third preset operation comprises any one of: touching the virtual button, tapping the virtual button, and sliding on the virtual button installed on the temple ([Fig. 8, wherein touching (all letters), tapping (b, z, I, etc.), and sliding (a, b, c, etc.) are all performed for various letters using the trackpad, i.e. virtual button, of the claimed smart glasses (see Fig. 6)]).
Gross, Kang, are considered analogous art within smart device . Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang, further in view of Gross2 to incorporate the teachings of Lee, because of the novel way to incorporate various gestural interaction approaches into smart glasses as input commands for execution on smart glasses, improving the interaction/understanding capabilities of smart devices (Lee, [pg. 2, Para. 2-4]).
Regarding claim 9, Gross in view of Kang, further in view of Gross2, further in view of Lee discloses: the smart glasses of claim 8.
Kang further discloses:
wherein the one or more computer programs further comprise instructions for:
receiving, through the Bluetooth component ([In view of the bluetooth component of Gross]), a first configuration instruction from the smart mobile terminal ([Fig. 3, Clock 384], [0196] the electronic device 101 may obtain and display a next user utterance 2329 and, after a preset time (e.g., N sec), display the microphone activation button 2330 and perform an operation corresponding to, e.g., the user utterance additionally entered, [Containing an application with a clock in a program module indicates time keeping which can reasonably be performed on any computing device capable of keeping time and reading programs, i.e. the smart phone (smart mobile terminal) of Gross, in view of the preset time threshold for user utterances, wherein that preset time can represent a first configuration instruction. Further in view of the smart mobile terminal/smart glasses system of Gross indicating the bluetooth component can transfer data, i.e. the preset time, between these two sources, resulting in the smartglasses receiving the configuration instruction time threshold]); and,
configuring the first preset duration to a duration indicated by the first configuration instruction ([0196] if no additional command is made within a preset time (e.g., N sec), stop displaying the speech processing application screen and terminate the speech processing application in operation 2423 [Determining to terminate speech processing based on a time threshold being exceeded indicates a preset configuration instruction, i.e. threshold time/duration, in view of the speech processing smart glasses of Gross]).
Regarding claim 33, Gross in view of Kang, further in view of Gross2 discloses: the method of claim 28.
Gross further discloses:
outputting, through the built-in speaker, a first prompt sound to prompt the user to start asking the question ([0139] a voice prompt will trigger from the voice assistant after the content finishes playback, and can ask the user to respond… [0204] The app would then prompt the user to record the message [In view of the previously disclosed questions of Gross ([0104]), indicating that the voice prompt trigger used for a response could also be used as the prompt for an initial recording. Recording a message based on a prompt indicates waking up a microphone for recording based on the prompt]); and,
obtaining, through the built-in microphone, the first speech ([Fig. 8, 825], [0124] cause a VA to perform the user requested operation 815 in response to the user voice command 825 [In view of the sent (825) and received (895) first and second speeches indicating the voice command 825 is a first speech, i.e. request]).
Kang further discloses:
wherein the first control instruction is triggered based on a first preset operation on a button performed by the user ([0159] For example, if a hardware button is pressed or pressed-and-held while the virtual keyboard 1021 is displayed, the electronic device 101 may obtain a user utterance through the microphone and receive a text corresponding thereto from the external server [Hardware button being pressed represents a first preset operation on the button, obtaining an utterance represents the first control instruction based on the button being pressed]);
the button comprises a physical button and/or a touch sensor based virtual button ([0078] A hardware button (e.g., the key 256) [Hardware button indicates a physical button]);
the step of obtaining, through the built-in microphone ([In view of the microphone of Gross]), the first speech of the user comprises:
in response to a second preset operation on the button performed by the user, waking up the built-in microphone ([0079] first operation may include receiving a first-type user input through the button while the first user interface is not displayed on the touchscreen display, after receiving the first-type user input, receiving a first user utterance through the microphone 288 [In view of the plurality of button input types previously disclosed in Kang, receiving an utterance after a button press indicates the button is pressed to wake up the microphone]),
wherein the second preset operation comprises:
any one of long pressing the virtual button, short pressing the virtual button, touching the virtual button, tapping the virtual button, sliding on the virtual button, and pressing and holding the physical buttons ([0088] the user inputs may be selected from one of a single press on the button, a double press on the button, a triple press on the button, a single press-and-hold on the button, or a double press-and-hold on the button [The examiner would like to note that due to the disjunctive nature of the claim, not all elements require a mapping; therefore, the “pressing and holding” of a physical button has been mapped]), and wherein a duration of the short pressing is shorter than a duration of the long pressing ([Defining single press and single press-and-hold for the same button indicates that the press-and-hold is of a longer duration than just press, otherwise there would be no need to define both operations]).
Gross in view of Kang, further in view of Gross2 does not disclose:
wherein the third preset operation comprises:
any one of touching the virtual button, tapping the virtual button, and sliding on the virtual button.
Lee discloses:
wherein the third preset operation comprises:
any one of touching the virtual button, tapping the virtual button, and sliding on the virtual button ([Fig. 8, wherein touching (all letters), tapping (b, z, I, etc.), and sliding (a, b, c, etc.) are all performed for various letters using the trackpad, i.e. virtual button, of the claimed smart glasses (see Fig. 6)]).
Gross, Kang, Gross2, and Lee are considered analogous art within smart device performance/interaction. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang, further in view of Gross2 to incorporate the teachings of Lee, because of the novel way to incorporate various gestural interaction approaches into smart glasses as input commands for execution on smart glasses, improving the interaction/understanding capabilities of smart devices (Lee, [pg. 2, Para. 2-4]).
Kang further discloses:
receiving, through a built-in Bluetooth component ([In view of the previously disclosed bluetooth component of Gross]), a first configuration instruction from the smart mobile terminal ([Fig. 3, Clock 384], [0196] the electronic device 101 may obtain and display a next user utterance 2329 and, after a preset time (e.g., N sec), display the microphone activation button 2330 and perform an operation corresponding to, e.g., the user utterance additionally entered, [Containing an application with a clock in a program module indicates time keeping which can reasonably be performed on any computing device capable of keeping time and reading programs, i.e. the smart phone (smart mobile terminal) of Gross, in view of the preset time threshold for user utterances, wherein that preset time can represent a first configuration instruction. Further, in view of the smart mobile terminal/smart glasses system of Gross indicating the bluetooth component can transfer data, i.e. the preset time, between these two sources, resulting in the smartglasses receiving the configuration instruction time threshold]), and configuring the first preset duration to a duration indicated by the first configuration instruction ([0196] if no additional command is made within a preset time (e.g., N sec), stop displaying the speech processing application screen and terminate the speech processing application in operation 2423 [Determining to terminate speech processing based on a time threshold being exceeded indicates a preset configuration instruction, i.e. threshold time/duration, in view of the speech processing smart glasses of Gross]); and,
wherein before obtaining, through the GAILLMs, the second speech comprising the reply corresponding to the question, the method further comprises:
terminating the operation of obtaining the first speech and outputting a second prompt sound to prompt the user that the question is asked, when any of following events is detected:
the user completing the second preset operation ([0109] The electronic device 101 may activate the microphone while the button is pressed and remains pressed and process a user utterance obtained through the activated microphone… [0125] The electronic device 101 may determine that input of the user utterance is complete according to release of the press-and-hold of the hardware button , [Wherein the second preset operation represents a type of button press, see claim 5 rejection, indicating completion of this operation is lifting of the finger from the button, i.e. only recording while the finger is pressing]),
the user performing a third preset operation on the button after completion of the second preset operation ([0150] if the user presses a hardware button twice, the microphone of the electronic device 101 may be activated and wait to receive a speech so that all voice commands issued later may be transferred to the external server until a user input corresponding to the continuous command mode termination is identified. Further, if the hardware button is pressed two times more, the electronic device 101 may deactivate the microphone [A double-press represents a third preset operation, in view of the button press and hold second preset operation, wherein the third operation can be performed after completion of the second preset operation resulting in two distinct utterance recordings]), and
occurring a silence of a first preset duration ([The examiner would like to note that this limitation does not need to be addressed due to the disjunctive nature of the claims. Further, consider automatic sending of text based on a certain time period passing since the text was entered of Kang [0089] which could be similarly applied to speech before transformation into text as disclosed in Kang]).
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gross in view of Kang, further in view of Gross2, further in view of Sourov (US-20240119928-A1).
Regarding claim 10, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 1.
Gross in view of Kang, further in view of Gross2 does not disclose:
wherein the smart glasses further comprise an indicator light and/or a buzzer electrically connected to the processor, and the one or more computer programs further comprise instructions for:
outputting, through the indicator light and/or the buzzer, prompt information, wherein the prompt information is configured to indicate a state of the smart glasses, the state comprises a working state and an idle state; and,
the working state comprises: a starting speech pickup status, a speech pickup status, a completing speech pickup status, and a speech processing status.
Sourov discloses:
wherein the smart glasses further comprise an indicator light and/or a buzzer electrically connected to the processor ([0067] The eyewear device 100 may include a user-facing indicator (e.g., an LED, a speaker 191, or a vibrating actuator), [In view of the audio processor 413 of Fig. 4, a vibrating actuator tracks to a buzzer]), and the one or more computer programs further comprise instructions for:
outputting, through the indicator light and/or the buzzer, prompt information, wherein the prompt information is configured to indicate a state of the smart glasses, the state comprises a working state and an idle state ([0114] In some implementations the step of translating the prioritized speech 804 includes transmitting or presenting an indicator (e.g., a sound, a light, or other cue) to the other device 800 notifying the wearer that a translation is in progress [Notifying a wearer that translation is in progress indicates outputting prompt information, i.e. the indicator, wherein the indicator represents the state of the glasses, i.e. translating or not, and the states comprise a work, i.e. translating, and idle, i.e. not translating, state. The idle state is indicated by only presenting the indicator when translating, meaning there is no indicator, i.e. blank prompt information, in an idle state]); and,
the working state comprises: a starting speech pickup status, a speech pickup status, a completing speech pickup status, and a speech processing status ([0076] translates the prioritized speech 804 into a select language 841 [translation is a speech processing status. Due to the disjunctive nature of the claim, i.e. the working state comprising the statuses, not all elements require a mapping. Further, consider the audio cues for starting/ending recording of Gross, indicating status determinations, which could be applied to the indicator light of Sourov without a change in functionality, see Sourov Fig. 7, 702 which processes received speech]).
Gross, Kang, Gross2, and Sourov are considered analogous art within smart glass information processing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang, further in view of Gross2 to incorporate the teachings of Sourov, because of the novel way to translate to the incoming translation (Sourov, [0114]).
Claim(s) 16, 37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gross in view of Kang, further in view of Gross2, further in view of Wang et al (US-20210116960-A1), hereinafter Wang.
Regarding claim 16, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 1.
Kang further discloses:
instructions for receiving, through the Bluetooth component ([In view of the bluetooth component of Gross]), a second configuration instruction from the smart mobile terminal ([Fig. 3, Clock 384], [0196] the electronic device 101 may obtain and display a next user utterance 2329 and, after a preset time (e.g., N sec), display the microphone activation button 2330 and perform an operation corresponding to, e.g., the user utterance additionally entered, [Containing an application with a clock in a program module indicates time keeping which can reasonably be performed on any computing device capable of keeping time and reading programs, i.e. the smart phone (smart mobile terminal) of Gross, in view of the preset time threshold for user utterances, wherein that preset time can represent a first configuration instruction. Further in view of the smart mobile terminal/smart glasses system of Gross indicating the bluetooth component can transfer data, i.e. the preset time, between these two sources, resulting in the smartglasses receiving the configuration instruction time threshold. A second configuration instructions can be for a different preset amount of time compared to the first config instruction of claim 1]); and,
configuring the second preset duration to a duration indicated by the second configuration instruction ([0196] if no additional command is made within a preset time (e.g., N sec), stop displaying the speech processing application screen and terminate the speech processing application in operation 2423 [Determining to terminate speech processing based on a time threshold being exceeded indicates a preset configuration instruction, i.e. threshold time/duration, in view of the speech processing smart glasses of Gross. A second configuration instructions can be for a different preset amount of time compared to the first config instruction of claim 1]).
Gross in view of Kang, further in view of Gross2 does not disclose:
instructions for controlling the smart glasses to enter a standby state, deactivating the chat function, controlling the microphone to enter a sleep state; and,
outputting a third prompt sound to prompt the user that the smart glasses is about to enter the standby state, when a silence of a second preset duration is detected after playing the second speech.
Wang discloses:
instructions for controlling the smart glasses to enter a standby state, deactivating the chat function, controlling the microphone to enter a sleep state ([0035] the device may power off (e.g., turn off, enter a sleep mode), [Sleep mode tracks to a standby state. Wherein the device can be smart glasses, see [0030], and include a microphone, see [0023], indicating the microphone enters a sleep state as part of the device. In view of the glasses of Gross in view of Kang which contain chat functions, indicating this would also be deactivated as the device, i.e. smart glasses, is]); and,
outputting a third prompt sound to prompt the user that the smart glasses is about to enter the standby state ([0018] provide a notification to a user informing them that the device will be powered off in a short period of time [Power off/sleep mode track to standby states]), when a silence of a second preset duration is detected after playing the second speech ([0035] an embodiment may provide an audible notification that informs the user that the device will power off in 30 seconds if user input is not received [Wherein power off tracks to a standby state, see “sleep mode” as example power off mode of previously claim element. 30 seconds represents a second preset duration after playing the second speech, i.e. the notification]).
Gross, Kang, Gross2, and Wang are considered analogous art within user-device audio communication interfacing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang, further in view of Gross2 to incorporate the teachings of Wang, because of the novel way to improve methods for powering off a device by entering the device into a standby state if no new information is retrieved in a given interval of time (Wang, [0043]).
Regarding claim 37, Gross in view of Kang, further in view of Gross2 discloses: the method of claim 28.
Gross further discloses:
a built-in Bluetooth component ([0076] wireless connections to a main device, including Bluetooth connections).
Kang further discloses:
receiving, through a built-in Bluetooth component ([In view of the bluetooth component of Gross]), a second configuration instruction from the smart mobile terminal ([Fig. 3, Clock 384], [0196] the electronic device 101 may obtain and display a next user utterance 2329 and, after a preset time (e.g., N sec), display the microphone activation button 2330 and perform an operation corresponding to, e.g., the user utterance additionally entered, [Containing an application with a clock in a program module indicates time keeping which can reasonably be performed on any computing device capable of keeping time and reading programs, i.e. the smart phone (smart mobile terminal) of Gross, in view of the preset time threshold for user utterances, wherein that preset time can represent a first configuration instruction. Further in view of the smart mobile terminal/smart glasses system of Gross indicating the bluetooth component can transfer data, i.e. the preset time, between these two sources, resulting in the smartglasses receiving the configuration instruction time threshold. A second configuration instructions can be for a different preset amount of time compared to the first config instruction of claim 1]); and,
configuring the second preset duration to a duration indicated by the second configuration instruction ([0196] if no additional command is made within a preset time (e.g., N sec), stop displaying the speech processing application screen and terminate the speech processing application in operation 2423 [Determining to terminate speech processing based on a time threshold being exceeded indicates a preset configuration instruction, i.e. threshold time/duration, in view of the speech processing smart glasses of Gross. A second configuration instructions can be for a different preset amount of time compared to the first config instruction of claim 1]).
Gross in view of Kang, further in view of Gross2 does not disclose:
wherein after playing, through the built-in speaker, the second speech, the method further comprises:
when a silence of a second preset duration is detected, controlling the smart wearable device to enter a standby state, deactivating the chat function, controlling the built-in microphone to enter a sleep state, and
outputting a third prompt sound to prompt the user that the smart wearable device is about to enter the standby state.
Wang discloses:
wherein after playing, through the built-in speaker, the second speech, the method further comprises:
when a silence of a second preset duration is detected ([0035] an embodiment may provide an audible notification that informs the user that the device will power off in 30 seconds if user input is not received [Wherein power off tracks to a standby state, see “sleep mode” as example power off mode of previously claim element. 30 seconds represents a second preset duration after playing the second speech, i.e. the notification]), controlling the smart wearable device to enter a standby state, deactivating the chat function, controlling the built-in microphone to enter a sleep state ([0035] the device may power off (e.g., turn off, enter a sleep mode), [Sleep mode tracks to a standby state. Wherein the device can be smart glasses, see [0030], and include a microphone, see [0023], indicating the microphone enters a sleep state as part of the device. In view of the glasses of Gross in view of Kang which contain chat functions, indicating this would also be deactivated as the device, i.e. smart glasses, is]), and
outputting a third prompt sound to prompt the user that the smart wearable device is about to enter the standby state ([0018] provide a notification to a user informing them that the device will be powered off in a short period of time [Power off/sleep mode track to standby states]).
Gross, Kang, Gross2, and Wang are considered analogous art within user-device audio communication interfacing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang, further in view of Gross2 to incorporate the teachings of Wang, because of the novel way to improve methods for powering off a device by entering the device into a standby state if no new information is retrieved in a given interval of time (Wang, [0043]).
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gross in view of Kang, further in view of Gross2, further in view of Shin (US-20240201938-A1).
Regarding claim 18, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 4.
Gross in view of Kang, further in view of Gross2 does not disclose:
wherein the one or more computer programs further comprise instructions for:
in response to a fourth preset operation for controlling a volume performed by the user on the button, playing the second speech at a volume indicated by the fourth preset operation.
Shin discloses:
wherein the one or more computer programs further comprise instructions for:
in response to a fourth preset operation for controlling a volume performed by the user on the button ([0036] The user 102 speaks the utterance 202 of “ASSISTANT, TURN VOLUME DOWN” to control the volume of the media playback on the television… [0064] a user can invoke an automated assistant by making an invocation gesture, pushing one or more invocation buttons (physical and/or virtual) [Invoking an assistant via button, wherein the command is to reduce volume indicates a fourth operation performed by the user on the button, in view of the plurality of operations which can be performed on a button of Kang]), playing the second speech at a volume indicated by the fourth preset operation ([0067] the generated output can include lowering the volume of the media playback based on the automated assistant query of “lower the volume by 10”.).
Gross, Kang, are considered analogous art within user. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang, further in view of Gross2 to incorporate the teachings of Shin, because of the novel way to consider location of speaker as well as vocal content of speech to reduce potential negative factors, i.e. noise, present in captured utterances through volume control (Shin, [0003]).
Kang further discloses:
wherein the fourth preset operation comprises: any one of sliding on the virtual button, touching the virtual button, and pressing the physical buttons ([0081] According to an embodiment of the present invention, the first-type user input may be one of a single press on the button, a double press on the button, a triple press on the button, a single press-and-hold on the button, or a double press-and-hold on the button, [Wherein any button press distinct from the first through third preset operations tracks to a fourth preset operation, i.e. if the first-third operations represent one-three presses, the fourth preset operation is a double press-and-hold or any equivalent alternative]).
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gross in view of Kang, further in view of Gross2, further in view of Jeong (US-20200160837-A1).
Regarding claim 19, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 1.
Gross in view of Kang, further in view of Gross2 does not disclose:
wherein the one or more computer programs further comprise:
instructions for receiving, through the Bluetooth component, a language setting instruction from the smart mobile terminal, and setting a language of the user to a target language type indicated by the language setting instruction, so that the speech-to-text engine converts the first speech into the first text based on the target language type; and,
instructions for receiving, through the Bluetooth component, an auto- language configuration instruction from the smart mobile terminal, and activating the automatic language detection function in response to the auto-language configuration instruction, so that the speech-to-text engine converts the first speech into the first text based on a language type obtained by an automatic language detection.
Jeong discloses:
instructions for receiving, through the Bluetooth component ([In view of the previously disclosed bluetooth component of Gross]), a language setting instruction from the smart mobile terminal ([Fig. 6, 400], [0248] a user may request a language conversion with respect to voice information to a specific voice recognition server which processes a first language [In view of the smart mobile terminal, i.e. artificial intelligence system 400, receiving a request for conversion indicating the server decision is made based a language setting from the smart mobile terminal 400]); and,
instructions for receiving, through the Bluetooth component ([In view of the previously disclosed bluetooth component of Gross]), an auto- language configuration instruction from the smart mobile terminal ([Fig. 4, Request sent from smart mobile terminal 400 to third server], [0224] For instance, if a user utters “Please order coffee”, a language of voice information may be determined as English and Korean. In this case, the voice information may be transmitted to a third server specified to a mixed language processing [Detection of language based on input speech by a smart mobile terminal as disclosed in Fig. 4 indicates the language detection performed by the smart mobile terminal is automatic, and a configuration instruction is sent to a third server, wherein the third server/system could be located on the smart glasses as disclosed in Gross, i.e. data retrieved/recorded on a smartphone, e.g. smart mobile terminal, to be sent to the glasses (see [0065]) in view of the translation chat function of Gross, [0222]]).
Gross, Kang, Gross2, and Jeong are considered analogous art within user-device conversational interaction. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang, further in view of Gross2 to incorporate the teachings of Jeong, because of the novel way to handle and translate a plurality of languages using a plurality of servers for translation, improving translation accuracy (Jeong, [0182]).
Gross further discloses:
setting a language of the user to a target language type indicated by the language setting instruction, so that the speech-to-text engine converts the first speech into the first text based on the target language type ([0222] a translated-chat function, where the user can speak or type one language and have their message translated into speech or text in another language [In view of the requested language setting instruction of Jeong]); and,
activating the automatic language detection function in response to the auto-language configuration instruction, so that the speech-to-text engine converts the first speech into the first text based on a language type obtained by an automatic language detection ([0222] Enabling the translation function in a private chat will continue to translate your new messages in the that chat until toggled off manually [Continued translation until a manual toggle off indicates that the language configuration instruction is automatic, i.e. based on languages which would be determined through a configuration instruction as disclosed in Jeong, see mixed language voice information]).
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gross in view of Kang, further in view of Gross2, further in view of Marti et al. (US-20220246168-A1).
Regarding claim 20, Gross in view of Kang, further in view of Gross2 discloses: the smart glasses of claim 1.
Gross in view of Kang, further in view of Gross2 does not disclose:
wherein the one or more computer programs further comprise instructions for:
receiving, through the Bluetooth component, a playback speed control instruction from the smart mobile terminal, and playing the second speech at a rate indicated by the playback speed control instruction.
Marti discloses: wherein the one or more computer programs further comprise instructions for:
receiving, through the Bluetooth component ([In view of the bluetooth component of Gross]), a playback speed control instruction from the smart mobile terminal ([0033] a playback speed associated with a user input received via a user interface (e.g., 1.5×, 2×, or the like) [Wherein a user interface can represent the smart mobile terminals as disclosed in Gross]), and playing the second speech at a rate indicated by the playback speed control instruction ([0033] audio playback module 140 is configured to playback the audio buffer, the synthesized explanatory phrase, or the like at an increased playback speed [In view of the audio representing a second speech of Gross, wherein 1.5x, 2x, etc. can represent rates indicated through the user input]).
Gross, Kang, Gross2, and Marti are considered analogous art within user-device communication. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang, further in view of Gross2 to incorporate the teachings of Marti, because of the novel way to variably set a playback speed dependent on the user and their background, reducing the chance of users losing track of information in a current content stream when the area of content is less familiar to the listener (Marti, [0005]).
Claim(s) 39 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gross in view of Kang, further in view of Gross2, further in view of Lee, further in view of Shin, further in view of Marti.
Regarding claim 39, Gross in view of Kang, further in view of Gross2, further in view of Lee discloses: the method of claim 33.
Gross in view of Kang, further in view of Gross2, further in view of Lee does not disclose:
wherein the method further comprises:
in response to a fourth preset operation for controlling a volume performed by the user on the button, playing the second speech at a volume indicated by the fourth preset operation, and
in response to a playback speed control instruction received through the Bluetooth component from the smart mobile terminal, playing the second speech at a rate indicated by the playback speed control instruction.
Shin discloses:
in response to a fourth preset operation for controlling a volume performed by the user on the button, playing the second speech at a volume indicated by the fourth preset operation ([0036] The user 102 speaks the utterance 202 of “ASSISTANT, TURN VOLUME DOWN” to control the volume of the media playback on the television… [0064] a user can invoke an automated assistant by making an invocation gesture, pushing one or more invocation buttons (physical and/or virtual) [Invoking an assistant via button, wherein the command is to reduce volume indicates a fourth operation performed by the user on the button]), playing the second speech at a volume indicated by the fourth preset operation ([0067] the generated output can include lowering the volume of the media playback based on the automated assistant query of “lower the volume by 10”.).
Gross, Kang, Gross2, Lee, and Shin are considered analogous art within user-device conversational interaction. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang, further in view of Gross2, further in view of Lee to incorporate the teachings of Shin, because of the novel way to consider location of speaker as well as vocal content of speech to reduce potential negative factors, i.e. noise, present in captured utterances (Shin, [0003]).
Kang further discloses:
wherein the fourth preset operation comprises: any one of sliding on the virtual button, touching the virtual button, and pressing the physical buttons ([0081] According to an embodiment of the present invention, the first-type user input may be one of a single press on the button, a double press on the button, a triple press on the button, a single press-and-hold on the button, or a double press-and-hold on the button, [Wherein any button press distinct from the first through third preset operations tracks to a fourth preset operation, i.e. if the first-third operations represent one-three presses, the fourth preset operation is a double press-and-hold or any equivalent alternative. The examiner would like to note that due to the disjunctive nature of the claim, not all elements require a mapping. Further, consider the virtual button of Lee]).
Gross in view of Kang, further in view of Gross2, further in view of Lee, further in view of Shin does not disclose:
in response to a playback speed control instruction received through the Bluetooth component from the smart mobile terminal, playing the second speech at a rate indicated by the playback speed control instruction.
Marti discloses:
in response to a playback speed control instruction received through the Bluetooth component from the smart mobile terminal ([0033] a playback speed associated with a user input received via a user interface (e.g., 1.5×, 2×, or the like) [Wherein a user interface can represent the smart mobile terminals as disclosed in Gross, further in view of the previously disclosed Bluetooth component of Gross]), playing the second speech at a rate indicated by the playback speed control instruction ([0033] audio playback module 140 is configured to playback the audio buffer, the synthesized explanatory phrase, or the like at an increased playback speed [In view of the audio representing a second speech of Gross, wherein 1.5x, 2x, etc. can represent rates indicated through the user input]).
Gross, Kang, Gross2, Lee, Shin, and Marti are considered analogous art within user-device communication. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Kang, further in view of Gross2, further in view of Lee, further in view of Shin to incorporate the teachings of Marti, because of the novel way to variably set a playback speed dependent on the user and their background, reducing the chance of users losing track of information in a current content stream when the area of content is less familiar to the listener (Marti, [0005]).
Claim(s) 21-23, 25-27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gross in view of Gross et al. (US-20240321279-A1), hereinafter Gross2.
Regarding claim 21, Gross discloses: a smart glasses control system based on generative artificial intelligence large language models (GAILLMs) ([0042] smartglasses that can be used in accordance with the methods and systems for using artificial intelligence to control mobile devices used for presentation and display of mobile device tasks [In view of the context of Gross being for voice conversation/response generation using smart glasses (see abstract), there is an indication that the artificial intelligence is based on generating dialog responses, i.e. presentation of language, indicating the artificial intelligence is a generative artificial intelligence large language model, as would be required for response generation based on input speech]), comprising:
smart glasses ([Fig. 2], [0042] FIG. 2 illustrates smartglasses);
a smart mobile terminal ([Fig. 1, Smart Phone 400 connected to smart glasses 200], [A smartphone tracks to a smart mobile terminal, see Fig. 3 of instant app]).
wherein the smart glasses comprise:
a microphone ([Fig. 2, microphone 225]);
a speaker ([Fig. 2, bone conduction speakers 235]); and,
a Bluetooth component ([Fig. 2], [0061] connectivity module and hardware 260 may use WiFi, Bluetooth); and,
wherein the smart glasses are configured for activating a chat function of the smart glasses in response to a first control instruction for activating the chat function ([0104] the microphone 225 of smartglasses 200 and smartglasses AI interface 280 receive an oral request or command audibly proceeded by language, such as “tell me” or “show me,” that is communicated to glasses' AI interface 280… [0197] an embodiment Vyrb implementation may permit a user to create a verbal “hotkey” configured to trigger on-demand activation of a customizable range of data retrieval and user action patterns [“tell me” or “show me” represent first control instructions, i.e. verbal hotkeys, to activate a chat function, i.e. for data retrieval operations, in view of [0037] keyword activation of the instant application.]),
obtaining a first speech of a user through the microphone ([Fig. 7, 710]), and sending the first speech to the smart mobile terminal through the Bluetooth component ([Fig. 8, Voice Assistant 835 and/or mobile device 870 represent smart mobile terminals, wherein the information sent is clearly speech, i.e. voice command 825], [Fig. 2, connection between smart glasses 200 and smart phone, i.e. smart mobile terminal, 400 indicating the connection can be bluetooth in view of the previously disclosed bluetooth component of Gross]), wherein the first speech comprises a question asked by the user ([0104] Examples of questions can include questions designed to perform tasks, open applications, including map or navigation, music, wallet, application store, calculator, watch, stopwatch, banking or money transfer, video, games, stocks, weather, etc., [Taking the received oral request by the smartglasses of step 710 in view of the question asking of [0104] indicates the oral requests are first speeches comprising questions]);
Gross does not disclose:
a model server;
the smart mobile terminal is configured for converting the first speech into a first text, and sending the first text to the model server; and,
the model server is configured for obtaining a second text through the GAILLMs based on the first text, and sending the second text to the smart mobile terminal, wherein the second text comprises a reply corresponding to the question.
Gross2 discloses:
a model server ([0034] Chatbot platform 335 may be an AI-powered contextual chatbot platform that may utilize machine learning and/or artificial intelligence for example as described herein. Chatbot platform 335 may utilize natural language processing, pattern matching, sequence to sequence models, recurrent neural networks, Naïve Bayes, long short term memory, and/or any other suitable processing, artificial intelligence types, and/or types of machine learning operations. Chatbot platform 335 (e.g., chatbot application 328 including chatbot platform 335) may communicate with and/or include one or more text-based or text-only content generators such as chatbots… Chatbot platform 335 may be fine-tuned using reinforcement learning, transfer learning, proximal policy optimization, and/or supervised learning [A chatbot platform consisting of chatbots, i.e. GAILLMs, indicates the chatbot platform is equivalent to a model server, fine-tuning the chatbot platform is equivalent to configuring the chatbot GAILLMs]).
the smart mobile terminal is configured for converting the first speech into a first text ([0033] Audio assistant 330 may be a voice user interface that operates using…speech-to-text, [An audio assistant tracks to a smart mobile terminal, see definition of audio assistant earlier in [0033] tracking to a hardware/software agent, i.e. a device (smart mobile terminal) containing an audio assistant]), and sending the first text to the model server ([0046] chatbot platform 335 may receive the text data and/or signals [A chatbot platform, i.e. consisting of a plurality of GAILLMs in the form of chatbots, tracks to a model server, wherein a model server is that used to generate responses to input text, see [0010] of the instant app]); and,
the model server is configured for obtaining a second text through the GAILLMs based on the first text ([0047] At step 460, chatbot platform 335 may select one or more chatbots for responding to the received text data and/or signals of the one or more user queries and/or prioritize responses from the one or more chatbots [Selecting chatbots to respond to input text tracks to a chatbot selection configuration from within the platform for obtaining a second text, i.e. a response, wherein the chatbots are comprised of GAILLMs as previously disclosed]), and sending the second text to the smart mobile terminal ([Fig. 4, 355], [0028] Text data and/or signals may be transferred via the exemplary disclosed communication techniques from chatbot platform 335 to audio assistant 330 as shown at 355 [Wherein an audio assistant tracks to a smart mobile terminal, i.e. the audio assistant can be contained within the smart mobile terminal]), wherein the second text comprises a reply corresponding to the question ([0047] utilizing a plurality of chatbots to provide a response to each user inquiry [In view of the input inquiries, i.e. questions, of Gross, see [0104] of Gross]).
Gross and Gross2 are considered analogous art within smart device chatbot communication. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross to incorporate the teachings of Gross2, because of the novel way to introduce an intermediate audio assistant before sending received audio to a chatbot, allowing for a voice interface for more powerful text-based chatbots traditionally lacking voice interfaces (Gross2, [0005]).
Gross further discloses:
the smart mobile terminal is further configured for converting the second text into a second speech, and sending the second speech to the smart glasses ([0134] For example, a voice command created in this way might look like: “Siri, Vyrb CNN”. Then the user's selected area of CNN headlines will be found by the app, then vocalized to the user through the voice assistant's native text-to-speech engine [In view of the smart mobile terminal, i.e. smart phone and/or voice assistant of Gross, vocalizing response, i.e. second, text, in view of the speakers of the smart glasses of Gross, indicates the vocalization can be sent to the speakers on the smart glasses]);
the smart glasses are further configured for receiving the second speech through the Bluetooth component ([Fig. 8, audio response 895], [In view of the previously disclosed Bluetooth component of Gross. A signal being sent to a user 805, wherein the user can be wearing smart glasses with Bluetooth capability indicates smart glasses configured for receiving second speech]), and playing the second speech through the speaker ([0077] Smartglasses 200 can chime when a text is received, and vocalize the text, [Vocalizing text, in view of the speaker of the smart glasses of Gross, indicates speech played through speakers]); and,
wherein the smart glasses further comprise a wireless communication component ([In view of the previously disclosed Bluetooth component of Gross]), and are further configured for:
obtaining device data of the smart glasses through a data acquisition device on the smart glasses ([Fig. 2, Camera 265], [In view of [0166] of the instant app which defined a data acquisition device to be a camera], [0116] smartglasses 200 or smartwatch 300 contains an embedded camera capable of transmitting visual information to a connected smartphone 400 AI interface 460, then the smartglasses mobile application 210 could assist the user in rapidly reading a QR code containing a cryptocurrency public address, [As the data acquisition device is a sensor, it is reasonable to assume that device data gathered by the data acquisition device is sensor data. Therefore, images gathered by the smart glasses track to device data]),
wherein the data acquisition device on the smart glasses comprises at least one component of: a position sensor, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera and a pedometer ([0116] smartglasses 200 or smartwatch 300 contains an embedded camera capable of transmitting visual information to a connected smartphone 400 AI interface 460);
converting the first speech into the first text using a built-in speech-to-text engine ([0237] 1. Speech to text functionality).
Gross2 further discloses:
sending, through the wireless communication component, the first text to the model server ([0028] The text data and/or signals may be transferred via the exemplary disclosed communication techniques from audio assistant 330 to chatbot platform 335 [Wherein a chatbot platform represents a model server]).
Gross further discloses:
the model server is further configured for:
determining whether the reply needs to be based on data collected by the data acquisition device on the smart glasses and/or a data acquisition device on the smart mobile terminal through the GAILLMs according to the first text from the smart glasses ([Fig. 7, S720 “Access lookup tables and present any information related to the requested device…for display and presentation”], [Fig. 7, S730 “Smartglasses receive optional override or switch command or request to change the paired or connected device used for presentation or display”], [Choosing a presentation device based on the type of task at a mobile device indicates a determination that, based on the first text, the connected device is chosen, wherein each device will necessarily have data to be collected (as disclosed in S720). If the task/command is presentation, then the requested device information is describing how the required data should be presented. The mobile device is the model server]);
when the reply corresponding to the question needs to be based on the data, generating description information of required data ([Considering the previously cited S720 of Gross, accessing information related to a requested device by a mobile application, i.e. model server, wherein the request comes from smart glasses, indicates a determination that the accessed description information is relevant to presenting the requested data, i.e. reply, in order to appropriately present the requested information in the form of a reply]) and
selecting to sending the description information to the smart glasses or the smart mobile terminal ([Fig. 7, S730], [As disclosed in S730, the smartglasses receiving the request to change presentation device will necessarily be comprised of description information to be read by the smart glasses which describe the override action in a way that the glasses can understand])); and
when the reply corresponding to the question does not need to be based on the data, obtaining the second text through the GAILLMs based on the first text ([As previously disclosed in [0115], the ability of the smartglasses to return answers to a user based on an input indicates a reply generated through GAILLMs (as previously disclosed) without the need for additional information about where the information is to be presented. Consider a situation in which the optional override is not received by the smart glasses]).
Gross2 further discloses:
the smart glasses are further configured for:
when the description information is received, determining the required data from the device data of the smart glasses according to the description information ([0047] At step 460, chatbot platform 335 may select one or more chatbots for responding to the received text data and/or signals of the one or more user queries and/or prioritize responses from the one or more chatbots. For example, chatbot platform 335 may select, prioritize, combine, analyze, integrate, process, and/or utilize response text data and/or signals from any desired number of suitable chatbots, [Combining results into one final output based on collective responses indicates the final output to be representing a combination of required data from the device data, i.e. response data, according to description information, i.e. which chatbots are to be used for generating the response based on a received transcription]), and
sending, through the wireless communication component ([In view of the previously disclosed Bluetooth component of Gross]), the required data to the model server ([There is no limitation preventing the determination of required device data according to description information from being performed in the chatbots, i.e. model servers, of Gross2. As such, an additional step to send the required data the model server is a step which will never occur considering a situation in which the required data is already gathered in the model server. Further, it appears to the examiner that the model server is responsible for generating the description information to be used to determine the required data. Therefore, this sending step appears to be “sending” information from a starting location to an ending location which is the same, i.e. the model server. This step does not appear to occur. As such, it does not require a mapping]);
the model server is further configured for:
inputting the required data to the GAILLMs to obtain the second text through the GAILLMs based on the required data and the first text from the smart glasses ([Fig. 11, “Finalize text and/or image response data/signals 475”], [Using chatbots, i.e. model server, to finalize a response, wherein that response is consisting of text and/or image data/signals indicates the response, i.e. second text, to be based on required data, i.e. image information, and first text]).
sending the second text to the smart glasses ([Fig. 4, 355], [0028] Text data and/or signals may be transferred via the exemplary disclosed communication techniques from chatbot platform 335 to audio assistant 330 as shown at 355 [Wherein an audio assistant tracks to a smart glasses, i.e. the audio assistant can be contained within the smart glasses]);
Gross further discloses:
wherein the GAILLMs associate the required data with the first text ([0059] mobile application 401 may include a device level voice assistant manager process, [Wherein the required data is the device data as previously disclosed. Further, as previously disclosed, sending a request or command to a mobile application, wherein that mobile application is used for controlling presentation of tasks (S715) indicates a situation in which the GAILLMs, i.e. the voice assistant on mobile application, associate the device data with the first text through the sending operation. As the command is sent to the mobile application to be answered/presented (S715), device data is gathered (S720) in relation to the request, indicating a situation in which the mobile application (containing the voice assistant) associates the device, i.e. required, data with the received first text]), and
obtain the second text using the required data and the first text ([Fig. 7, S740 “smartglasses communicate the override…controlling the paired or connected devices used for display or presentation of tasks or applications”[As previously disclosed in [0140] of Gross, vocalizing a question/comment asked by the voice assistant indicates a received text, i.e. prompt, to be vocalized, wherein the vocalization is performed based on the device connected to display and/or presentation of tasks (S740 of Fig. 7) indicating the second text to be received by the user using the required device data and first text]).
Gross2 further discloses:
the smart glasses are further configured for converting the second text sent by the model server into the second speech using a built-in text-to-speech engine ([Fig. 11, “Convert Text to Voice Data/Signals”], [0054] At step 495, audio assistant 330 may convert the finalized text data and/or signals to voice data and/or signals and transfer the voice data and/or signals to the exemplary disclosed device (e.g., glasses 310).
Regarding claim 22, Gross in view of Gross2 discloses: the system of claim 21.
Gross2 further discloses:
wherein the system further comprises a conversion server ([0046] audio assistant 330 may receive voice data and/or signals and convert the voice data and/or signals to text data [An audio assistant tracks to a conversion server, i.e. that which converts audio to text]);
the smart mobile terminal is further configured for sending the first speech to the conversion server ([0028] Audio or sound data and/or signals (e.g., of the speech or voice query provided at 340) may be transferred via the exemplary disclosed communication techniques from the exemplary disclosed device (e.g., glasses 310, headphones 315, and/or user device 320) to audio assistant 330 as shown at 345 [A smart mobile terminal tracks to an exemplary disclosed device, in view of the smart mobile terminal devices of Gross, the audio assistant responsible for converting the speech into text indicates it is a conversion server]);
the conversion server is configured for converting the first speech into the first text through a speech-to-text engine ([0033] Audio assistant 330 may be a voice user interface that operates using…speech-to-text [Wherein an audio assistant tracks to a conversion server, in view of the audio assistant speech-to-text capability disclosed in [0033]]), and sending the first text to the model server ([0028] The text data and/or signals may be transferred via the exemplary disclosed communication techniques from audio assistant 330 to chatbot platform 335 [Wherein a chatbot platform represents a model server]);
the model server is further configured for obtaining the second text through the GAILLMs based on the first text from the conversion server ([Fig. 4, 335], [0047] Predetermined algorithms, machine learning operations, user input and/or criteria (e.g., user input regarding desired length of responses and desired chatbots), and/or any other suitable criteria may be used for selecting, prioritizing, analyzing, integrating, and/or utilizing a plurality of chatbots to provide a response to each user inquiry [Using chatbots, i.e. model servers, to obtain responses, wherein those chatbots consist of GAILLMs, indicates the chatbot obtains a response, i.e. second text, before sending back as a response based on an input audio/text 345 from conversion server, i.e. voice assistant, 330]), and sending the second text to the conversion server ([Fig. 4, 355], [0028] Text data and/or signals may be transferred via the exemplary disclosed communication techniques from chatbot platform 335 to audio assistant 330 as shown at 355 [Wherein an audio assistant represents a conversion server]); and
the conversion server is further configured for converting the second text into the second speech through a text-to-speech engine ([Fig. 4, 330], [0028] Audio assistant 330 may convert the text data and/or signals to audio or sound data and/or signals), and sending the second speech to the smart mobile terminal ([Fig. 4, 360], [0028] The audio or sound data and/or signals may be transferred via the exemplary disclosed communication techniques from audio assistant 330 to the exemplary disclosed device (e.g., glasses 310, headphones 315, and/or user device 320) as shown at 360 [Wherein the user device represents a smart mobile terminal]).
Regarding claim 23, Gross in view of Gross2 discloses: the system of claim 21.
Gross2 further discloses:
wherein the system further comprises a conversion server ([Fig. 4, 330], [0028] Audio assistant 330 may convert the audio or sound data and/or signals to text data and/or signals [An audio assistant tracks to a conversion server]);
the smart glasses are further configured for sending the first speech to the conversion server ([0028] user 305 may provide a voice (e.g., speech) query at 340. Audio or sound data and/or signals (e.g., of the speech or voice query provided at 340) may be transferred via the exemplary disclosed communication techniques from the exemplary disclosed device (e.g., glasses 310, headphones 315, and/or user device 320) to audio assistant 330 as shown at 345);
the conversion server is configured for converting the first speech into the first text through a speech-to-text engine ([0033] Audio assistant 330 may be a voice user interface that operates using…speech-to-text), and
sending the first text to the model server ([0028] The text data and/or signals may be transferred via the exemplary disclosed communication techniques from audio assistant 330 to chatbot platform 335 as shown at 350 [Wherein the chatbot platform represents the model server and an audio assistant represents a smart mobile terminal in view of the smart mobile terminals of Gross]);
the model server is further configured for obtaining the second text through the GAILLMs based on the first text from the conversion server ([Fig. 4, 335], [0047] Predetermined algorithms, machine learning operations, user input and/or criteria (e.g., user input regarding desired length of responses and desired chatbots), and/or any other suitable criteria may be used for selecting, prioritizing, analyzing, integrating, and/or utilizing a plurality of chatbots to provide a response to each user inquiry [Using chatbots, i.e. model servers, to obtain responses, wherein those chatbots consist of GAILLMs, indicates the chatbot obtains a response, i.e. second text, before sending back as a response based on an input audio/text 345]), and
sending the second text to the conversion server ([Fig. 4, 355], [0056] The exemplary disclosed application may respond with a text-to-speech vocalization of a textual response of chatbot platform 335); and
the conversion server is further configured for converting the second text into the second speech through a text-to-speech engine ([0033] Audio assistant 330 may be a voice user interface that operates using natural language generation, text-to-speech), and
sending the second speech to the smart glasses ([Fig. 4, 360], [0032] For example, glasses 310 may include a microphone and/or a speaker for receiving a speech or voice query at 340 and emitting a sound (e.g., voice or speech) response at 365).
Regarding claim 25, Gross in view of Gross2 discloses: the system of claim 21.
Gross2 further discloses:
wherein the system further comprises a chat history sever ([0057] the exemplary disclosed graphical user interface (e.g., of the exemplary disclosed application of the exemplary disclosed modules) may include a history that may allow users to access their previous queries and responses [Allowing users to access previous chat entries indicates that they are stored on a server, wherein Gross2 discloses their system being comprised of numerous servers ([0065])]);
the smart mobile terminal is further configured for generating chat logs based on data sent by the smart glasses during a chat ([Fig. 6], [A smart mobile terminal, i.e. smartphone, containing chat logs based on previous queries, wherein those queries can be gathered and sent from smart glasses as disclosed in Gross and Gross2]), associating the chat logs with a login account of the user ([Fig. 7, automatic email export], [0056] The exemplary disclosed application may include a settings panel allowing a user to login… [0050] conduct desired (e.g., extensive) research using voice queries, which may for example be shared automatically with the user's account [Automatic email export to a predetermined email, in view of the login feature of Gross2, indicates that the logins are used to determine the automatic email export location, further indicating chat logs associated with user accounts]), and storing the chat logs in the smart mobile terminal or the chat history server ([0057] The exemplary disclosed application may allow the user to access their recent queries (e.g., last ten queries) and/or responses via a History tab in a menu drawer of the application [Wherein the application is located on a smart mobile terminal, i.e. smartphone (see Fig. 6), indicating the chat logs are also located on the smart mobile terminal, wherein smartphones traditionally consist of memory for storing information]);
the smart mobile terminal is further configured for sending the first text and the chat logs to the model server ([0046] At step 455 and as shown at 350 of FIG. 4, chatbot platform 335 may receive the text data and/or signals…[0047] For example, chatbot platform 335 may select, prioritize, combine, analyze, integrate, process, and/or utilize response text data and/or signals from any desired number of suitable chatbots [Chatbot responses track to a type of text data, in view of the previously disclosed chat history of Gross2, wherein Gross2 discloses combining/analyzing/utilizing multiple responses from chatbots, indicating previous historical responses, i.e. the stored logs (text data of previous responses), could be sent to the chatbots along with the first text]);
the model server is further configured for obtaining the second text through the GAILLMs based on the first text and the chat logs ([0058] The chatbot application may be configured to generate a second text data or signal associated with a response to the at least one voice query based on operating or communicating with the at least one chatbot [Wherein the chatbot platform/application represents a model server, the chatbots are GAILLMs as previously disclosed, and the response text can be generated based on a first text and chat logs as previously disclosed]);
the smart mobile terminal is further configured for in response to a query operation of the user, obtaining target chat logs corresponding to the query operation ([Fig. 6, History], [Presenting chat log history on a smartphone, i.e. smart mobile terminal, wherein the smart mobile terminal is not required to display chat history (see Fig. 5), indicates that chat logs are obtained in response to a query operation, i.e. “Can you show me our past conversations?”]), and exporting the target chat logs based on a preset export manner ([Fig. 7], [Automatic email/SMS exports track to preset export manners]).
Regarding claim 26, Gross in view of Gross2 discloses: the system of claim 25.
Gross further discloses:
wherein the preset export manner comprises: exporting the target chat logs to a preset social media platform ([0202] Auto speech-to-text reposting to other social media platforms [In view of the text containing chat logs of Gross2]), or exporting the target chat logs to a designated device ([The examiner would like to note that due to the disjunctive nature of the claim, this element does not require a mapping. Further, refer to the previously disclosed smartphone, i.e. designated device, of Gross2 containing chat logs indicating those logs are exported to the device in view of the smart glasses system of Gross directly connecting smartphone to smart glasses]).
Regarding claim 27, Gross in view of Gross2 discloses: the system of claim 21.
Gross2 further discloses:
wherein the smart glasses are further configured for sending the device data of the smart glasses to the smart mobile terminal ([0028] Audio or sound data and/or signals (e.g., of the speech or voice query provided at 340) may be transferred via the exemplary disclosed communication techniques from the exemplary disclosed device (e.g., glasses 310, headphones 315, and/or user device 320) to audio assistant 330 as shown at 345, [Data picked up by a microphone on the smart glasses tracks to device data of the smart glasses]);
the smart mobile terminal is further configured for obtaining device and environmental data of the smart mobile terminal through the data acquisition device on the smart mobile terminal before sending the first text to the model server ([Audio data received by a microphone in an environment on a device, see above claim element mapping, indicates the received audio signals to be either device and/or environmental data of the smart mobile terminal, i.e. the device receiving the sounds in the environment]);
wherein the data acquisition device on the smart mobile terminal comprises at least one component of: the position sensor, the inertial measurement unit sensor, the temperature sensor, the proximity sensor, the humidity sensor, the electronic compass, the timer, the camera and the pedometer ([0029] User device 320 may include a camera and a microphone (e.g., and/or a speaker) for receiving, recording, emitting, and/or playing sound… User device 320 may include any suitable sensors for use with system 300 such as, for example, a location sensor [A camera, in view of the camera to send images of Gross, indicates sending sensing data, i.e. images, to the smart mobile terminal, i.e. audio assistant 330 of Gross2, as previously disclosed]); and
the smart mobile terminal is further configured for:
when the smart mobile terminal receives the description information, determining the required data from the device data of the smart glasses and the device and environmental data of the smart mobile terminal according to the description information ([0047] At step 460, chatbot platform 335 may select one or more chatbots for responding to the received text data and/or signals of the one or more user queries and/or prioritize responses from the one or more chatbots. For example, chatbot platform 335 may select, prioritize, combine, analyze, integrate, process, and/or utilize response text data and/or signals from any desired number of suitable chatbots, [Combining results into one final output based on collective responses indicates the final output to be representing a combination of required data from the device/environmental data, i.e. response data, according to description information, i.e. which chatbots are to be used for generating the response based on a received transcription]), and
sending the required data to the model server ([0028] The text data and/or signals may be transferred via the exemplary disclosed communication techniques from audio assistant 330 to chatbot platform 335 as shown at 350 [Wherein the chatbot platform represents the model server. Further, in the context of response generation, received input queries are required data to be answered]).
Claim(s) 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gross in view of Gross2, further in view of Marti et al. (US-20220246168-A1).
Regarding claim 24, Gross in view of Gross2 discloses: the system of claim 21.
Gross in view of Gross2 does not disclose:
wherein the smart mobile terminal is further configured for in response to an operation for adjusting a playback speed performed by the user on a user interface of the smart mobile terminal, adjusting the playback speed of the second speech to a target speed indicated by the operation, and sending the second speech with the target speed to the smart glasses.
Marti discloses:
wherein the smart mobile terminal is further configured for in response to an operation for adjusting a playback speed performed by the user on a user interface of the smart mobile terminal ([0033] a playback speed associated with a user input received via a user interface [Wherein a smart mobile terminal can represent/contain a user interface, i.e. a smartphone]), adjusting the playback speed of the second speech to a target speed indicated by the operation ([0033] audio playback module 140 is configured to playback the audio buffer, the synthesized explanatory phrase, or the like at an increased playback speed (e.g., 1.5×, 2×, or the like) until the audio signal catches up with the media content stream 150 [Wherein 1.5x, 2x, etc. could all represent adjustments to be set through the user interface, indicating adjustment based on a target speed indicated by the operation of user input]).
Gross, Gross2, and Marti are considered analogous art within user-device communication. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Gross in view of Gross2 to incorporate the teachings of Marti, because of the novel way to variably set a playback speed dependent on the user and their background, reducing the chance of users losing track of information in a current content stream when the area of content is less familiar to the listener (Marti, [0005]).
Gross further discloses:
sending the second speech with the target speed to the smart glasses ([0193] The voice assistant will then interface with the Vyrb app to collect the designated data in real time, and use the voice assistant's text-to-speech engine to vocalize it for the user. [Retrieving data, wherein that data represents text to be vocalized into speech on smart glasses, see [0075] relaying output to smart glasses, in view of the audio playback based on speed of Marti indicates sending a second speech with a target speed to smart glasses]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Harris et al. (US-20180157336-A1) discloses “Systems and methods described herein are for transmitting a command to a remote system. A processing system determines the identity of the user based on the unique identifier and the biometric information. Thereafter, a sensor detects a gesture performed by the user. The sensor is configured to detect the gesture performed by the user when the user is located within the detectable range of the wireless antenna. The processing system determines an action associated with the detected gesture based on the identity of the user and sends a command to a remote computer system to cause it to perform the action associated with the detected gesture.” (abstract). Specifically, Harris discloses V-glasses which can process both gesture and vocal commands with the aide of a smart mobile device. See entire document.
Gross et al. (US-10908419-B2) discloses “Smartglasses, methods, and systems to control presentation and display of information related to mobile device tasks and applications to the smartglasses or a paired or connected smartwatch or smartphone, based on oral commands or requests, and enhance presentation and display augmented reality information, based on default settings of a device mobile application controlling display of information related to tasks and applications and oral override or switch commands. The smartglasses comprise corrective lenses, printed circuit board, battery, camera, microphone bone conducting speakers, connectivity module, artificial intelligence interface and run a smartglasses mobile application, the mobile application and artificial intelligence interface and are configured and programmed to communicate the requested tasks and applications to a mobile device application controlling display and presentation of information related to the requested tasks or applications, which can reside in the smartglasses, smartwatch, or smartphone.” (abstract). This is work by the same inventors as those cited as ‘Gross’ and ‘Gross2’ above. See entire document.
Park et al. (US-20240420491-A1) discloses “Network infrastructure for user-specific generative intelligence. Providing user-specific context to a generically trained LLM introduces a variety of complications (privacy, resource utilization, training costs, etc.). Various aspects of the present disclosure provide novel user-specific data structures, privacy and access control, layers of data, and session management, within a network infrastructure for generative intelligence. For example, user-specific embedding vectors may be used to provide user context to a generically trained foundation model. In some variants, edge devices capture multiple modalities of user context (images, audio; not just text). Privacy and access control mechanisms also allow a user to control information that is captured and sent to the foundation model. Session management further decouples a user's conversational state from the foundation model's session state. These concepts and others may be used to emulate e.g., a chatbot based virtual assistant that responds based on user context.” (abstract). Park does not beat the EFD of the instant application and is being cited to demonstrate the state of the art.
Huang et al. (US-20260087801-A1) discloses “A method for conversational interactions with an artificially intelligent (AI) assistant at a pair of smart glasses is described. The method includes, invoking an AI assistant at the pair of smart glasses without providing a query, wherein the artificially intelligent assistant has access to camera data provided by a camera of the pair of smart glasses. The method further includes, in response to invoking the artificially intelligent assistant at the pair of smart glasses, (i) determining, based in part on the camera data, that the AI assistant should provide assistance to a user related to an object present within the camera data, and (ii) in response to the determining, providing, via an output modality of the pair of smart glasses, a communication to the user that includes the assistance to the user related to the object present within the camera data.” (abstract). Huang does not beat the EFD of the instant application and is being cited to demonstrate the state of the art.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THEODORE JOHN WITHEY whose telephone number is (703)756-1754. The examiner can normally be reached Monday - Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571) 272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THEODORE WITHEY/Examiner, Art Unit 2655
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655
Prosecution Insights