DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
All objections/rejections not mentioned in this Office Action have been withdrawn by the Examiner.
Examiner Note
Applicant’s amendments and comments regarding the Office Action dated 07 October 2025 are greatly appreciated.
It is noted that applicant presented arguments directed to the separation of the variable recognizer and the intent recognizer, specifically indicating that “claim 1 further requires two distinct components: a variable recognizer and an intent recognizer.” Though the argument that distinct components are currently required is not persuasive, it is believed that limitations directed to presenting these components as distinct would be promising for claims 1 and 11 in overcoming the Fuegen reference.
Examiner attempted to contact applicant’s representative, and left messages at least on 29 May 2026 and 3 June 2026, in an attempt to discuss further amendment which would affirmatively incorporate known distinctions, such as separate training of separate neural networks described in the Instant Application at paragraph [0084]. As of the date of this Office Action, no response has been received. Further, because appropriate amendments are not currently available for search and consideration, it could not be determined whether such amendments acceptable to the applicant would overcome the cited references.
Applicant is invited to contact the examiner, at their earliest convenience, to discuss possible amendments directed to the above described subject matter.
Response to Amendments
Applicant’s amendment filed on 06 March 2026 has been entered.
In view of the amendment to the claim(s), the amendment of claim(s) 1 and 11 have been acknowledged and entered.
In view of the amendment to claim(s) 1 and 11, the rejection of claims 1-20 under 35 U.S.C. §103 is maintained as modified in response to the amendments for the reasons discussed below.
In light of the amended claims, new grounds for rejection under 35 U.S.C. §112 are provided in the action below.
Response to Arguments
Applicant’s arguments regarding the prior art rejections under 35 U.S.C. §103, see pages 6-11 of the Response to Non-Final Office Action dated 07 October 2025, which was received on 06 March 2026 (hereinafter Response and Office Action, respectively), have been fully considered.
With respect to the rejection(s) of claim(s) 1 under 35 U.S.C. §103 in light of Fuegen (U.S. Pat. No. 1107462, hereinafter Fuegen), Non-Patent Literature to Liu (Liu, B. and Lane, I., 2016. Joint online spoken language understanding and language modeling with recurrent neural networks. arXiv preprint arXiv:1609.01462.(2016), hereinafter Liu), and Avijeet (U.S. Pat. App. Pub. No. 2022/0130378, hereinafter Avijeet) applicant argues that Fuegen, Liu, and Avijeet fail to teach or suggest (1) “a variable recognizer that recognizes one or more enumerated variable values directly from the speech audio ... wherein the variable recognizer computes a variable value probability, for each of the one or more enumerated variable values ... and outputs the enumerated variable value ... with the highest variable value probability,” and (2) “variable recognizer that recognizes one or more enumerated variable values directly from the speech audio features, wherein the one or more enumerated variable values are drawn from a predefined list of candidate variable values associated with a domain.” Applicant’s arguments are not persuasive.
In presenting the first argument, applicant relies on the explanation “the claimed invention requires a separate recognizer that identifies enumerated variable values from the speech audio and computes a probability for each candidate value,” further explaining that Fuegen fails to disclose such a component. (Response, pg. 6). In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “a separate recognizer that identifies enumerated variable values from the speech audio and computes a probability for each candidate value”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Respectfully, claim 1 discloses “variable recognizer that recognizes one or more enumerated variable values directly from the speech audio features.” Applicant does not claim a “separate recognizer” and provides no indication that the variable recognizer and the intent recognizer are necessarily separate (the components are described functionally and the functions could be performed by a single device/system). It is agreed that Fuegen describes “an end-to-end spoken language understanding network.” However, such a distinction is not relevant based on the claims as currently presented.
Further, applicant does not claim “identifying enumerated variable values.” (emphasis added). Both prior to and after the amendments, applicant claims “recogniz[ing] one or more enumerated variable values directly from the…” speech audio/speech audio features. Recognition is understood in the art as identifying something actually detected (e.g., the identification of words which define the detected speech features of a spoken instruction as “speech recognition”, the identification of a specific voice which defines the detected combination of pitch, tone, etc. in a vocalization as “voice recognition,” identification of a specific intent which defines the conveyed meaning of the detected combination of words in “intent recognition,” etc.). As such, identification is broader than recognition, as identification is not tethered to the detection alone. Applicant is invited to amend the claims during normal prosecution such that the claims reflect the desired limitations and said limitations can be substantively examined.
Regarding the argument that the cited references fail to teach or suggest “a variable recognizer that recognizes one or more enumerated variable values directly from the speech audio ... wherein the variable recognizer computes a variable value probability, for each of the one or more enumerated variable values ... and outputs the enumerated variable value ... with the highest variable value probability,” this argument is not persuasive.
Regarding the arguments above, applicant's arguments do not comply with 37 CFR 1.111(c) because they do not clearly point out the patentable novelty which he or she thinks the claims present in view of the state of the art disclosed by the references cited or the objections made. Further, they do not show how the amendments avoid such references or objections. Applicants argument assert what the applicant understands to exist in the cited reference. Then, in a disconnected fashion, recites an element of the claim or an interpretation thereof. Followed by a statement, without connection, that the element of the claim is not disclosed. For example, “The Office Action appears to rely on passages in Fuegen describing intent recognition and optional slot filling to meet this limitation. However, a careful review of Fuegen demonstrates that Fuegen does not disclose or suggest a variable recognizer as recited in claim 1.”
Respectfully, nothing has been demonstrated in this statement. As presented in the Office Action, Fuegen teaches “the system applies ‘the network to extract semantic meaning directly from the raw audio or audio features’ which ‘may involve performing domain or intent classification, or performing a slot-filling task {recognizing one or more enumerated variable values...}’ and ‘the system may apply the network to extract semantic meaning directly from the raw audio…without generating an intermediate textual representation of the audio {...directly from the speech audio}’” (Fuegen, Col. 5, lines 48- col. 6, line 4). Without equivocation, examiner has mapped the variable recognizer to the system utilizing the network, and performing the above described functions. (Id.) Respectfully, the assertions from the applicant that “Fuegen does not disclose or suggest a variable recognizer”, “Fuegen does not disclose a variable recognizer that recognizes enumerated variable values,” or “This interpretation is not supported by Fuegen,” provide no information as to what the applicant believes is deficient about this mapping. Applicant further presents a citation from Fuegen to assert that Fuegen discloses “identifying slot categories or parameters in an utterance, rather than recognizing and selecting among enumerated candidate variable values as required by the claims.” First, Fuegen is not limited to applicant’s recitation, as they specifically and unequivocally disclose both the determination of a list of parameters and a determination of slots for each of those parameters. Second, applicants claims do not require such an interpretation (the phrase “enumerated candidate variable values” conflates “enumerated variable values” and “candidate variable values”, and there is no requirement in the claims tantamount to “recognizing and selecting”). But further, even if both were true, in what way does this interpretation of Fuegen fail to teach “recognizing and selecting among enumerated candidate variable values”? Respectfully, there is no clear connection to be drawn.
Though applicant concludes that “Nothing in Fuegen describes: enumerated candidate variable values, computing a probability for each candidate value, or selecting the candidate value with the highest probability” and “Fuegen does not disclose: predefined lists of candidate variable values, computing probabilities for each candidate value, or selecting a candidate value from such a list,” each of these statements are made without even a bare minimum of support. Applicant draws no direct connection between the cited portions of the reference (and associated explanations) which were presented in the Office Action and what applicant believes is recited in the claims which is not addressed by those specific portions. As such, applicants arguments fail to comply with 37 CFR 1.111(c) and are improper.
If applicant believes that the “enumerated variable values” are somehow different from the “slots and parameters” of Fuegen, applicant is expressly requested to explain how they are different. The slots of Fuegen, and as understood in the art, are a variable data structure. A parameter bound to a specific slot within a defined domain-specific NLU task represents a choice from a finite, closed set of valid identifiers. From the applicant perspective, in what clear and tangible way are the enumerated variable values different from the values described in Fuegen?
With respect to the rejection(s) of claim(s) 11 under 35 U.S.C. §103 in light of Fuegen, Liu, and Avijeet applicant argues that Fuegen, Liu, and Avijeet fail to teach or suggest “wherein outputting the request for the virtual assistant action is conditioned on a delayed indication of the variable value probability of the variable value with the highest variable value probability.” Applicant’s arguments are duly noted. However, applicant’s arguments amount to little more than attorney argument regarding perceived advantages (“represents an explicit design choice… allows the system to use the delayed variable- recognition result as a gating or conditioning input”) or differences which do not correspond to the claimed subject matter (“the architecture involves a staged information flow in which the result of the variable-recognition process is not immediately consumed, but is instead delayed and then supplied as conditioning information…”). As these arguments do not address asserted deficiencies in the rejection as previously presented, and, as previously indicated, applicants claims do not recite any such separation between the variable recognizer and the intent recognizer, such arguments are moot.
However, regarding applicant’s arguments as applied to claims 11, 13 and 15, it is respectfully noted that applicants described explicit design choice and/or staged information flow is not reflected in the claims as presented. Claim 13 recites “wherein the probability is delayed” and claim 15 recites “wherein an indication of the value of the variable is delayed.” There is no structure recited with respect to either limitation and there is no causal relationship with any structure. The delay could occur for any reason, and even be completely unrelated to the design of the system or method. The current amendment in claim 11 does not remedy this problem, in that “wherein outputting the request for the virtual assistant action is conditioned on a delayed indication of the variable value probability of the variable value with the highest variable value probability” merely calls for the delay to occur. The delay is not caused or generated from any portion of the method. Applicant is invited to amend the claims during normal prosecution such that the claims reflect the desired limitations and said claims can be substantively examined. Therefore, the rejection of claim 11 is maintained over the arguments provided by the applicant.
Applicant further argues that the rejection(s) of dependent claims 2-10 and 12-20 should be withdrawn for at least the same reasons as independent claims 1 and 11. Applicant’s arguments in light of the amended claims are not persuasive for the same reasons as discussed above with relation to claims 1 and 11.
The Applicant has not provided any further statement and therefore, the Examiner directs the Applicant to the below rationale.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 11-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Regarding claim 11, the limitation “wherein outputting the request for the virtual assistant action is conditioned on a delayed indication of the variable value probability of the variable value with the highest variable value probability” lacks specification support. Claim 11 recites “wherein outputting the request for the virtual assistant action is conditioned on a delayed indication of the variable value probability of the variable value with the highest variable value probability” at lines 13-15. However, the limitation does not have clear support in the specification or claims as filed.
Upon review of the response, applicant does not indicate where specification support for this limitation can be found. Upon further review of the specification, the limitation as presented is not recited in the specification. Further, the claim limitation “delayed indication” does not occur in the specification, and the word “delay” only occurs once, with relation to “time-delayed peak”. As applicant appears to be attempting to distinguish between the “request is conditioned on the variable value probability” in claim 12 and the “request…is conditioned on a delayed indication of the variable value probability,” in claim 11 as amended, clear specification support for such a distinction is required. Therefore, claim 11 contains limitations which lack specification support and the claim is rejected.
Regarding claims 12-20, claims 12-20 depend from claim 11, and incorporate all limitations therefrom. Therefore, claims 12-20 are rejected for at least the same reasons as described with reference to claim 11.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, claim 1 recites the limitation “the speech audio features” in line 2. There is insufficient antecedent basis for this limitation in the claim.
Regarding claim 11, the limitation “wherein outputting the request for the virtual assistant action is conditioned on a delayed indication of the variable value probability of the variable value with the highest variable value probability” lacks clarity. Claim 11 recites “wherein outputting the request for the virtual assistant action is conditioned on a delayed indication of the variable value probability of the variable value with the highest variable value probability” at lines 13-15. This limitation appears to be an amalgamation of claims 12-15. However, as claims 12-15 are still present and depend from claim 11 as amended, the relationship between these limitations is unclear. There are numerous duplicated parts which are not addressed in the claims (e.g., claim 15 recites “an indication of the value of the plurality of enumerated variable values . Further, based on the current structure, it is unclear what relationship is being established between the delayed indication and the outputting of the request. Applicant hasn’t established that an indication of the variable value probability is related to the outputting of a request for virtual assistant action. As such, is the applicant indicating the delay in the indication as somehow relating to the conditioning? Is the applicant defining the indication as being delayed, and then conditioning the output on the “delayed indication” as a defined part? In essence, which of (1) delay, (2) indication, or (3) delayed indication, is the applicant further conditioning the output of the request upon? As applicant has disclosed both further delays and further indications in the claim set which must be accounted for, the relationship of these parts is necessary for clarity in the claims. Therefore, claim 11 lacks clarity and is rejected under 35 USC 112(b).
Regarding claims 2-10 and 12-20, claims 2-10 and 12-20 depend from claims 1 and 11, and incorporate all limitations therefrom. Therefore, claims 2-10 and 12-20 are rejected for at least the same reasons as described with reference to claims 1 and 11.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1-5, 8, 11-15, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fuegen in view of Liu and Avijeet.
Regarding claim 1, Fuegen discloses A machine for recognizing an intent in speech audio (Systems and methods described with reference to “end-to-end (E2E) SLU system 300”; Fuegen, ¶ Col. 4, lines 44-49), the machine comprising: a variable recognizer that recognizes one or more enumerated variable values directly from the speech audio features, (“the system may receive audio input...for a slot filling task” where the audio input “may include raw audio data, such as frequency information corresponding to an audio recording” where the system applies “the network to extract semantic meaning directly from the raw audio or audio features” which “may involve performing domain or intent classification, or performing a slot-filling task {recognizing one or more enumerated variable values...}” and “the system may apply the network to extract semantic meaning directly from the raw audio…without generating an intermediate textual representation of the audio {...directly from the speech audio}”; Fuegen, ¶ Col. 5, lines 48- col. 6, line 4) wherein the one or more enumerated variable values are drawn from a predefined list of candidate variable values associated with a domain (Discloses that the slot-filling task includes “a list of parameters and an identification of a slot for each of the parameters” where applicant’s variables are understood to correspond to the slots of Fuegen, and applicant’s “enumerated variable values,” which are generated for a corresponding variable, correspond to the parameters/values as understood in the relevant art, which are generated to fill a slot for the slot-filling task in Fuegen. For determining the corresponding variable, the classification task is constrained by a multi-layer bidirectional GRU encoder tied to a closed-ended softmax decoding layer which necessarily produces an output based on a finite set of intent/domain posterior probabilities. As the system of Fuegen does not have an open-ended token generation mechanism and does not produce generative text, the output vocabulary for variable values is structurally predefined {drawn from a predefined list of candidate variable values}. Fuegen further discloses “domain and intent classification” followed by performing “slot filling based on different identified intents.” As such, the enumerated variable value is associated with the slot-filling task and with a domain.; Fuegen, ¶ Col. 3, lines 25-37; Col. 5, lines 7-13; Col. 5, lines 48- col. 6, line 4), and wherein the one or more enumerated variable values represent one or more of a... thing (The slot-filling task includes “a list of parameters and an identification of a slot for each of the parameters”, which are generated to fill a slot for the slot-filling task in Fuegen, and the parameters necessarily correspond to a “thing.”; Fuegen, ¶ Col. 5, lines 48- col. 6, line 4) , and wherein the variable recognizer computes, for each candidate variable in the predefined list, a variable value probability indicating a likelihood that the candidate variable value is present in the speech audio (the system can generate, based on the audio features extracted from the audio input, “a list of parameters and an identification of a slot for each of the parameters (in a slot-filling task) {...for each of a plurality of enumerated variable values}” and may further include multiple “slot lists {of an enumerated variable value of the plurality of enumerated values} and a corresponding probability that the network assigns to the respective classifications/lists {computes a variable value probability}.” It is noted that applicant has shifted to speech audio generally. As such, this is understood as referring to generalized probabilities for speech audio, as opposed to probabilities in the “speech audio features,” which reflect the features specifically detected at the variable recognizer.; Fuegen, ¶ Col. 5, lines 36-64), and wherein the variable recognizer computes, for each candidate variable in the predefined list, a variable value probability indicating a likelihood that the candidate variable value is present in the speech audio, a variable value probability indicating a likelihood that the candidate variable value is present in the speech audio (the system can generate, based on the audio features extracted from the audio input, “a list of parameters and an identification of a slot for each of the parameters (in a slot-filling task) {...for each of a plurality of enumerated variable values}” and may further include multiple “slot lists {of an enumerated variable value of the plurality of enumerated values} and a corresponding probability that the network assigns to the respective classifications/lists {computes a variable value probability}.”; Fuegen, ¶ Col. 5, lines 36-64); and an intent recognizer that processes the speech audio features (the system “extract[s] semantic meaning {that processes...} directly from the raw audio or audio features {the speech audio features}” such as for “domain or intent classifications {an intent recognizer...}”; Fuegen, ¶ Col. 5, line 57 - Col. 6, line 4), computes an intent probability of the speech audio having the intent (The system computes “a corresponding [posterior] probability” for selection of “domain or intent classification” which the system “assigns to the respective classifications,” where the posterior probability is a likelihood, given prior information that a specified hypothesis among a plurality of hypotheses is correct. As such, the posterior probability for a multitude of hypotheses is used to select a most likely hypothesis from a list of candidate hypotheses.; Fuegen, ¶ Col. 6, lines 5-12), and in response to the intent probability being above an intent threshold (“the system may generate an output” which “may be a domain or intent classification” where the output of a classification is based on the previously calculated posterior probability, and where the selection of an output based on a posterior probability is in response to the intent probability being above an intent threshold (in the absence of a predefined threshold, the highest probability is selected and the threshold is set by the probability of the next highest posterior probability {intent probability}); Fuegen, ¶ Col. 6, lines 5-12), produces [an output to]… a virtual assistant… (the system then provides the output to “a dialog manager or hands-free interface for a device such as a mobile device or virtual reality (VR) system {produces [an output to]... a virtual assistant...}”; Fuegen, ¶ Col. 6, lines 5-12). However, Fuegen fail(s) to expressly recite wherein the one or more enumerated variable values represent one or more of a person [or a] place, and outputs the enumerated variable value of the one or more enumerated variable values with the highest variable value probability, wherein the output is a request for a virtual assistant action.
Liu teaches “a recurrent neural network (RNN) model that jointly performs intent detection, slot filling, and language modeling.” (Liu, Abstract). Regarding claim 1, Liu teaches wherein the one or more enumerated variable values represent one or more of a... place or thing (“slot filling task can also be viewed as assigning an appropriate semantic label to each word in the given input text. In the below example from ATIS (Hemphill et al., 1990) corpus following the popular in/out/begin (IOB) annotation method, Seattle and San Diego are the from and to locations respectively according to the slot labels, and tomorrow is the departure date,” where the Seattle and San Diego are parameters corresponding to a place.; Liu, ¶ Pg. 2, col. 2, para. 4) and outputs the enumerated variable value of the one or more enumerated variable values with the highest variable value probability (Regarding the slot filling task as part of an end-to-end spoken language understanding system, Liu recites that “Given an utterance consisting of a sequence of words w = (w1;w2;:::;wT), the goal of slot filling is to find a sequence of semantic labels s = (s1; s2;:::; sT), one for each word in the utterance, such that” the posterior probability of the semantic meaning is maximized (see equation 2).; Liu, ¶ Pg. 2, col. 2, para. 5-6).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, to incorporate the teachings of Liu to include outputs the enumerated variable value of the plurality of enumerated variable values with the highest variable value probability. The systems and methods of Liu “update the intent prediction” in real time as the utterance arrives and “uses it as contextual features in a joint model” which shows improvement on SLU tasks where the “joint model outperforms the independent task training model by 22.3% on intent detection error rate” and “shows advantageous performance” in “realistic ASR settings with noisy speech input”, as recognized by Liu. (Liu, Abstract). However, Fuegen and Liu fail(s) to expressly recite wherein the one or more enumerated variable values represent… a person, [and] wherein the output is a request for a virtual assistant action.
Avijeet teaches a speech processing system “configured to process the received speech signal to extract one or more of intents and entities.” (Avijeet, ¶ [0003]). Regarding claim 1, Avijeet teaches wherein the one or more enumerated variable values represent one or more of a person, place or thing (Discloses a unified neural network which determines intents and entities without generating human readable text, which further teaches that “parsing intents and entities may be an example of shallow semantic parsing performed by a neural network where entities may be identified (also known as slot-filling or frame semantic parsing)” and where “entities may be lists of keywords defining objects of one class (e.g., person, organization, location, automobile, name of place, food, etc.)”; Avijeet, ¶ [0068], [0075]) [and] wherein the output is a request for a virtual assistant action (“The input interface 508 may direct speech to a first unified neural network 502 (where the speech may be processed and converted into intents and entities for the DM response)” where “the DM 506 may require that speech be converted to intents and entities (e.g., converted by first unified neural network 502) for the DM 506 to understand and then the DM may generate an appropriate response to these intents and entities,” where the “intents and entities... converted by first unified neural network” for the DM to “generate an appropriate response” are a request for a virtual assistant action.; Avijeet, ¶ [0066]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, to incorporate the teachings of Avijeet to include wherein the one or more enumerated variable values represent… a person, [and] wherein the output is a request for a virtual assistant action. The “one or more unified neural networks” of Avijeet “may provide efficient processing of speech signals by reducing complexity by use of fewer neural networks,” which “may improve speed and accuracy of processing (e.g., processing of the speech signal, processing of natural language generation, processing of dialogue manager response, etc.), which may be beneficial in certain applications, such as, e.g., virtual assistants and the like capable of communication with the user,” thus incorporating and improving upon the known benefits of interacting through a virtual assistant by further streamlining the communication with said virtual assistant, as recognized by Avijeet. (Avijeet, ¶ [0089]-[0090]).
Regarding claim 2, the rejection of claim 1 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. Fuegen further discloses wherein: the variable recognizer indicates the variable value probability of the speech audio features having an enumerated variable value (The system generates an output, where the output may be “an identification of a slot for each of the parameters (in a slot-filling task)” {outputs an enumerated variable value of the plurality of enumerated variable values} where the output may include “slot lists and a corresponding probability that the network assigns to the” respective slot lists.; Fuegen, ¶ Col. 6, lines 5-12). However, Fuegen fail(s) to expressly recite the intent recognizer conditions its output of a request for an action on the variable value probability of the speech audio having an enumerated variable values.
The relevance of Liu is described above with relation to claim 1. Regarding claim 2, Liu teaches the intent recognizer conditions its output of a request for an action on the variable value probability of the speech audio features having an enumerated variable values (Discloses a SLU-LM including an intent model which outputs “intent at each time step as input word sequence arrives” where “the intent output from each time step is fed back to the RNN state, and thus the entire intent output history are modeled and can be used as context to other tasks” and “at time step t, input to the system is the word at index t of the utterance, and outputs are the intent class, the slot label, and the next word prediction. The RNN state ht encodes the information of all the words, intents, and slot labels seen previously,” which includes both the slot determination and the associated probabilities (see Equations (7)-(10)).; Liu, ¶ pg. 3, col. 2, para. 8; pg. 4, col. 1, para. 1; pg. 4, col. 2, para. 2; Equations (7)-(10)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to further incorporate the teachings of Liu to include the intent recognizer conditions its output of a request for an action on the variable value probability of the speech audio having an enumerated variable values. The systems and methods of Liu “update the intent prediction” in real time as the utterance arrives and “uses it as contextual features in a joint model” which shows improvement on SLU tasks where the “joint model outperforms the independent task training model by 22.3% on intent detection error rate” and “shows advantageous performance” in “realistic ASR settings with noisy speech input”, as recognized by Liu. (Liu, Abstract).
Regarding claim 3, the rejection of claim 2 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. However, Fuegen fail(s) to expressly recite wherein the conditioning is based on a delayed indication of the variable value probability of the speech audio having an enumerated variable value.
The relevance of Liu is described above with relation to claim 1. Regarding claim 3, Liu teaches wherein the conditioning is based on a delayed indication of the variable value probability of the speech audio features having the enumerated variable value (The disclosed SLU-LM provides for “the intent and slot label outputs at current step, together with the intent and slot label history that is encoded in the RNN state” to “serve as context to the language model” As such, later received input results and later received “slot labels at the current step” which are a delayed indication of the enumerated variable value and the associated variable value probability.; Liu, ¶ pg. 3, col. 2, para. 8; pg. 4, col. 1, para. 1).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to further incorporate the teachings of Liu to include wherein the conditioning is based on a delayed indication of the variable value probability of the speech audio having an enumerated variable value. The systems and methods of Liu “update the intent prediction” in real time as the utterance arrives and “uses it as contextual features in a joint model” which shows improvement on SLU tasks where the “joint model outperforms the independent task training model by 22.3% on intent detection error rate” and “shows advantageous performance” in “realistic ASR settings with noisy speech input”, as recognized by Liu. (Liu, Abstract).
Regarding claim 4, the rejection of claim 1 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. However, Fuegen fail(s) to expressly recite wherein the intent recognizer conditions its output of a request for an action on which value of the plurality of enumerated variable values has the highest variable value probability.
The relevance of Liu is described above with relation to claim 1. Regarding claim 4, Liu teaches wherein the intent recognizer conditions its output of a request for an action on which value of the one or more enumerated variable values has the highest variable value probability (Discloses a SLU-LM including an intent model which outputs “intent at each time step as input word sequence arrives” where “the intent output from each time step is fed back to the RNN state, and thus the entire intent output history are modeled and can be used as context to other tasks” and “at time step t, input to the system is the word at index t of the utterance, and outputs are the intent class, the slot label, and the next word prediction. The RNN state ht encodes the information of all the words, intents, and slot labels seen previously,” which includes both the slot determination and the associated probabilities (see Equations (7)-(10)).; Liu, ¶ pg. 3, col. 2, para. 8; pg. 4, col. 1, para. 1; pg. 4, col. 2, para. 2; Equations (7)-(10)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to further incorporate the teachings of Liu to include wherein the intent recognizer conditions its output of a request for an action on which value of the plurality of enumerated variable values has the highest variable value probability. The systems and methods of Liu “update the intent prediction” in real time as the utterance arrives and “uses it as contextual features in a joint model” which shows improvement on SLU tasks where the “joint model outperforms the independent task training model by 22.3% on intent detection error rate” and “shows advantageous performance” in “realistic ASR settings with noisy speech input”, as recognized by Liu. (Liu, Abstract).
Regarding claim 5, the rejection of claim 4 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. Fuegen further discloses wherein the conditioning is based on …[an] indication of which value of the plurality of enumerated variable values has the highest variable value probability (“the system may generate an output” which “may be a domain or intent classification” where the output of a classification is based on the previously calculated posterior probability, and where the selection of an output based on a posterior probability is in response to the intent probability being above an intent threshold (in the absence of a predefined threshold, the highest probability is selected and the threshold is set by the probability of the next highest posterior probability {intent probability}), and the output “may be configured for use by a dialog manager or hands-free interface for a device such as a mobile device or virtual reality (VR) system {produces a request for a virtual assistant action}”; Fuegen, ¶ Col. 6, lines 5-12). However, Fuegen fail(s) to expressly recite wherein the indication is a delayed indication.
The relevance of Liu is described above with relation to claim 1. Regarding claim 5, Liu teaches wherein the indication is a delayed indication (The disclosed SLU-LM provides for “the intent and slot label outputs at current step, together with the intent and slot label history that is encoded in the RNN state” to “serve as context to the language model” As such, later received input results in later received “slot labels at the current step” which are a delayed indication of the enumerated variable value and the associated variable value probability.; Liu, ¶ pg. 3, col. 2, para. 8; pg. 4, col. 1, para. 1).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to further incorporate the teachings of Liu to include wherein the indication is a delayed indication. The systems and methods of Liu “update the intent prediction” in real time as the utterance arrives and “uses it as contextual features in a joint model” which shows improvement on SLU tasks where the “joint model outperforms the independent task training model by 22.3% on intent detection error rate” and “shows advantageous performance” in “realistic ASR settings with noisy speech input”, as recognized by Liu. (Liu, Abstract).
Regarding claim 8, the rejection of claim 1 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. Fuegen further discloses wherein no human-readable speech transcription is computed (“Extracting the semantic meaning may be done without generating an intermediate textual representation of the audio,” which includes no human-readable speech transcript; Fuegen, ¶ Col. 6, lines 1-4).
Regarding claim 11, Fuegen discloses A method of recognizing an intent from speech audio by a computer system (Systems and methods described with reference to “end-to-end (E2E) SLU system 300”; Fuegen, ¶ Col. 4, lines 44-49), the method comprising: obtaining speech audio (The system can “receive audio input” such as “for domain/intent classification or for a slot filling task.”; Fuegen, ¶ Col. 5, lines 48-57); processing features of the speech audio to recognize a variable value directly from the speech audio, (“the system may receive audio input...for a slot filling task” where the audio input “may include raw audio data, such as frequency information corresponding to an audio recording” where the system applies “the network to extract semantic meaning directly from the raw audio or audio features” which “may involve performing domain or intent classification, or performing a slot-filling task {recognizing one or more enumerated variable values...}” and “the system may apply the network to extract semantic meaning directly from the raw audio…without generating an intermediate textual representation of the audio {...directly from the speech audio}”; Fuegen, ¶ Col. 5, lines 48- col. 6, line 4) wherein the variable value comprises one of a … thing, (Discloses that the slot-filling task includes “a list of parameters and an identification of a slot for each of the parameters” where applicant’s variables are understood to correspond to the slots of Fuegen, and applicant’s “enumerated variable values,” which are generated for a corresponding variable, correspond to the parameters/values as understood in the relevant art, which are generated to fill a slot for the slot-filling task in Fuegen, and the parameters necessarily correspond to a “thing.”; Fuegen, ¶ Col. 5, lines 48- col. 6, line 4) wherein processing features of the speech audio comprises computing a variable value probability of the speech audio having any of a plurality of enumerated variable values (the system can generate, based on the audio features extracted from the audio input, “a list of parameters and an identification of a slot for each of the parameters (in a slot-filling task) {...for each of a plurality of enumerated variable values}” and may further include multiple “slot lists {of an enumerated variable value of the plurality of enumerated values} and a corresponding probability that the network assigns to the respective classifications/lists {computes a variable value probability}.”; Fuegen, ¶ Col. 5, lines 36-64); outputting the variable value… (The system generates an output, where the output may be “an identification of a slot for each of the parameters (in a slot-filling task)” {outputs the variable value} where the output may include “slot lists and a corresponding [posterior] probability that the network assigns to the” respective slot lists.; Fuegen, ¶ Col. 6, lines 5-12); processing the features of the speech audio (the system “extract[s] semantic meaning {that processes...} directly from the raw audio or audio features {the speech audio features}” such as for “domain or intent classifications {an intent recognizer...}”; Fuegen, ¶ Col. 5, line 57 - Col. 6, line 4) to compute an intent probability of the speech audio having the intent (The system computes “a corresponding [posterior] probability” for selection of “domain or intent classification” which the system “assigns to the respective classifications,” where the posterior probability is a likelihood, given prior information that a specified hypothesis among a plurality of hypotheses is correct. As such, the posterior probability for a multitude of hypotheses is used to select a most likely hypothesis from a list of candidate hypotheses.; Fuegen, ¶ Col. 6, lines 5-12); and in response to the intent probability being above an intent threshold (“the system may generate an output” which “may be a domain or intent classification” where the output of a classification is based on the previously calculated posterior probability, and where the selection of an output based on a posterior probability is in response to the intent probability being above an intent threshold (in the absence of a predefined threshold, the highest probability is selected and the threshold is set by the probability of the next highest posterior probability {intent probability}); Fuegen, ¶ Col. 6, lines 5-12), outputting... [to] virtual assistant action (the system then provides the output to “a dialog manager or hands-free interface for a device such as a mobile device or virtual reality (VR) system {produces [an output to]... a virtual assistant...}”; Fuegen, ¶ Col. 6, lines 5-12). However, Fuegen fail(s) to expressly recite wherein the variable value comprises one of a person [or a] place, outputting the variable value with the highest variable value probability and wherein the output is a request for a virtual assistant action, and wherein outputting the request for the virtual assistant action is conditioned on a delayed indication of the variable value probability of the variable value with the highest variable value probability.
The relevance of Liu is described above with relation to claim 1. Regarding claim 11, Liu teaches wherein the variable value comprises one of a… place or thing, (“slot filling task can also be viewed as assigning an appropriate semantic label to each word in the given input text. In the below example from ATIS (Hemphill et al., 1990) corpus following the popular in/out/begin (IOB) annotation method, Seattle and San Diego are the from and to locations respectively according to the slot labels, and tomorrow is the departure date,” where the Seattle and San Diego are parameters corresponding to a place.; Liu, ¶ Pg. 2, col. 2, para. 4); outputting the variable value with the highest variable value probability (Regarding the slot filling task as part of an end-to-end spoken language understanding system, Liu recites that “Given an utterance consisting of a sequence of words w = (w1;w2;:::;wT), the goal of slot filling is to find a sequence of semantic labels s = (s1; s2;:::; sT), one for each word in the utterance, such that” the posterior probability of the semantic meaning is maximized (see equation 2).; Liu, ¶ Pg. 2, col. 2, para. 5-6) wherein outputting the request for the virtual assistant action is conditioned on a delayed indication of the variable value probability of the variable value with the highest variable value probability (Regarding the slot filling task as part of an end-to-end spoken language understanding system, Liu recites that “Given an utterance consisting of a sequence of words w = (w1;w2;:::;wT), the goal of slot filling is to find a sequence of semantic labels s = (s1; s2;:::; sT), one for each word in the utterance, such that” the posterior probability of the semantic meaning is maximized (see equation 2), where “the intent and slot label outputs at current step, together with the intent and slot label history that is encoded in the RNN state” to “serve as context to the language model” As such, later received input results and later received “slot labels at the current step” are a delayed indication of the enumerated variable value and the associated highest variable value probability. As the output requires the associated input, the output is necessarily conditioned on receiving the input and thus any associated delay in receiving the input; Liu, ¶ Pg. 2, col. 2, para. 5-6; pg. 3, col. 2, para. 8; pg. 4, col. 1, para. 1).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, to incorporate the teachings of Liu to include wherein the variable value comprises one of a… place or thing, outputting the variable value with the highest variable value probability, and wherein outputting the request for the virtual assistant action is conditioned on a delayed indication of the variable value probability of the variable value with the highest variable value probability. The systems and methods of Liu “update the intent prediction” in real time as the utterance arrives and “uses it as contextual features in a joint model” which shows improvement on SLU tasks where the “joint model outperforms the independent task training model by 22.3% on intent detection error rate” and “shows advantageous performance” in “realistic ASR settings with noisy speech input”, as recognized by Liu. (Liu, Abstract). However, Fuegen and Liu fail(s) to expressly recite wherein the variable value comprises… a person, [and] wherein the output to a virtual assistant is a request for a virtual assistant action.
The relevance of Avijeet is described above with relation to claim 1. Regarding claim 11, Avijeet teaches wherein the variable value comprises one of a person, place or thing, (Discloses a unified neural network which determines intents and entities without generating human readable text, which further teaches that “parsing intents and entities may be an example of shallow semantic parsing performed by a neural network where entities may be identified (also known as slot-filling or frame semantic parsing)” and where “entities may be lists of keywords defining objects of one class (e.g., person, organization, location, automobile, name of place, food, etc.)”; Avijeet, ¶ [0068], [0075]), [and] outputting a request for a virtual assistant action (“The input interface 508 may direct speech to a first unified neural network 502 (where the speech may be processed and converted into intents and entities for the DM response)” where “the DM 506 may require that speech be converted to intents and entities (e.g., converted by first unified neural network 502) for the DM 506 to understand and then the DM may generate an appropriate response to these intents and entities,” where the “intents and entities... converted by first unified neural network” for the DM to “generate an appropriate response” are a request for a virtual assistant action.; Avijeet, ¶ [0066]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, to incorporate the teachings of Avijeet to include wherein the variable value comprises… a person, [and] wherein the output to a virtual assistant is a request for a virtual assistant action. The “one or more unified neural networks” of Avijeet “may provide efficient processing of speech signals by reducing complexity by use of fewer neural networks,” which “may improve speed and accuracy of processing (e.g., processing of the speech signal, processing of natural language generation, processing of dialogue manager response, etc.), which may be beneficial in certain applications, such as, e.g., virtual assistants and the like capable of communication with the user,” thus incorporating and improving upon the known benefits of interacting through a virtual assistant by further streamlining the communication with said virtual assistant, as recognized by Avijeet. (Avijeet, ¶ [0089]-[0090]).
Regarding claim 12, the rejection of claim 11 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. However, Fuegen fail(s) to expressly recite wherein outputting a request is conditioned on the variable value probability of the variable value with the highest probability.
The relevance of Liu is described above with relation to claim 1. Regarding claim 12, Liu teaches wherein outputting a request is conditioned on the variable value probability of the variable value with the highest probability (Discloses a SLU-LM including an intent model which outputs “intent at each time step as input word sequence arrives” where “the intent output from each time step is fed back to the RNN state, and thus the entire intent output history are modeled and can be used as context to other tasks” and “at time step t, input to the system is the word at index t of the utterance, and outputs are the intent class, the slot label, and the next word prediction. The RNN state ht encodes the information of all the words, intents, and slot labels seen previously,” which includes both the slot determination and the associated probabilities (see Equations (7)-(10)).; Liu, ¶ pg. 3, col. 2, para. 8; pg. 4, col. 1, para. 1; pg. 4, col. 2, para. 2; Equations (7)-(10)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to further incorporate the teachings of Liu to include wherein outputting a request is conditioned on the variable value probability of the variable value with the highest probability. The systems and methods of Liu “update the intent prediction” in real time as the utterance arrives and “uses it as contextual features in a joint model” which shows improvement on SLU tasks where the “joint model outperforms the independent task training model by 22.3% on intent detection error rate” and “shows advantageous performance” in “realistic ASR settings with noisy speech input”, as recognized by Liu. (Liu, Abstract).
Regarding claim 13, the rejection of claim 12 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. Fuegen further teaches wherein the variable value probability is delayed (“the output (and/or probabilities) may be stored locally in a memory and/or transmitted via a network,” where transmission of the probabilities via a network necessarily includes a delay (e.g., latency); Fuegen, ¶ Col. 6, lines 12-17).
Regarding claim 14, the rejection of claim 11 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. However, Fuegen fail(s) to expressly recite wherein outputting a request is conditioned on which value of the plurality of enumerated variable values has the highest variable value probability.
The relevance of Liu is described above with relation to claim 1. Regarding claim 14, Liu teaches wherein outputting the request is conditioned on which value of the plurality of enumerated variable values has the highest variable value probability (Regarding the slot filling task as part of an end-to-end spoken language understanding system, Liu recites that “Given an utterance consisting of a sequence of words w = (w1;w2; :::;wT), the goal of slot filling is to find a sequence of semantic labels s = (s1; s2; :::; sT), one for each word in the utterance, such that” the posterior probability of the semantic meaning is maximized (see equation 2) {highest variable value probability}, where “the intent and slot label outputs at current step, together with the intent and slot label history that is encoded in the RNN state” to “serve as context to the language model” As such, later received input results and later received “slot labels at the current step” are a delayed indication of the enumerated variable value and the associated variable value probability.; Liu, ¶ Pg. 2, col. 2, para. 5-6; pg. 3, col. 2, para. 8; pg. 4, col. 1, para. 1).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to further incorporate the teachings of Liu to include wherein outputting a request is conditioned on which value of the plurality of enumerated variable values has the highest variable value probability. The systems and methods of Liu “update the intent prediction” in real time as the utterance arrives and “uses it as contextual features in a joint model” which shows improvement on SLU tasks where the “joint model outperforms the independent task training model by 22.3% on intent detection error rate” and “shows advantageous performance” in “realistic ASR settings with noisy speech input”, as recognized by Liu. (Liu, Abstract).
Regarding claim 15, the rejection of claim 14 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. Fuegen further teaches wherein an indication of the value of the plurality of enumerated variable values is delayed (“the output (and/or probabilities) may be stored locally in a memory and/or transmitted via a network,” where transmission of the output and the probabilities via a network necessarily includes a delay (e.g., latency); Fuegen, ¶ Col. 6, lines 12-17).
Regarding claim 18, the rejection of claim 11 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. Fuegen further discloses wherein no human-readable speech transcription is computed (“Extracting the semantic meaning may be done without generating an intermediate textual representation of the audio.”; Fuegen, ¶ Col. 6, lines 1-4).
Claims 6 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fuegen, Liu, and Avijeet as applied to claim 1 and 11 above, and further in view of Seo.
Regarding claim 6, the rejection of claim 1 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. However, Fuegen, Liu, and Avijeet fail to expressly recite wherein one of the recognizers produces a score and the other recognizer is called in response to the score being above a score threshold.
Seo teaches systems and methods for “improving a sentence intent analysis rate through named entity structuring.” (Seo, ¶ [0002]). Regarding claim 6, Seo teaches wherein one of the recognizers produces a score (“determining of the utterance intent may be performed” and “accuracy of the utterance intent is... compared to a preset target value.” The comparison to a preset target value indicates that the accuracy of the utterance intent is a value {score}.; Seo, ¶ [0018]) and the other recognizer is called in response to the score being above a score threshold (“when accuracy of the utterance intent {score} is small compared to a preset target value,” the “method may further include restructuring the relationship of the named entity to reset the relationship information”; Seo, ¶ [0018]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to incorporate the teachings of Seo to include wherein one of the recognizers produces a score and the other recognizer is called in response to the score being above a score threshold. Seo discloses systems and methods of “improving accuracy of determining a named entity and an utterance intent of a user through named entity structuring”, such as for slot filling, whereby a system can determine “an utterance intent through named entity structuring without collecting similar named entities… even in a case of a higher utterance pattern,” thus improving the quality of intent determination as well as downstream processing, as recognized by Seo. (Seo, ¶ [0009]-[0011], [0164]).
Regarding claim 16, the rejection of claim 11 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. However, Fuegen, Liu, and Avijeet fail to expressly recite wherein one of the variable value and intent probability computations is performed in response to the other probability computation having a result that is above a threshold.
The relevance of Seo is described above with relation to claim 6. Regarding claim 16, Seo teaches wherein one of the variable value and intent probability computations is performed in response to the other probability computation having a result that is above a threshold (“a similarity between features {a result} may be measured through weight calculation between the features (S1350), and when the similarity is greater than or equal to a preset threshold (S1360), the named entity may be structured by clustering the named entity (S1370) {...the other probability computation having a result that is above a threshold}” and then “through the named entity recognition considering the named entity relationship, the apparatus for speech recognition may determine the utterance intent” where the determination of intent is based in part on a probability determination (e.g., “the accuracy of the utterance intent is 90%”) {one of the probability computations is performed in response to...}; Seo, ¶ [0209], [0214]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to incorporate the teachings of Seo to include wherein one of the variable value and intent probability computations is performed in response to the other probability computation having a result that is above a threshold. Seo discloses systems and methods of “improving accuracy of determining a named entity and an utterance intent of a user through named entity structuring”, such as for slot filling, whereby a system can determine “an utterance intent through named entity structuring without collecting similar named entities… even in a case of a higher utterance pattern,” thus improving the quality of intent determination as well as downstream processing, as recognized by Seo. (Seo, ¶ [0009]-[0011], [0164]).
Claims 7 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fuegen, Liu, and Avijeet as applied to claim 1 and 11 above, and further in view of Evermann.
Regarding claim 7, the rejection of claim 1 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. Fuegen further teaches further comprising a domain recognizer that processes speech audio features (“the system may apply the network to extract semantic meaning directly from the raw audio or audio features” including performing domain classification.; Fuegen, ¶ Col. 5, lines 35-47; Col. 6, lines 5-12) and computes a domain probability of the speech audio referring to a specific domain, (“The output may include” multiple domain classifications and “a corresponding probability that the network assigns to the respective classifications”; Fuegen, ¶ Col. 6, lines 5-12). However, Fuegen, Liu, and Avijeet fail to expressly recite wherein the intent recognizer is associated with the domain and called in response to the probability of the speech audio referring to a specific domain being above a domain threshold.
Evermann teaches systems and methods for “processing a speech input to a infer user intent therefrom.” (Evermann, ¶ [0002]). Regarding claim 7, Evermann teaches wherein the intent recognizer is associated with the domain (“The confidence score can be used, for example, to help determine which of two candidate domains is most likely to accurately reflect or represent the intent of the input.”; Everman, ¶ [0011]) and called in response to the probability of the speech audio referring to a specific domain being above a domain threshold (The score for a recognizer’s confidence that the particular item is in correct domain {i.e., referring to a specific domain} is compared against “a predetermined confidence threshold” where “if no candidate domain satisfies a predetermined confidence threshold, the digital assistant will not provide a response.”; Everman, ¶ [0011]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to incorporate the teachings of Evermann to include further comprising a domain recognizer that computes a probability of the speech audio referring to a specific domain, wherein the domain recognizer is called in response to the probability of the speech audio referring to a specific domain being above a domain threshold. The systems and methods taught by Evermann can “infer user intent from a speech input so as to account for possible speech recognition errors” which improves speech recognition quality and ease of use by a user. (Evermann, ¶ [0005], [0007]).
Regarding claim 17, the rejection of claim 11 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. Fuegen further teaches further comprising processing features of the speech audio (“the system may apply the network to extract semantic meaning directly from the raw audio or audio features” including performing domain classification.; Fuegen, ¶ Col. 5, lines 35-47; Col. 6, lines 5-12) to compute a domain probability of the speech audio referring to a specific domain, (“The output may include” multiple domain classifications and “a corresponding probability that the network assigns to the respective classifications”; Fuegen, ¶ Col. 6, lines 5-12). However, Fuegen, Liu, and Avijeet fail to expressly recite further comprising processing features of the speech audio to compute a domain probability of the speech audio referring to a specific domain, wherein the intent is associated with the domain and computing the probability of the speech audio having the intent is performed in response to the probability of the speech audio referring to the domain being above a threshold.
The relevance of Evermann is described above with relation to claim 7. Regarding claim 17, Evermann teaches further comprising processing features of the speech audio to compute a domain probability of the speech audio referring to a specific domain, (“The candidate text strings 406 are processed by the natural language processor to determine a respective candidate domain 414 for each respective candidate text string 406,” where the domain is selected based on a calculated confidence score.; Everman, ¶ [0123], [0095]) wherein the intent is associated with the domain (the domains are associated with the “intent represented by the token sequence”; Everman, ¶ [0097]) and computing the probability of the speech audio having the intent is performed (“each domain and/or actionable intent is associated with an intent deduction confidence score representing a confidence that the determined domain and/or actionable intent correctly reflects the intent represented by the token sequence.”; Everman, ¶ [0097]) in response to the probability of the speech audio referring to the domain being above a threshold (the determination of the intent deduction confidence score is performed in response to the selection of a domain, where “the domain having the highest confidence score (e.g., based on the relative importance of its various triggered nodes) is selected.”; Everman, ¶ [0095]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to incorporate the teachings of Evermann to include further comprising processing features of the speech audio to compute a domain probability of the speech audio referring to a specific domain, wherein the intent is associated with the domain and computing the probability of the speech audio having the intent is performed in response to the probability of the speech audio referring to the domain being above a threshold. The systems and methods taught by Evermann can “infer user intent from a speech input so as to account for possible speech recognition errors” which improves speech recognition quality and ease of use by a user. (Evermann, ¶ [0005], [0007]).
Claims 9-10 and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fuegen, Liu, and Avijeet as applied to claim 1 and 11 above, and further in view of Gruber (U.S. Pat. App. Pub. No. 2017/0178626, hereinafter Gruber).
Regarding claim 9, the rejection of claim 1 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. However, Fuegen, Liu, and Avijeet fail to expressly recite further comprising a network client with access to a web application programing interface (web API), wherein, in response to the intent recognizer producing a request for a virtual assistant action, the network client performs a request to the web API, the request having, as an argument, the value output by the variable recognizer.
Gruber teaches “intelligent automated assistant system” for engagement with “the user in an integrated, conversational manner using natural language dialog.” (Gruber, Abstract). Regarding claim 9, Gruber teaches further comprising a network client with access to a web application programing interface (web API), (“assistant 1002 can call external services 1360 that interface with functionality and applications on a device via APIs or by other means” where “external services 1360 include web-enabled services and/or functionality related to or installed on the hardware device itself,” thus disclosing connection with web-enabled services {i.e., a network client} via an API {i.e., access to a web API}; Gruber, ¶ [0086]-[0088]) wherein, in response to the intent recognizer producing a request for a virtual assistant action, the network client performs a request to the web API (“Once assistant 1002 has determined the user’s intent, using the techniques described herein, assistant 1002 can call external services 1340{in response to the intent recognizer producing a request for a virtual assistant action}” that “interface with functionality and applications on a device via APIs… to perform functions and operations that might otherwise be initiated using a conventional user interface on the device,” where “functions and operations may include, for example, setting an alarm, making a telephone call, sending a text message or email message, adding a calendar event, and the like. Such functions and operations may be performed as add-on functions in the context of a conversational dialog between a user and assistant 1002. {the network client performs a request to the web API}”; Gruber, ¶ [0087]), the request having, as an argument, the value output by the variable recognizer (“active ontologies 1050 may be operable to perform and/or implement various types of functions, operations, actions,” thus an argument in the context of a web API, and “some nodes of an active ontology may correspond to domain concepts such as restaurant and its property restaurant name,” where the association of “restaurant and its property restaurant name” is a named entity {value output by the variable recognizer}; Gruber, ¶ [0197], [0206]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to incorporate the teachings of Gruber to include further comprising a network client with access to a web application programing interface (web API), wherein, in response to the intent recognizer producing a request for a virtual assistant action, the network client performs a request to the web API, the request having, as an argument, the value output by the variable recognizer. “The intelligent automated assistant systems of various embodiments of the present invention can unify, simplify, and improve the user's experience with respect to many different applications and functions of an electronic device, and with respect to services that may be available over the Internet,” as recognized by Gruber. (Gruber, ¶ [0010]).
Regarding claim 10, the rejection of claim 9 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. However, Fuegen, Liu, and Avijeet fail to expressly recite further comprising a speech synthesis engine, wherein, in response to receiving a response from the web API, the speech synthesis engine synthesizes speech audio containing information from the web API response and outputs the synthesized speech audio for a user of the virtual assistant.
The relevance of Gruber is described above with relation to claim 9. Regarding claim 10, Gruber teaches further comprising a speech synthesis engine, (“Speech output, may include...Synthesized speech,” thus a speech synthesis engine; Gruber, ¶ [0152], [0153]) wherein, in response to receiving a response from the web API, the speech synthesis engine synthesizes speech audio containing information from the web API response (“assistant 1002 can call external services 1360 that interface with functionality and applications on a device via APIs… to perform functions and operations that might otherwise be initiated using a conventional user interface on the device,” where said functions and operations include “output data/information which may be generated by intelligent automated assistant 1002... [including] Speech output... [such as] Synthesized speech”; Gruber, ¶ [0087], [0148], [0152], [0153]) and outputs the synthesized speech audio for a user of the virtual assistant (The assistant can include “taking input from the user as voice spoken to the assistant and sending output from the assistant to the user, for example as synthesized speech, in reply.”; Gruber, ¶ [0097]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to incorporate the teachings of Gruber to include further comprising a speech synthesis engine, wherein, in response to receiving a response from the web API, the speech synthesis engine synthesizes speech audio containing information from the web API response and outputs the synthesized speech audio for a user of the virtual assistant. “The intelligent automated assistant systems of various embodiments of the present invention can unify, simplify, and improve the user's experience with respect to many different applications and functions of an electronic device, and with respect to services that may be available over the Internet,” as recognized by Gruber. (Gruber, ¶ [0010]).
Regarding claim 19, the rejection of claim 11 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. However, Fuegen, Liu, and Avijeet fail to expressly recite wherein outputting a request for a virtual assistant action is performed by making a request to a web application programing interface (web API), the request having as an argument, the variable value with the highest probability.
The relevance of Gruber is described above with relation to claim 9. Regarding claim 19, Gruber teaches wherein outputting a request for a virtual assistant action is performed by making a request to a web application programing interface (web API) (“Once assistant 1002 has determined the user’s intent, using the techniques described herein, assistant 1002 can call external services 1340 {wherein outputting a request for a virtual assistant action is performed}” that “interface with functionality and applications on a device via APIs… to perform functions and operations that might otherwise be initiated using a conventional user interface on the device,” where “functions and operations may include, for example, setting an alarm, making a telephone call, sending a text message or email message, adding a calendar event, and the like. Such functions and operations may be performed as add-on functions in the context of a conversational dialog between a user and assistant 1002. {is performed by making a request to the web API}”; Gruber, ¶ [0087]), the request having as an argument, the variable value with the highest probability (“active ontologies 1050 may be operable to perform and/or implement various types of functions, operations, actions,” thus an argument in the context of a web API, and “some nodes of an active ontology may correspond to domain concepts such as restaurant and its property restaurant name,” where the association of “restaurant and its property restaurant name” is a named entity {variable value with the highest probability; Gruber, ¶ [0197], [0206]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to incorporate the teachings of Gruber to include wherein outputting a request for a virtual assistant action is performed by making a request to a web application programing interface (web API), the request having as an argument, the variable value with the highest probability. “The intelligent automated assistant systems of various embodiments of the present invention can unify, simplify, and improve the user's experience with respect to many different applications and functions of an electronic device, and with respect to services that may be available over the Internet,” as recognized by Gruber. (Gruber, ¶ [0010]).
Regarding claim 20, the rejection of claim 19 is incorporated. Fuegen, Liu, and Avijeet disclose all of the elements of the current invention as stated above. However, Fuegen, Liu, and Avijeet fail to expressly recite further comprising receiving a response from the web API; and synthesizing speech audio containing information from the web API response.
The relevance of Gruber is described above with relation to claim 9. Regarding claim 20, Gruber teaches further comprising receiving a response from the web API (“assistant 1002 can call external services 1360 that interface with functionality and applications on a device via APIs… to perform functions and operations that might otherwise be initiated using a conventional user interface on the device,” where said functions and operations include “output data/information which may be generated by intelligent automated assistant 1002... [including] Speech output... [such as] Synthesized speech”; Gruber, ¶ [0087], [0148], [0152], [0153]); and synthesizing speech audio containing information from the web API response (The assistant can include “taking input from the user as voice spoken to the assistant and sending output from the assistant to the user, for example as synthesized speech, in reply.”; Gruber, ¶ [0097]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the end-to-end spoken language understanding system of Fuegen, as modified by the joint intent detection/slot filling/language modeling systems of Liu, and as modified by the unified neural network systems of Avijeet, to further incorporate the teachings of Gruber to include further comprising receiving a response from the web API; and synthesizing speech audio containing information from the web API response. “The intelligent automated assistant systems of various embodiments of the present invention can unify, simplify, and improve the user's experience with respect to many different applications and functions of an electronic device, and with respect to services that may be available over the Internet,” as recognized by Gruber. (Gruber, ¶ [0010]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Solomon (U.S. Pat. App. Pub. No. 20180233141) discloses an intelligent natural language assistant for receiving natural language user input from the user, parsing the user input at an intent handler to determine an intent template with slots, populating the slots in the intent template with information from user input, and performing resolution on the intent template to partially resolve unresolved information.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Sean E. Serraguard whose telephone number is (313)446-6627. The examiner can normally be reached 07:00-17:00 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel C. Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Sean E Serraguard/Primary Examiner, Art Unit 2657
Prosecution Insights