DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This action is in reply to the claims filed on 22 November 2024.
Claims 1-20 are currently pending and have been examined.
Claim Objections
Claims 5-9 are objected to because of the following informalities:
Claim 5 recites “storing, in the memory bank and for each of one or more of the plurality of…”. This appears to be a typographical error of “storing, in the memory bank, for each of one or more of the plurality of…”.
Claims 6-9 are dependent on claim 5. Therefore claims 6-9 inherit the deficiencies of claim 5.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., an abstract idea) without significantly more.
Independent claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claims regard a process that, as drafted under its broadest reasonable interpretation, covers performance of the limitations as a mental process and mathematics, but for the recitation of generic computer hardware in the case of claim 1, 19, and 20 (e.g., an encoder neural network, a prediction neural network and a memory bank (claim 1, 19, and 20); one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform the method (claim 19); and one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the method (claim 20)).
In regards to the processing of independent claims 1-20, the claimed functionality could be practiced as a mental process in the following manner:
obtaining audio data representing a spoken utterance; (a human can receive data)
processing the audio data to generate an embedding of the spoken utterance; (a human can mentally process data to create a mathematical vector) and
processing the embedding of the spoken utterance to generate a prediction about the spoken utterance, (a human can mentally receive data and generate a prediction based on the data) the processing comprising:
maintaining respective embeddings for a plurality of preceding spoken utterances that were previously processed; (a human can mentally maintain vectors using pen and paper)
determining, using the embedding of the spoken utterance and the respective embeddings of the preceding spoken utterances, one or more embeddings of respective preceding spoken utterances that are relevant to generating the prediction about the spoken utterance, wherein the one or more relevant embeddings are a proper subset of the embeddings maintained; (a human can mentally look at data and determine which data is relevant to a prediction) and
processing (i) the embedding of the spoken utterance and (ii) the respective embeddings of the one or more determined preceding spoken utterances to generate the prediction about the spoken utterance. (a human can mentally receive data and make a prediction based on the data)
This judicial exception is not integrated into a practical application. Outside of the identified abstract idea, the claimed invention only includes an encoder neural network, a prediction neural network and a memory bank (claim 1, 19, and 20); one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform the method (claim 19); and one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the method (claim 20), which amount to no more than mere instructions to implement an otherwise abstract idea using generic components. Note that the computing components here are being used for their ordinary purpose of executing a program to carry out a process (i.e., being used as a tool) instead of being improved as a tool.
Independent claims 1, 19, and 20 do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the above additional element(s) merely use a computer as a tool to perform an abstract idea, which does not render a claim as being significantly more than the judicial exception.
Therefore, claims 1, 19, and 20 are not eligible subject matter under 35 USC 101.
The remaining dependent claims fail to add patent eligible subject matter to their respective parent claims:
Claim 2 further details the process of making a vector that can be mentally understood by a human. Claim 2 further include(s) the additional element(s): training an encoder neural network. This/these additional element(s) alone or in ordered combination does no more than merely use a computer as a tool to perform an abstract idea (see MPEP 2106.05(f)), which does not integrate the claim(s) into a practical application nor does it render a claim as being significantly more than the abstract idea.
Claim 3 further regards performing a mathematical attention process that can be mentally understood by a human.
Claim 4 further regards storing mathematical vectors in a way that could be performed by a human with a pen and paper.
Claim 5 further regards processing mathematical vectors in a way that can be mentally understood by a human.
Claim 6 further regards processing mathematical vectors in a way that can be mentally understood by a human.
Claim 7 further regards processing mathematical vectors in a way that can be mentally understood by a human.
Claim 8 further regards processing mathematical vectors using attention data in a way that can be mentally understood by a human.
Claim 9 further regards processing mathematical vectors using attention data in a way that can be mentally understood by a human.
Claim 10 further regards making a determination of unneeded data in a way that can be mentally understood by a human and be performed by a human with pen and paper.
Claim 11 further regards making a determination of unneeded data in a way that can be mentally understood by a human.
Claim 12 further regards making a determination of unneeded data in a way that can be mentally understood by a human.
Claim 13 further regards determining where various mathematical vectors can be stored in a way that can be mentally performed by a human with pen and paper. Claim 13 further include(s) the additional element(s): different memory banks. This/these additional element(s) alone or in ordered combination does no more than merely use a computer as a tool to perform an abstract idea (see MPEP 2106.05(f)), which does not integrate the claim(s) into a practical application nor does it render a claim as being significantly more than the abstract idea.
Claim 14 further regards managing data in a way that can be mental understood by a human with pen and paper.
Claim 15 further regards making a determination from data that can be mentally understood by a human with pen and paper.
Claim 16 further details the process of making a prediction that can be mentally understood by a human.
Claim 17 further details the data that can be mentally understood by a user. Claim 17 further include(s) the additional element(s): a same particular device. This/these additional element(s) alone or in ordered combination does no more than merely use a computer as a tool to perform an abstract idea (see MPEP 2106.05(f)), which does not integrate the claim(s) into a practical application nor does it render a claim as being significantly more than the abstract idea.
Claim 18 further regards storing data in a way that can be done by a human mentally with pen and paper and further processing that data in a way that can be mentally understood by a human.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 3-4, and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen (US 20170372200 A1) in view of Nair (US 20220101837 A1).
Regarding claim 1, Chen teaches a method comprising:
obtaining audio data representing a spoken utterance; (see at least Paragraph [0118] “at block 604, a system as described herein can apply the end-to-end memory network model 220 to multiple turns of input in a conversation.”; Paragraph [0032] “input devices, such as […] microphone(s),”)
processing the audio data using an encoder neural network to generate an embedding of the spoken utterance; (see at least Paragraph [0084] “The model, e.g., model 220, can embed inputs, e.g., utterances, into a continuous space”; Paragraph [0117] “at block 602, a system as described herein can use a neural network (NN) to build an end-to-end memory network [….] 220,”) and
processing the embedding of the spoken utterance using a prediction neural network to generate a prediction about the spoken utterance, (see at least Paragraph [0091] “A goal of the language understanding model described herein is to assign a semantic tags for each word in the current utterance. That is, given c=w.sub.1; :::; w.sub.n, the model can predict y=y.sub.1; :::; y.sub.n […] operation engine 218 can use the Elman RNN architecture”; Examiner notes the model can predict a tag for each word) the processing comprising:
maintaining, in a memory bank, respective embeddings for a plurality of preceding spoken utterances that were previously processed by the encoder neural network; (see at least Paragraph [0084] “The model, e.g., model 220, can embed inputs, e.g., utterances, into a continuous space and store historic inputs, e.g., historic utterances, x embeddings to the memory”)
determining, using the embedding of the spoken utterance and the respective embeddings of the preceding spoken utterances, one or more embeddings of respective preceding spoken utterances that are relevant to generating the prediction about the spoken utterance, wherein the one or more relevant embeddings are a (see at least Paragraph [0087] “In the embedding space, in various examples operation engine 218 can estimate an attention vector […] operation engine 218 can calculate similarity between the current utterance u and each memory vector m.sub.i by taking the inner product followed by a softmax”; Paragraph [0088] “In order to encode the knowledge from history, operation engine 218 can sum a history vector h, which is a sum over the memory embeddings weighted by attention distribution”; Examiner notes that the historical utterance embeddings are weighted so that important embeddings are given all the weight, and unimportant embeddings are given negligible weight in the prediction.) and
processing (i) the embedding of the spoken utterance and (ii) the respective embeddings of the one or more determined preceding spoken utterances to generate the prediction about the spoken utterance. (see at least Paragraph [0082] “For language understanding, e.g., spoken language understanding an end-to-end memory network model can take a discrete set of history utterances {x.sub.i} that are stored in memory, a current utterance c=w.sub.1; :::; w.sub.T, and outputs corresponding semantic tags y=y.sub.1; :::; y.sub.T that include intent and slot information”)
Chen does not teach:
determining, using the embedding of the spoken utterance and the respective embeddings of the preceding spoken utterances, one or more embeddings of respective preceding spoken utterances that are relevant to generating the prediction about the spoken utterance, wherein the one or more relevant embeddings are a proper subset of the embeddings maintained in the memory bank.
However Nair teaches:
determining one or more embeddings of respective spoken utterances that are relevant to generating the prediction, wherein the one or more relevant embeddings are a proper subset of the embeddings maintained in the memory bank. (see at least Paragraph [0051] “in step S510, the intent prediction module 224 obtains utterances by selecting a subset of utterances from among the larger set of utterances, within the topic-tagged utterances 132 stored in DB 106 […] the selected subset of utterances obtained in step S510 would include only a portion of all the utterances”; Paragraph [0053] “in step S520, the embedding module 410 of the intent prediction module 224 may encode each utterance within the subset of utterances obtained in step S510.” Of Nair)
This step of Nair is applicable to the method of Chen as they both share characteristics and capabilities, namely, they are directed to performing a prediction using utterances. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to have modified the subset of Chen to be a proper subset as taught by Nair. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to modify Chen in order to organize and identify large numbers of utterances (see paragraph [0036] of Nair).
Regarding claim 3, Chen in view of Nair teaches the method of claim 1. Chen further teaches:
wherein processing (i) the embedding of the spoken utterance and (ii) the respective embeddings of the one or more determined preceding spoken utterance to generate the prediction about the spoken utterance comprises applying a cross-attention mechanism between the embedding of the spoken utterance and the respective embeddings of the one or more determined preceding spoken utterance. (see at least Paragraph [0087] “In the embedding space, in various examples operation engine 218 can estimate an attention vector […] operation engine 218 can calculate similarity between the current utterance u and each memory vector m.sub.i by taking the inner product followed by a softmax”)
Regarding claim 4, Chen in view of Nair teaches the method of claim 1. Chen further teaches:
wherein the memory bank stores, for each of one or more of the plurality of preceding spoken utterances, one or more unaggregated embeddings generated by the encoder neural network in response to processing the preceding spoken embedding. (see at least Paragraph [0086] “Memory Representation: To store the knowledge from previous turns, operation engine 218 can convert each input, e.g., utterance, from a previous turn, as a vector x.sub.i, into a memory vector m.sub.i with dimension d by encoding the inputs in a continuous space through W.sub.mem.”)
Regarding claim 17, Chen in view of Nair teaches the method of claim 1. Chen further teaches:
wherein one or more of:
the spoken utterance and each preceding spoken utterance were spoken by a same speaker, (see at least Paragraph [0084] “the historic inputs model 220 can store, include all historic inputs from the source of the input”) or
the spoken utterance and each preceding spoken utterance were captured by a same particular device, and the encoder neural network and prediction neural network are executed on the particular device.
Regarding claim 18, Chen in view of Nair teaches the method of claim 1. Chen further teaches:
maintaining, for each of one or more of the plurality of preceding spoken utterances whose embeddings are stored by the memory bank, audio data representing the preceding spoken utterance, (see at least Paragraph [0084] “The model, e.g., model 220, can embed inputs, e.g., utterances, into a continuous space and store historic inputs, e.g., historic utterances, x embeddings to the memory.”)
wherein processing (i) the embedding of the spoken utterance and (ii) the respective embeddings of the one or more determined preceding spoken utterance to generate the prediction about the spoken utterance comprises:
further processing the respective audio data representing the one or more determined preceding spoken utterances to generate the prediction about the spoken utterance. (see at least Paragraph [0082] “For language understanding, e.g., spoken language understanding an end-to-end memory network model can take a discrete set of history utterances {x.sub.i} that are stored in memory, a current utterance c=w.sub.1; :::; w.sub.T, and outputs corresponding semantic tags y=y.sub.1; :::; y.sub.T that include intent and slot information”; Paragraph [0087] “operation engine 218 can calculate similarity between the current utterance u and each memory vector m.sub.i by taking the inner product followed by a softmax as follows in Equation 3:”)
Claim 19:
Claim(s) 19 is/are directed to a system. Claim(s) 19 recite limitations parallel in nature as those addressed above for claim(s) 1, which are directed towards a method. Claim(s) 19 is/are therefore rejected for the same reasons as set above for claim(s) 1, respectively. Claim 19 further recites one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform the method (see at least Paragraph [0047] “computer-readable media 114 can store instructions executable by the processing unit(s) 112 that, as discussed above, can represent a processing unit incorporated in computing device 102.” Of Chen).
Claim 20:
Claim(s) 20 is/are directed to a non-transitory computer storage media. Claim(s) 20 recite limitations parallel in nature as those addressed above for claim(s) 1, which are directed towards a method. Claim(s) 20 is/are therefore rejected for the same reasons as set above for claim(s) 1, respectively. Claim 20 further recites one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the method (see at least Paragraph [0047] “computer-readable media 114 can store instructions executable by the processing unit(s) 112 that, as discussed above, can represent a processing unit incorporated in computing device 102.” of Chen).
Claim(s) 2 and 5-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen (US 20170372200 A1) in view of Nair (US 20220101837 A1) in further view of Kim (US 20190392816 A1).
Regarding claim 2, Chen in view of Nair teaches the method of claim 1.
wherein the encoder neural network has been configured through training to generate an embedding of the spoken utterance that encodes both (i) lexical features of the spoken utterance (Paragraph [0086] “To store the knowledge from previous turns, operation engine 218 can convert each input, e.g., utterance, from a previous turn, as a vector x.sub.i, into a memory vector m.sub.i with dimension d by encoding the inputs in a continuous space through W.sub.mem”; Paragraph [0091] “A goal of the language understanding model described herein is to assign a semantic tags for each word in the current utterance. That is, given c=w.sub.1; :::; w.sub.n, the model can predict y=y.sub.1; :::; y.sub.n where each tag y.sub.i is aligned with the word w.sub.i.”)
Chen in view of Nair does not teach:
wherein the encoder neural network has been configured through training to generate an embedding of the spoken utterance that encodes both (i) lexical features of the spoken utterance and (ii) paralinguistic features of the spoken utterance.
However, Kim teaches:
wherein the encoder neural network has been configured through training to generate an embedding of the spoken utterance that encodes both (i) lexical features of the spoken utterance and (ii) paralinguistic features of the spoken utterance. (see at least Paragraph [0133] “The utterance feature extraction unit 320 may extract the utterance feature of the speaker based on the analysis result of the speech analysis unit 315. As an example, the utterance feature of the speaker may include one or more of word/topic, stem/ending, or utterance speed/style. As another example, the utterance feature of the speaker may include one or more of accent, accent, level of voice, intensity, or length” of Kim)
This step of Kim is applicable to the method of Chen as they both share characteristics and capabilities, namely, they are directed to processing speech embeddings. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to have modified the embeddings of Chen to incorporate paralinguistic features as taught by Kim. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to modify Chen in order to perform proper speech recognition based on the dialect, etc. of the user (see paragraph [0004] of Kim).
Regarding claim 5, Chen in further view of Nair teaches the method of claim 1. Chen further teaches:
wherein maintaining the embeddings for the plurality of preceding spoken utterances comprises
processing, by the encoder neural network, the preceding spoken utterance to generate a full embedding of the preceding spoken utterance; (Paragraph [0086] “To store the knowledge from previous turns, operation engine 218 can convert each input, e.g., utterance, from a previous turn, as a vector x.sub.i, into a memory vector m.sub.i with dimension d by encoding the inputs in a continuous space through W.sub.mem.”; Paragraph [0074] “The modules of the operation engine 218 stored on computer-readable media 204 can include one or more modules, […] for operating neural networks”) and
processing the full embedding of the preceding spoken utterance to generate the aggregated embedding of the preceding spoken utterance, (see at least Paragraph [0088] “In order to encode the knowledge from history, operation engine 218 can sum a history vector h, which is a sum over the memory embeddings weighted by attention distribution”)
Chen does not teach:
storing, in the memory bank, a respective aggregated embedding;
wherein the aggregated embedding has a lower dimensionality than the full embedding.
However, Nair teaches:
storing, in the memory bank, a respective aggregated embedding;
wherein the aggregated embedding has a lower dimensionality than the full embedding. (Paragraph [0055] “In step S530, nonlinear dimensionality reduction is performed. For example, in step S530, the nonlinear dimensionality reduction module 420 of the intent prediction module 224 may use uniform manifold approximation and projection (UMAP) to reduce the dimensionality of the embeddings generated in step S520 […] the dimensionality of the obtained vectors may be reduced from 512 dimensions to 40-100 dimensions in order to make the obtained vectors (i.e., the obtained embeddings) more suitable for clustering.” Of Nair)
The motivation for making this modification to the teachings of Chen is the same as that set forth above, in the rejection of claim 1.
Chen in view of Nair does not teach:
wherein maintaining the embeddings for the plurality of preceding spoken utterances comprises storing, in the memory bank, for each of one or more of the plurality of preceding spoken utterances, a respective aggregated embedding that has been generated by performing operations.
However, Kim teaches:
storing, in the memory bank, for each of one or more of the plurality of spoken utterances, a respective embedding. (see at least Paragraph [0159] “When receiving word data, the control unit 325 may provide word data as an input of the first learning model 330 to […] obtain word embedding information. […] The control unit 325 may store the obtained word embedding information in a memory.” Of Kim)
This step of Kim is applicable to the method of Chen as they both share characteristics and capabilities, namely, they are directed to processing speech embeddings. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to have modified the method of Chen to incorporate storing the embeddings in memory as taught by Kim. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to modify Chen in order to acquire and update the embedding information (see paragraph [0149] of Kim).
Regarding claim 6, Chen in view of Nair in further view of Kim teaches the method of claim 5. Chen further teaches:
the full embedding comprises a plurality of embedding elements that each correspond to a respective element of the preceding spoken utterance; (see at least Paragraph [0086] “To store the knowledge from previous turns, operation engine 218 can convert each input, e.g., utterance, from a previous turn, as a vector x.sub.i, into a memory vector m.sub.i with dimension d by encoding the inputs in a continuous space through W.sub.mem.”; Examiner notes that the values of the vector correspond to traits in the utterance.) and
processing the full embedding of the preceding spoken utterance to generate the aggregated embedding of the preceding spoken utterance comprises combining the plurality of embedding elements.(see at least Paragraph [0088] “In order to encode the knowledge from history, operation engine 218 can sum a history vector h, which is a sum over the memory embeddings weighted by attention distribution”)
Regarding claim 7, Chen in view of Nair in further view of Kim teaches the method of claim 5. Chen further teaches:
wherein maintaining the embeddings for the plurality of preceding spoken utterances comprises iteratively performing operations (see at least Paragraph [0118] “a system as described herein can apply the end-to-end memory network model 220 to multiple turns of input in a conversation.”) comprising:
identifying, from a set of full embeddings stored by the memory bank, one or more full embeddings to aggregate; (see at least Paragraph [0087] “operation engine 218 can calculate similarity between the current utterance u and each memory vector m.sub.i by taking the inner product followed by a softmax”) and
processing the one or more identified full embeddings to generate respective aggregated embeddings. (Paragraph [0088] “In order to encode the knowledge from history, operation engine 218 can sum a history vector h, which is a sum over the memory embeddings weighted by attention distribution”; Examiner notes the memory vectors that are not similar to the current utterance are given nominal weights and have negligible impact on the history vector.)
Regarding claim 8, Chen in view of Nair in further view of Kim teaches the method of claim 7. Chen further teaches:
wherein identifying, from a set of full embeddings stored by the memory bank, one or more full embeddings to aggregate comprises:
identifying a particular full embedding based on one or more of:
one or more attention values corresponding to the particular full embedding that were computed during respective executions of the prediction neural network, (see at least Paragraph [0087] “operation engine 218 can estimate an attention vector based on input and context vectors. In some examples, operation engine 218 can learn attention vectors using various machine learning algorithms. […] operation engine 218 can calculate similarity between the current utterance u and each memory vector m.sub.i by taking the inner product followed by a softmax […] where softmax […] can be viewed as attention distribution for modeling knowledge carryover”; Paragraph [0088] “In order to encode the knowledge from history, operation engine 218 can sum a history vector h, which is a sum over the memory embeddings weighted by attention distribution”) or
an amount of time since the full embedding was added to the memory bank.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen (US 20170372200 A1) in view of Nair (US 20220101837 A1) in further view of Kim (US 20190392816 A1) in further view of Wu (US 20220270587 A1).
Regarding claim 9, Chen in view of Nair in further view of Kim teaches the method of claim 8. Chen does in view of Nair in further view of Kim not teach:
wherein identifying a particular full embedding based on one or more attention values corresponding to the particular full embedding that were computed during respective executions of the prediction neural network comprises:
identifying the particular full embedding based on one or more of:
a measure of central tendency of the one or more attention values corresponding to the particular full embedding, or
a maximum attention value from the one or more attention values corresponding to the particular full embedding.
However, Wu teaches:
identifying the particular embedding based on one or more of:
a maximum attention value from the one or more attention values corresponding to the particular embedding (see at least Paragraph [0125] “In some embodiments, the speech synthesis apparatus determines an ith target symbol corresponding to the maximum attention value from the ith group of attention values”; Paragraph [0126] “The symbol corresponding to the maximum attention value of the ith group of attention values is taken as the ith target symbol”; Paragraph [0103] “The speech synthesis apparatus converts each symbol in the symbol sequence into a vector to obtain the initial feature vector set, and obtains the feature vector set.”)
This step of Wu is applicable to the method of Chen as they both share characteristics and capabilities, namely, they are directed to analyzing speech using attention values. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to have modified the step of identifying a full embedding of Chen to incorporate identifying the particular embedding based on a maximum attention value as taught by Wu. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to modify Chen in order to get the correct data needed to perform an operation (i.e. pronunciation) completely (see paragraph [0132] of Wu).
Claim(s) 10-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen (US 20170372200 A1) in view of Nair (US 20220101837 A1) in further view of Klingler (US 20230360641 A1).
Regarding claim 10, Chen in view of Nair teaches the method of claim 1. Chen further teaches:
wherein maintaining the embeddings for the plurality of preceding spoken utterances comprises iteratively performing operations comprising:
identifying one or more embeddings as candidate for pruning; (see at least Paragraph [0087] “operation engine 218 can calculate similarity between the current utterance u and each memory vector m.sub.i by taking the inner product followed by a softmax”; Paragraph [0023] “Attention model as used herein sets the amount to weight to apply to inputs from the stored inputs. […] In order to improve user experience, end-to-end memory networks for contextual, e.g., multi-turn, language understanding can augment or be incorporated in conversation understanding systems to learn what input is important and identify that some input can be discarded”; Examiner notes unimportant embeddings are given small weights making them inconsequential.) and
Chen in view of Nair does not teach:
removing the one or more identified embeddings from the memory bank.
However, Klingler teaches:
removing the one or more identified embeddings from the memory bank. (see at least Paragraph [0046] “deleting the older embeddings can also be advantageous in freeing memory resources” of Klingler)
This step of Klingler is applicable to the method of Chen as they both share characteristics and capabilities, namely, they are directed to managing embeddings for acoustic models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to have modified the method of Chen to incorporate removing embeddings from the memory bank as taught by Klingler. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to modify Klingler in order to reduce power consumption and improve battery life (see paragraph [0046] of Klingler).
Regarding claim 11, Chen in view of Nair in further view of Klingler teaches the method of claim 10. Chen further teaches:
wherein identifying one or more embeddings as candidate for pruning comprises:
identifying a particular embedding based on one or more of:
one or more attention values corresponding to the particular embedding that were computed during respective executions of the prediction neural network, (see at least Paragraph [0087] “operation engine 218 can estimate an attention vector based on input and context vectors. In some examples, operation engine 218 can learn attention vectors using various machine learning algorithms. […] operation engine 218 can calculate similarity between the current utterance u and each memory vector m.sub.i by taking the inner product followed by a softmax […] where softmax […] can be viewed as attention distribution for modeling knowledge carryover”; Paragraph [0088] “In order to encode the knowledge from history, operation engine 218 can sum a history vector h, which is a sum over the memory embeddings weighted by attention distribution”) or
an amount of time since the embedding was added to the memory bank.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen (US 20170372200 A1) in view of Nair (US 20220101837 A1) in further view of Klingler (US 20230360641 A1) in further view of Wu (US 20220270587 A1).
Regarding claim 12, Chen in view of Nair in further view of Klingler teaches the method of claim 11. Chen in view of Nair in further view of Klingler does not teach:
wherein identifying a particular embedding based on one or more attention values corresponding to the particular embedding that were computed during respective executions of the prediction neural network comprises:
identifying the particular embedding based on one or more of:
a measure of central tendency of the one or more attention values corresponding to the particular embedding, or
a maximum attention value from the one or more attention values corresponding to the particular embedding.
However, Wu teaches:
identifying the particular embedding based on one or more of:
a measure of central tendency of the one or more attention values corresponding to the particular embedding, or
a maximum attention value from the one or more attention values corresponding to the particular embedding. (see at least Paragraph [0125] “In some embodiments, the speech synthesis apparatus determines an ith target symbol corresponding to the maximum attention value from the ith group of attention values”; Paragraph [0126] “The symbol corresponding to the maximum attention value of the ith group of attention values is taken as the ith target symbol”; Paragraph [0103] “The speech synthesis apparatus converts each symbol in the symbol sequence into a vector to obtain the initial feature vector set, and obtains the feature vector set.”; of Wu Examiner notes that attention is not given to symbols that do not have the maximum attention value, so they are ignored/discarded.)
This step of Wu is applicable to the method of Chen as they both share characteristics and capabilities, namely, they are directed to analyzing speech using attention values. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to have modified the step of identifying a the particular embedding of Chen to incorporate identifying the particular embedding based on a maximum attention value as taught by Wu. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to modify Chen in order to get the correct data needed to perform an operation (i.e. pronunciation) completely (see paragraph [0132] of Wu).
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen (US 20170372200 A1) in view of Nair (US 20220101837 A1) in further view of Ginart (see attached NPL).
Regarding claim 13, Chen in view of Nair teaches the method of claim 1. Chen in view of Nair does not teach:
wherein maintaining the embeddings for the plurality of preceding spoken utterances comprises:
maintaining embeddings for respective preceding spoken utterances in a plurality of different memory banks that each store embeddings at respective different resolutions.
However, Ginart teaches:
maintaining embeddings for respective preceding spoken utterances in a plurality of different memory banks that each store embeddings at respective different resolutions.(Page 9 “To size the MD embedding layer we apply MDs within individual embedding matrices by partitioning them.” Page 5 “we can apply mixed dimensions within the users and items based on partitions” of Ginart; Examiner notes the memory is partitioned into multiple partitions each with their own dimensions (i.e. resolutions).)
This step of Ginart is applicable to the method of Chen as they both share characteristics and capabilities, namely, they are directed to managing embeddings. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to have modified the method of Chen to incorporate maintaining embeddings in different memory banks based on resolution as taught by Ginart. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to modify Chen in order to reduce how much memory is used by the model (see Page 1: Introduction of Ginart).
Claim(s) 14 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen (US 20170372200 A1) in view of Nair (US 20220101837 A1) in further view of Mei (US 20200218989 A1).
Regarding claim 14, Chen in view of Nair teaches the method of claim 1. Chen in view of Nair does not teach:
wherein maintaining the embeddings for the plurality of preceding spoken utterances comprises:
maintaining data representing a graph that includes (i) a plurality of nodes corresponding to respective preceding spoken utterances and (ii) a plurality edges connecting respective nodes, each edge between a first node and a second node representing a relationship between the preceding spoken utterance corresponding to the first node and the preceding spoken utterance corresponding to the second node.
However, Mei teaches:
wherein maintaining the embeddings for the plurality of preceding spoken utterances comprises:
maintaining data representing a graph that includes (i) a plurality of nodes corresponding to respective preceding spoken utterances and (ii) a plurality edges connecting respective nodes, each edge between a first node and a second node representing a relationship between the preceding spoken utterance corresponding to the first node and the preceding spoken utterance corresponding to the second node. (Paragraph [0070] “a path indicating an association relationship between messages comprised in the sample message sequence can be generated with respect to one of the sample message sequence. For example, each of the messages in the sample message sequence can be added into the path in a chronological order. In some embodiments, a message received from the user can be represented by one of a node and a directed edge in the conversation graph, and a message received from the conversation server can be represented by another one of a node and a directed edge in the conversation graph.”; Paragraph [0055] “Although the conversation 410 is illustrated in a chat via texts, it can be a telephone conversation.”; Fig.4-5 and 8-10 of Mei)
This step of Mei is applicable to the method of Chen as they both share characteristics and capabilities, namely, they are directed to analyzing utterances to make a prediction. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to have modified the method of Chen to incorporate maintaining a graph with nodes and edges as taught by Mei. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to modify Chen in order to handle unexpected queries from users in conversation (see paragraph [0037] of Mei).
Regarding claim 15, Chen in view of Nair in further view of Mei teaches the method of claim 14. Chen in view of Nair does not teach:
wherein determining one or more preceding spoken utterances that are relevant to the prediction about the spoken utterance comprises:
determining a first preceding spoken utterance that is relevant to the prediction about the spoken utterance;
determining a second preceding spoken utterance whose corresponding node in the graph shares an edge with the node corresponding to the first preceding spoken utterance.
However, Mei teaches:
wherein determining one or more preceding spoken utterances that are relevant to the prediction about the spoken utterance (see at least Paragraph [0056] “The right side of FIG. 4 shows a conversation graph 420 based on which the conversation server 430 can provide automatic reply such as the messages N1 to N4 upon receiving the user's messages L1 to L3.”; Paragraph [0055] “Although the conversation 410 is illustrated in a chat via texts, it can be a telephone conversation.”; Fig. 4-5 and 8-10 of Mei; Examiner notes that the system receives a sequence of nodes that match the sequence of nodes in the current conversation to predict what to say to the user next.) comprises:
determining a first preceding spoken utterance that is relevant to the prediction about the spoken utterance; (see at least Paragraph [0056] “The right side of FIG. 4 shows a conversation graph 420 based on which the conversation server 430 can provide automatic reply such as the messages N1 to N4 upon receiving the user's messages L1 to L3.”; Paragraph [0055] “Although the conversation 410 is illustrated in a chat via texts, it can be a telephone conversation.”; Fig. 4-5 and 8-10 of Mei; Examiner notes that the system receives a sequence of nodes that match the sequence of nodes in the current conversation to predict what to say to the user next.) and
determining a second preceding spoken utterance whose corresponding node in the graph shares an edge with the node corresponding to the first preceding spoken utterance. (see at least Paragraph [0056] “The right side of FIG. 4 shows a conversation graph 420 based on which the conversation server 430 can provide automatic reply such as the messages N1 to N4 upon receiving the user's messages L1 to L3.”; Paragraph [0055] “Although the conversation 410 is illustrated in a chat via texts, it can be a telephone conversation.”; Fig. 4-5 and 8-10 of Mei; Examiner notes that the system receives a sequence of nodes that match the sequence of nodes in the current conversation to predict what to say to the user next.)
The motivation for making this modification to the teachings of Chen is the same as that set forth above, in the rejection of claim 14.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen (US 20170372200 A1) in view of Nair (US 20220101837 A1) in further view of Rongali (see attached NPL).
Regarding claim 16, Chen in view of Nair teaches the method of claim 1. Chen in view of Nair does not teach:
wherein the prediction neural network does not process a transcription of the spoken utterance when generating the prediction about the spoken utterance.
However, Rongali teaches:
wherein the prediction neural network does not process a transcription of the spoken utterance when generating the prediction about the spoken utterance. (see at least Page 1 “Traditional SLU systems consist of a two-stage pipeline, an Automatic Speech Recognition (ASR) component that processes customer speech and generates a text transcription (ex. play the song watermelon sugar), followed by a Natural Language Understanding (NLU) component that maps the transcription to an actionable hypothesis consisting of intents and slots (ex. Intent: PlaySong, Slots: SongName watermelon sugar). An end-to-end (E2E) system that goes directly from speech to the hypothesis would help make the SLU system smaller and faster, allowing it to be stored on an edge device” of Rongali)
This step of Rongali is applicable to the method of Chen as they both share characteristics and capabilities, namely, they are directed to speech processing applications. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to have modified the method of Chen to incorporate skipping the transcription process as taught by Rongali. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to modify Chen in order to better optimize the pipeline and eliminate cascading errors (see Page 1: Introduction of Rongali).
Additionally, in regard to claim 16, the Examiner further notes the recited “when” on line 2 does not move to distinguish the claimed invention from the cited art. This phrase is a conditional/contingent limitation with the noted “not process…” step(s) not necessarily performed. The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. Language that suggests or makes optional but does not require steps to be performed or does not limit a claim to a particular structure does not limit the scope of a claim or claim limitation. [See Ex parte Schulhauser, Appeal 2013-007847 (PTAB April 28, 2016) for an analysis of contingent claim limitations in the context of both method claims and system claims.; MPEP §2111.04 II].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIELLE ELIZABETH ZEVITZ whose telephone number is (703)756-1070. The examiner can normally be reached Mo-Th 10am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Lynda Jasmin can be reached at (571) 272-6782. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIELLE ELIZABETH ZEVITZ/Examiner, Art Unit 3629
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655
Prosecution Insights