CTNF 18/430,586 CTNF 97153 DETAILED ACTION Notice of AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Priority Regarding Indian Patent App. No. IN202321006562, filed Feb. 1, 2023, receipt is acknowledged of certified copies of papers required by 37 CFR 1.55. Information Disclosure Statement Except as set forth below, the IDS submitted on 2/20/2025 has been considered. The information disclosure statement filed 2/20/2025 fails to comply with 37 CFR 1.98(a)(2), which requires a legible copy of each cited foreign patent document; each non-patent literature publication or that portion which caused it to be listed; and all other information or that portion which caused it to be listed. It has been placed in the application file, but the information referred to therein has not been considered. A copy of NPL32 (Krotov, et al., “Large associative memory problem in neurobiology and machine learning”) has not been provided. Therefore, this NPL32 has not been considered. Drawings 06-22 AIA The drawings are objected to because Figures 3 and 5 should be corrected to comply with the applicable sections of 37 CFR 1.84 set forth below. In particular, such figures should be drawings using India ink or its equivalent. (a) Drawings. There are two acceptable categories for presenting drawings in utility and design patent applications. (1) Black ink. Black and white drawings are normally required. India ink, or its equivalent that secures solid black lines, must be used for drawings; or Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance. Claim Rejections - 35 USC § 112 The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. Claims 7-8 and 10-15 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention. Claim 7 recites a particular mathematical equation PNG media_image1.png 52 256 media_image1.png Greyscale However, claim 7 does not specify what each of the variables means, and therefore the metes and bounds of the claim are not clear. See MPEP 2173.02 . The examiner suggests amending the claim to recite descriptions for the variables as set forth on page 17, lines 10-20 of the instant specification. Claim 8 recites “wherein the feature mapping maps a key and a query from a first dimension to a second dimension, and wherein the feature mapping approximates a kernel.” However, claim 8 depends from claim 5, which recites “a feature mapping of the key and a transposed value” and “a feature mapping of the key”, so it is unclear which “feature mapping” claim 8 is referring to. MPEP 2173.02 explains that “if the language of a claim, given its broadest reasonable interpretation, is such that a person of ordinary skill in the relevant art would read it with more than one reasonable interpretation, then a rejection under 35 U.S.C. 112(b) ... is appropriate.” Here, because “the feature mapping” could be interpreted as pertaining to either of the “feature mapping” options of claim 5, this limitation is indefinite. Claim 10 recites “wherein the feature mapping is a positive random feature mapping...” However, claim 10 depends from claim 5, which recites “a feature mapping of the key and a transposed value” and “a feature mapping of the key”, so it is unclear which “feature mapping” claim 10 is referring to. MPEP 2173.02 explains that “if the language of a claim, given its broadest reasonable interpretation, is such that a person of ordinary skill in the relevant art would read it with more than one reasonable interpretation, then a rejection under 35 U.S.C. 112(b) ... is appropriate.” Here, because “the feature mapping” could be interpreted as pertaining to either of the “feature mapping” options of claim 5, this limitation is indefinite. Claim 11 recites “wherein the feature mapping is a positive random feature mapping...” However, claim 11 depends from claim 5, which recites “a feature mapping of the key and a transposed value” and “a feature mapping of the key”, so it is unclear which “feature mapping” claim 11 is referring to. MPEP 2173.02 explains that “if the language of a claim, given its broadest reasonable interpretation, is such that a person of ordinary skill in the relevant art would read it with more than one reasonable interpretation, then a rejection under 35 U.S.C. 112(b) ... is appropriate.” Here, because “the feature mapping” could be interpreted as pertaining to either of the “feature mapping” options of claim 5, this limitation is indefinite. Claim 12 recites “wherein the feature mapping is generated by: ...” and “generating the feature mapping from at least four variables...” However, claim 12 depends from claim 5, which recites “a feature mapping of the key and a transposed value” and “a feature mapping of the key”, so it is unclear which “feature mapping” claim 12 is referring to. MPEP 2173.02 explains that “if the language of a claim, given its broadest reasonable interpretation, is such that a person of ordinary skill in the relevant art would read it with more than one reasonable interpretation, then a rejection under 35 U.S.C. 112(b) ... is appropriate.” Here, because “the feature mapping” could be interpreted as pertaining to either of the “feature mapping” options of claim 5, this limitation is indefinite. Claim 12 further recites “deriving a value for a parameter from the query and the key.” It’s unclear if “a value” is meant to refer to the same “value” originally introduced in claim 1 or some other value, and therefore it is unclear if the “parameter” is derived from the “value” originally introduced in claim 1 or some other value. Claims 13-15 depend from claim 12 and do not remedy the deficiencies of claim 12 and are therefore rejected for the same reasons explained above with respect to claim 12. Claim 15 recites “determining a value for the parameter using the updated hidden state elements.” It’s unclear if this “a value” is meant to refer to the same “value” originally introduced in claim 1, the “value” of claim 12, or some other value. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-21-aia AIA Claim s 1-4 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over US 20220391706 A1, hereinafter referenced as METZ , in view of Ritter, Samuel, et al. "Been there, done that: Meta-learning with episodic recall." International conference on machine learning . PMLR, 2018, hereinafter referenced as RITTER , and further in view of US 20220138501 A1, hereinafter referenced as EBRAHIMI . Regarding Claim 1 METZ teaches: A method of training a neural network configured to perform a machine learning task by processing a network input to generate a network output, (METZ, para. 0021: “ The training system 100 is configured to train a trainee neural network 110 to perform a machine learning task. The trainee neural network 110 can be configured to process a trainee network input 102 and to generate a trainee network output 112 that represents a prediction about the trainee network input 102 for the machine learning task. Example machine learning tasks for which training system 100 can train the trainee neural network 110 are discussed below.”) wherein the neural network comprises a neural network layer that is configured to process a layer input in accordance with at least a parameter tensor to generate a layer output, the parameter tensor comprising a plurality of network parameters and having a plurality of dimensions each having a respective plurality of indices, (METZ, para. 0022: “The neural network layer 120 can be configured to process a layer input 122 using at least a parameter tensor to generate a layer output 124. The parameter tensor can include multiple network parameters, and can have multiple dimensions each with multiple indices.”) the method comprising: performing, at each of a plurality of iterations: (METZ, para. 0094: “The system can perform steps 204-212 at each of multiple training stages for the trainee neural network.”) performing, using a plurality of training examples, a training step to obtain respective new gradients of a loss function for the machine learning task with respect to each of the plurality of network parameters of the parameter tensor; (METZ, para. 0095: “The system performs, using one or more training examples, a training step to obtain respective new gradients of a loss function for the machine learning task with respect to each of the multiple network parameters of the parameter tensor (step 204).”) for each network parameter of the plurality of network parameters of the parameter tensor: (METZ, para. 0097: “ The system can perform steps 208-212 for each of the network parameters of the parameter tensor of the neural network layer. For example, the system can perform the steps 208-212 for each network parameter in parallel.”) generating an optimizer network input from at least the new gradient with respect to the network parameter; (METZ, para. 0148: “The system performs, using one or more training examples, a training step to obtain respective new gradients of a loss function for the machine learning task with respect to each of the multiple network parameters of the neural network layer (step 404).” METZ, para. 0149: “The system generates an optimizer network input from at least the respective new gradients (step 406).”) processing the optimizer network input using an optimizer neural network, (METZ, para. 0150: “The system processes the optimizer network input using an optimizer neural network to generate an optimizer network output defining one or more hyperparameter values of an optimizer (step 408). The optimizer neural network can be a recurrent neural network configured to maintain an internal state across training stages. For example, the optimizer neural network can be the optimizer neural network 350 described above with reference to FIG. 3. The optimizer can be an Adam optimizer.” applying the update to the network parameter (METZ, para. 0030: “update the network parameter by applying the parameter update 134.”) METZ, para. 0116: “update the network parameters by applying the parameter update 334.”) However, METZ fails to explicitly teach: wherein the optimizer neural network comprises a sequence of one or more cells that each maintain one or more hidden states and wherein the processing comprises, for each cell: generating a cell input for the cell from at least the optimizer network input; and processing the cell input for the cell to generate a cell output defining an update to at least one hidden state of the cell and wherein the processing comprises: obtaining latent embeddings from the cell input, wherein the latent embeddings comprise a query, a key, and a value; generating the cell output from the hidden state using the query of the latent embeddings; and determining an update to the hidden state from the key and value of the latent embeddings; and generating an optimizer network output defining an update for the network parameter using the cell output of a last cell in the sequence of one or more cells However, in a related field of endeavor (meta-learning, see p. 1, section 1), RITTER teaches and makes obvious: wherein the optimizer neural network comprises a sequence of one or more cells that each maintain one or more hidden states and wherein the processing comprises, for each cell: (RITTER, p. 2, section 3: “To execute such learning algorithms the LSTM must store relevant information from the recent history in its cell state . As a result, at the end of the agent’s exposure to a task, the cell state contains the hard-won results of the agent’s exploration. ... To remedy this forgetting problem in L2RL, we propose a simple solution: add an episodic memory system that stores the cell state along with a contextual cue and reinstates that stored cell state when a similar cue is reencountered later .”; Examiner’s Note: As shown in Fig. 1, the LSTM architecture has a series of cells, where each cell has a state (corresponding to recited “hidden state”) corresponding to a hidden layer (i.e., a layer that is not the input or output layer); the METZ-RITTER combination now modifies METZ so that the optimizer network of METZ is an LSTM having cell states as in RITTER; the examiner further notes that METZ at para. 0134 discloses that the optimizer NN can be a LSTM) generating a cell input for the cell from at least the optimizer network input; and (RITTER, p. 3, section 3: PNG media_image2.png 320 386 media_image2.png Greyscale RITTER, p. 4, section 4.1: “In this and all following experiments, we also tried a variant of the episodic memory architecture in which retrieved values from the DND, cep were fed to the LSTM as inputs instead of added to the working memory through the r-gate.”; Examiner’s Note: The LSTM input (corresponding to recited “optimizer network input”) is passed through the input to each cell using equations (3)-(6); the METZ-RITTER combination now modifies METZ to use the cells and hidden states of RITTER, including the additional episodic memory added by RITTER) processing the cell input for the cell to generate a cell output defining an update to at least one hidden state of the cell and wherein the processing comprises: (RITTER, p. 3, section 3: PNG media_image2.png 320 386 media_image2.png Greyscale Examiner’s Note: The output from the previous cell is input into the subsequent cell using equations (3)-(6); the METZ-RITTER combination now modifies METZ to use the cells and hidden states of RITTER, including the additional episodic memory added by RITTER) obtaining latent embeddings from the cell input, wherein the latent embeddings comprise ... a key, and a value; (RITTER, pp. 2-3, section 3: “To implement this proposal, we draw inspiration from recent memory architectures for RL. Pritzel et al. (2017) proposed the differentiable neural dictionary (DND), which stores key/value pairs in each row of an array (see also Blundell et al., 2016). ... Inspired by this success, we implement our cell state-based episodic memory as a DND that stores embeddings of task contexts c as keys and stores LSTM cell states as values.” Examiner’s Note: As shown in Fig. 1, RITTER discloses storing key/value pairs with respect to the cell state; the METZ-RITTER combination now modifies METZ to use the cells and hidden states of RITTER, such that the key/value pairs of a state can be created using the input to such cell) determining an update to the hidden state from the key and value of the latent embeddings; and (RITTER, p. 3, section 3: PNG media_image2.png 320 386 media_image2.png Greyscale Examiner’s Note: the hidden state update is derived using the equations of (3)-(6) which rely on the key/values of the episodic memory; the METZ-RITTER combination now modifies METZ to use the cells and hidden states of RITTER, such that the key/value pairs of a state can be used to update the state) generating an optimizer network output defining an update for the network parameter using the cell output of a last cell in the sequence of one or more cells; and (RITTER, p. 3, section 3: PNG media_image2.png 320 386 media_image2.png Greyscale Examiner’s Note: As shown in Fig. 1, the LSTM output (corresponding to recited “optimizer network output”) is the last layer of the LSTM; the METZ-RITTER combination now modifies METZ to use the cells and hidden states of RITTER, such that the last LSTM layer comprises the optimizer network output that is used to determine the gradient to use for backpropagation as in METZ) Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of METZ with RITTER as explained above. As disclosed by RITTER, one of ordinary skill would have been motivated to do so in order to help train agents to “cope with open-ended environments, which present the agent with an unbound series of tasks.” (p. 1, section 1). As further disclosed by RITTER, one of ordinary skill would have been motivated to do so in order to modify the LSTM cells to “add an episodic memory system that stores the cell state ... inspired in part by evidence that human episodic memory retrieves past working memory states.” (p. 2, section 3). However, METZ and RITTER fail to explicitly teach: where the latent embeddings comprise a query , ... generating the cell output from the hidden state using the query of the latent embeddings; and However, in a related field of endeavor (neural networks, including RNNs, see para. 0004), EBRAHIMI teaches and makes obvious: where the latent embeddings comprise a query , ... (EBRAHIMI, para. 0011: “In some non-limiting embodiments or aspects, the indicator comprises a slice of each flag tensor of a plurality of flag tensors, the plurality of flag tensors comprising a query flag tensor, a key flag tensor, and a value flag tensor . Additionally or alternatively, updating each hidden state segment based on the attention mechanism comprises: concatenating each hidden state segment with each of a plurality of flag vectors from each slice of each flag tensor; determining at least one query vector, at least one key vector, and at least one value vector based on the hidden state segments having the flag vectors concatenated therewith ; and updating each hidden state segment of the plurality of hidden state segments based on the at least one query vector, the at least one key vector, and the at least one value vector.”; EBRAHIMI, para. 0109: “Additionally or alternatively, at least one query vector, at least one key vector, and at least one value vector may be determined based on the hidden state segments having the flag vectors concatenated therewith. Additionally or alternatively, each hidden state segment may be updated based on the query vector(s), the key vector(s), and/or the value vector(s). ”; Examiner’s Note: EBRAHIMI discloses that the query/key/value set of tensors can be applied to RNN architectures; the METZ-RITTER-EBRAHIMI combination now modifies the RNN LSTM of METZ to utilize the query/key/value tensors of EBRAHIMI) generating the cell output from the hidden state using the query of the latent embeddings; and ( EBRAHIMI, para. 0109: “Additionally or alternatively, at least one query vector, at least one key vector, and at least one value vector may be determined based on the hidden state segments having the flag vectors concatenated therewith. Additionally or alternatively, each hidden state segment may be updated based on the query vector(s), the key vector(s), and/or the value vector(s). ”; EBRAHIMI, para. 0125: “Additionally, attention subsystem 604 may multiply these appended segments by a respective (trainable) weight matrix (e.g., W.sub.q, W.sub.k, and W.sub.v) to form query (Q), key (K), and value (V) matrices.” EBRAHIMI, para. 0142: “In some non-limiting embodiments or aspects, attention subsystem 704 may determine attentional weights (e.g., dot-product attention) based on the query (Q), key (K), and value (V) matrices using a multi-head formulation with m heads.” Examiner’s Note: EBRAHIMI discloses that the query/key/value set of tensors can be applied to RNN architectures; the METZ-RITTER-EBRAHIMI combination now modifies the RNN LSTM of METZ to utilize the query/key/value tensors of EBRAHIMI and to update the hidden state using at least the query embeddings of EBRAHIMI) Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of METZ with RITTER and EBRAHIMI as explained above. As disclosed by EBRAHIMI, one of ordinary skill would have been motivated to do so in order to implement a type of attention mechanism into the RNN. (see para. 0109). Regarding Claim 2 METZ, RITTER, and EBRAHIMI disclose the method of claim 1 as explained above. METZ further teaches: wherein the optimizer network input comprises (i) a respective magnitude and (ii) a respective direction of the new gradient of the loss function for the network parameter. (METZ, para. 0066: “As another example, the optimizer network output 152 for each network parameter can include two elements: a first element whose value defines a scalar direction d of the parameter update 134 for the network parameter and a second element whose value defines a magnitude m of the parameter update 134 for the network parameter.”; METZ, para. 0070: “ For example, the training engine 130 can determine the nominal term from the gradient for the network parameter using a hand-designed optimizer such as Adam, Aggregated Momentum (AggMo), or both, and add the nominal term (or a weighted version of the nominal term) to the initial parameter update to generate the final parameter update 134.”) Regarding Claim 3 METZ, RITTER, and EBRAHIMI disclose the method of claim 1 as explained above. However, METZ fails to explicitly teach: wherein the cell input for each cell after a first cell of the sequence is generated from the cell output of a previous cell in the sequence. However, in a related field of endeavor (meta-learning, see p. 1, section 1), RITTER teaches and makes obvious: wherein the cell input for each cell after a first cell of the sequence is generated from the cell output of a previous cell in the sequence. (RITTER, p. 3, section 3: PNG media_image2.png 320 386 media_image2.png Greyscale Examiner’s Note: The output from the previous cell is input into the subsequent cell using equations (3)-(6); the METZ-RITTER-EBRAHIMI combination now modifies METZ to use the cells and hidden states of RITTER, including the additional episodic memory added by RITTER) Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of METZ with RITTER and EBRAHIMI as explained above. As disclosed by RITTER, one of ordinary skill would have been motivated to do so in order to help train agents to “cope with open-ended environments, which present the agent with an unbound series of tasks.” (p. 1, section 1). As further disclosed by RITTER, one of ordinary skill would have been motivated to do so in order to modify the LSTM cells to “add an episodic memory system that stores the cell state ... inspired in part by evidence that human episodic memory retrieves past working memory states.” (p. 2, section 3). Regarding Claim 4 METZ, RITTER, and EBRAHIMI disclose the method of claim 1 as explained above. However, METZ fails to explicitly teach: wherein the optimizer neural network further comprises one or more neural network layers that receive a cell output from a last cell in the sequence and generate the optimizer network output, and wherein generating an optimizer network output defining an update for the network parameter using the cell output of a last cell in the sequence comprises providing the cell output from the last cell to the one or more neural network layers. However, in a related field of endeavor (meta-learning, see p. 1, section 1), RITTER teaches and makes obvious: wherein the optimizer neural network further comprises one or more neural network layers that receive a cell output from a last cell in the sequence and generate the optimizer network output, and (RITTER, p. 3, section 3: PNG media_image2.png 320 386 media_image2.png Greyscale Examiner’s Note: The output from the previous cell is input into the subsequent cell using equations (3)-(6); the METZ-RITTER-EBRAHIMI combination now modifies METZ to use the cells and hidden states of RITTER, such that each LSTM node of METZ has a cell as explicitly taught by RITTER, where the last layer in the METZ LSTM corresponds to the network output and the final cell) wherein generating an optimizer network output defining an update for the network parameter using the cell output of a last cell in the sequence comprises providing the cell output from the last cell to the one or more neural network layers. (RITTER, p. 3, section 3: PNG media_image2.png 320 386 media_image2.png Greyscale Examiner’s Note: The output from the previous cell is input into the subsequent cell using equations (3)-(6); the METZ-RITTER-EBRAHIMI combination now modifies METZ to use the cells and hidden states of RITTER, such that each LSTM node of METZ has a cell as explicitly taught by RITTER, where the last layer in the METZ LSTM corresponds to the network output and the final cell and this final cell is used to generate the network update for backpropagation as in METZ) Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of METZ with RITTER and EBRAHIMI as explained above. As disclosed by RITTER, one of ordinary skill would have been motivated to do so in order to help train agents to “cope with open-ended environments, which present the agent with an unbound series of tasks.” (p. 1, section 1). As further disclosed by RITTER, one of ordinary skill would have been motivated to do so in order to modify the LSTM cells to “add an episodic memory system that stores the cell state ... inspired in part by evidence that human episodic memory retrieves past working memory states.” (p. 2, section 3). Regarding Claim 18 METZ teaches: A method for generating a network output conditioned on a network input, the method comprising: (METZ, para. 0021: “ The training system 100 is configured to train a trainee neural network 110 to perform a machine learning task. The trainee neural network 110 can be configured to process a trainee network input 102 and to generate a trainee network output 112 that represents a prediction about the trainee network input 102 for the machine learning task. Example machine learning tasks for which training system 100 can train the trainee neural network 110 are discussed below.”) receiving a network input; (METZ, para. 0075: “After deployment, the trainee neural network 110 can receive new trainee network inputs 102 and process the new trainee network inputs 102 according to the trained values for the network parameters of the trainee neural network 110 to generate new trainee network outputs 112 for the new trainee network inputs 102.”) processing the network input using an optimizer neural network that is configured to process the network input and generate an optimizer network output, ... (METZ, para. 0030: “At each training stage and for each network parameter of the parameter tensor of the neural network layer 120, the training engine 130 can use the generated trainee network outputs 112 to generate an optimizer network input 132 for the per-parameter optimizer neural network 150, provide the optimizer network input 132 to the per-parameter optimizer neural network 150 for generating an optimizer network output 152, determine a parameter update 134 for the network parameter using the optimizer network output 152, and update the network parameter by applying the parameter update 134.”) generating an optimizer network input from at least the network input; (METZ, para. 0030: “At each training stage and for each network parameter of the parameter tensor of the neural network layer 120, the training engine 130 can use the generated trainee network outputs 112 to generate an optimizer network input 132 for the per-parameter optimizer neural network 150, provide the optimizer network input 132 to the per-parameter optimizer neural network 150 for generating an optimizer network output 152, determine a parameter update 134 for the network parameter using the optimizer network output 152, and update the network parameter by applying the parameter update 134.”) However, METZ fails to explicitly teach: wherein the optimizer neural network comprises a sequence of one or more cells that each maintain one or more hidden states and wherein the processing comprises: performing, for each cell: generating a cell input for the cell from at least the optimizer network input; processing the cell input for the cell to generate a cell output defining an update to at least one hidden state of the cell and wherein the processing comprises: obtaining latent embeddings from the cell input, wherein the latent embeddings comprise a query, a key, and a value; generating the cell output from the hidden state using the query of the latent embeddings; and determining an update to the hidden state from the key and value of the latent embeddings; and generating an optimizer network output using the cell output of a last cell in the sequence of one or more cells. However, in a related field of endeavor (meta-learning, see p. 1, section 1), RITTER teaches and makes obvious: wherein the optimizer neural network comprises a sequence of one or more cells that each maintain one or more hidden states and wherein the processing comprises: (RITTER, p. 2, section 3: “To execute such learning algorithms the LSTM must store relevant information from the recent history in its cell state . As a result, at the end of the agent’s exposure to a task, the cell state contains the hard-won results of the agent’s exploration. ... To remedy this forgetting problem in L2RL, we propose a simple solution: add an episodic memory system that stores the cell state along with a contextual cue and reinstates that stored cell state when a similar cue is reencountered later .”; Examiner’s Note: As shown in Fig. 1, the LSTM architecture has a series of cells, where each cell has a state (corresponding to recited “hidden state”) corresponding to a hidden layer (i.e., a layer that is not the input or output layer); the METZ-RITTER combination now modifies METZ so that the optimizer network of METZ is an LSTM having cell states as in RITTER; the examiner further notes that METZ at para. 0134 discloses that the optimizer NN can be a LSTM) performing, for each cell: generating a cell input for the cell from at least the optimizer network input; (RITTER, p. 3, section 3: PNG media_image2.png 320 386 media_image2.png Greyscale RITTER, p. 4, section 4.1: “In this and all following experiments, we also tried a variant of the episodic memory architecture in which retrieved values from the DND, cep were fed to the LSTM as inputs instead of added to the working memory through the r-gate.”; Examiner’s Note: The LSTM input (corresponding to recited “optimizer network input”) is passed through the input to each cell using equations (3)-(6); the METZ-RITTER combination now modifies METZ to use the cells and hidden states of RITTER, including the additional episodic memory added by RITTER) processing the cell input for the cell to generate a cell output defining an update to at least one hidden state of the cell and wherein the processing comprises: (RITTER, p. 3, section 3: PNG media_image2.png 320 386 media_image2.png Greyscale Examiner’s Note: The output from the previous cell is input into the subsequent cell using equations (3)-(6); the METZ-RITTER combination now modifies METZ to use the cells and hidden states of RITTER, including the additional episodic memory added by RITTER) obtaining latent embeddings from the cell input, wherein the latent embeddings comprise ... a key, and a value; obtaining latent embeddings from the cell input, wherein the latent embeddings comprise ... a key, and a value; (RITTER, pp. 2-3, section 3: “To implement this proposal, we draw inspiration from recent memory architectures for RL. Pritzel et al. (2017) proposed the differentiable neural dictionary (DND), which stores key/value pairs in each row of an array (see also Blundell et al., 2016). ... Inspired by this success, we implement our cell state-based episodic memory as a DND that stores embeddings of task contexts c as keys and stores LSTM cell states as values.” Examiner’s Note: As shown in Fig. 1, RITTER discloses storing key/value pairs with respect to the cell state; the METZ-RITTER combination now modifies METZ to use the cells and hidden states of RITTER, such that the key/value pairs of a state can be created using the input to such cell) determining an update to the hidden state from the key and value of the latent embeddings; and (RITTER, p. 3, section 3: PNG media_image2.png 320 386 media_image2.png Greyscale Examiner’s Note: the hidden state update is derived using the equations of (3)-(6) which rely on the key/values of the episodic memory; the METZ-RITTER combination now modifies METZ to use the cells and hidden states of RITTER, such that the key/value pairs of a state can be used to update the state) generating an optimizer network output using the cell output of a last cell in the sequence of one or more cells. (RITTER, p. 3, section 3: PNG media_image2.png 320 386 media_image2.png Greyscale Examiner’s Note: As shown in Fig. 1, the LSTM output (corresponding to recited “optimizer network output”) is the last layer of the LSTM; the METZ-RITTER combination now modifies METZ to use the cells and hidden states of RITTER, such that the last LSTM layer comprises the optimizer network output that is used to determine the gradient to use for backpropagation as in METZ) Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of METZ with RITTER as explained above. As disclosed by RITTER, one of ordinary skill would have been motivated to do so in order to help train agents to “cope with open-ended environments, which present the agent with an unbound series of tasks.” (p. 1, section 1). As further disclosed by RITTER, one of ordinary skill would have been motivated to do so in order to modify the LSTM cells to “add an episodic memory system that stores the cell state ... inspired in part by evidence that human episodic memory retrieves past working memory states.” (p. 2, section 3). However, METZ and RITTER fail to explicitly teach: where the latent embeddings comprise a query , ... generating the cell output from the hidden state using the query of the latent embeddings; and However, in a related field of endeavor (neural networks, including RNNs, see para. 0004), EBRAHIMI teaches and makes obvious: where the latent embeddings comprise a query , ... (EBRAHIMI, para. 0011: “In some non-limiting embodiments or aspects, the indicator comprises a slice of each flag tensor of a plurality of flag tensors, the plurality of flag tensors comprising a query flag tensor, a key flag tensor, and a value flag tensor . Additionally or alternatively, updating each hidden state segment based on the attention mechanism comprises: concatenating each hidden state segment with each of a plurality of flag vectors from each slice of each flag tensor; determining at least one query vector, at least one key vector, and at least one value vector based on the hidden state segments having the flag vectors concatenated therewith ; and updating each hidden state segment of the plurality of hidden state segments based on the at least one query vector, the at least one key vector, and the at least one value vector.”; EBRAHIMI, para. 0109: “Additionally or alternatively, at least one query vector, at least one key vector, and at least one value vector may be determined based on the hidden state segments having the flag vectors concatenated therewith. Additionally or alternatively, each hidden state segment may be updated based on the query vector(s), the key vector(s), and/or the value vector(s). ”; Examiner’s Note: EBRAHIMI discloses that the query/key/value set of tensors can be applied to RNN architectures; the METZ-RITTER-EBRAHIMI combination now modifies the RNN LSTM of METZ to utilize the query/key/value tensors of EBRAHIMI) generating the cell output from the hidden state using the query of the latent embeddings; and ( EBRAHIMI, para. 0109: “Additionally or alternatively, at least one query vector, at least one key vector, and at least one value vector may be determined based on the hidden state segments having the flag vectors concatenated therewith. Additionally or alternatively, each hidden state segment may be updated based on the query vector(s), the key vector(s), and/or the value vector(s). ”; EBRAHIMI, para. 0125: “Additionally, attention subsystem 604 may multiply these appended segments by a respective (trainable) weight matrix (e.g., W.sub.q, W.sub.k, and W.sub.v) to form query (Q), key (K), and value (V) matrices.” EBRAHIMI, para. 0142: “In some non-limiting embodiments or aspects, attention subsystem 704 may determine attentional weights (e.g., dot-product attention) based on the query (Q), key (K), and value (V) matrices using a multi-head formulation with m heads.” Examiner’s Note: EBRAHIMI discloses that the query/key/value set of tensors can be applied to RNN architectures; the METZ-RITTER-EBRAHIMI combination now modifies the RNN LSTM of METZ to utilize the query/key/value tensors of EBRAHIMI and to update the hidden state using at least the query embeddings of EBRAHIMI) Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of METZ with RITTER and EBRAHIMI as explained above. As disclosed by EBRAHIMI, one of ordinary skill would have been motivated to do so in order to implement a type of attention mechanism into the RNN. (see para. 0109). Regarding Claim 19 METZ teaches: A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations of (METZ, para.0153: “Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.”) The remaining limitations correspond to the method of claim 1, and therefore this claim is rejected for the same reasons explained above with respect to claim 1. Regarding Claim 20 METZ teaches: One or more non-transitory computer-readable media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of (METZ, para.0153: “Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.”) The remaining limitations correspond to the method of claim 1, and therefore this claim is rejected for the same reasons explained above with respect to claim 1 . Allowable Subject Matter Claims 5-17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. However, claims 7-8 and 10-15 would further need to overcome the rejections under 35 U.S.C. 112(b). 13-03-01 AIA The following is a statement of reasons for the indication of allowable subject matter: Claim 5 would be considered allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims because none of the references of record either alone or in combination fairly disclose or suggest the combination of limitations specified in claim 5, including at least: wherein determining an update to the hidden state from the key and value of the latent embeddings comprises: generating an update to the key-value hidden state element from at least a sum of a product between a feature mapping of the key and a transposed value; generating an update to the key hidden state element from at least a feature mapping of the key; and determining the update to the hidden state by: updating the key-value hidden state element by computing a sum of the key-value hidden state element multiplied by a discount factor with the update to the key-value hidden state element, and updating the key hidden state element by computing a sum of the key hidden state element multiplied by the discount factor with the update to the key-value hidden state element. The closest prior art of record discloses: US 20220391706 A1, hereinafter referenced as METZ , teaches the general framework for using a neural network optimizer to train a trainee neural network. (paras. 0021-0022, 0094-0097). Ritter, Samuel, et al. "Been there, done that: Meta-learning with episodic recall." International conference on machine learning . PMLR, 2018, hereinafter referenced as RITTER , teaches the concept of modifying a LSTM to utilize an episodic memory that stores cell state with respect to key-value pairs. (pp. 2-3, section 3 and Fig. 1). US 20220138501 A1, hereinafter referenced as EBRAHIMI , teaches modifying a RNN to utilize query, key, value tensors. (paras. 0011, 0109). However, the examiner has found that the distinct feature of the Applicant's claimed invention over the prior art is the explicit claiming of the aforementioned limitations in combination with all the other limitations as specified in claim 5. Therefore, because these features are not found in the prior art, claim 5 would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Claims 6-15 depend from claim 5 and would be allowable for the same reasons explained with respect to claim 5, if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Claim 16 would be considered allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims because none of the references of record either alone or in combination fairly disclose or suggest the combination of limitations specified in claim 16, including at least: wherein the optimizer neural network is trained to minimize a combination of loss functions, wherein a first loss function in the combination is the loss for a training machine learning task and a second loss function is an imitation loss that measures a mean squared error between updates generated by the optimizer neural network and corresponding updates generated by a momentum-based machine learning optimizer. The closest prior art of record discloses: US 20220391706 A1, hereinafter referenced as METZ , teaches the general framework for using a neural network optimizer to train a trainee neural network. (paras. 0021-0022, 0094-0097). Ritter, Samuel, et al. "Been there, done that: Meta-learning with episodic recall." International conference on machine learning . PMLR, 2018, hereinafter referenced as RITTER , teaches the concept of modifying a LSTM to utilize an episodic memory that stores cell state with respect to key-value pairs. (pp. 2-3, section 3 and Fig. 1). US 20220138501 A1, hereinafter referenced as EBRAHIMI , teaches modifying a RNN to utilize query, key, value tensors. (paras. 0011, 0109). US 20210276598 A1 (AMIRLOO ABOLFATHI) discloses using combinations of loss functions, including a combination of MSE error and imitation loss. (para. 0213). However, the examiner has found that the distinct feature of the Applicant's claimed invention over the prior art is the explicit claiming of the aforementioned limitations in combination with all the other limitations as specified in claim 16. Therefore, because these features are not found in the prior art, claim 16 would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Claim 17 depends from claim 16 and would be allowable for the same reasons explained with respect to claim 16, if rewritten in independent form including all of the limitations of the base claim and any intervening claims . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. US 20230113643 A1 (Mittal). Discloses a particular A3M module 123 that is a “few-shot meta-learner.” (para. 0021). Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL C LEE whose telephone number is (571)272-4933. The examiner can normally be reached M-F 12:00 pm - 8:00 pm ET. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at 571-272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /MICHAEL C. LEE/Examiner, Art Unit 2128 Application/Control Number: 18/430,586 Page 2 Art Unit: 2128 Application/Control Number: 18/430,586 Page 3 Art Unit: 2128 Application/Control Number: 18/430,586 Page 4 Art Unit: 2128 Application/Control Number: 18/430,586 Page 5 Art Unit: 2128 Application/Control Number: 18/430,586 Page 6 Art Unit: 2128 Application/Control Number: 18/430,586 Page 7 Art Unit: 2128 Application/Control Number: 18/430,586 Page 8 Art Unit: 2128 Application/Control Number: 18/430,586 Page 9 Art Unit: 2128 Application/Control Number: 18/430,586 Page 10 Art Unit: 2128 Application/Control Number: 18/430,586 Page 11 Art Unit: 2128 Application/Control Number: 18/430,586 Page 12 Art Unit: 2128 Application/Control Number: 18/430,586 Page 13 Art Unit: 2128 Application/Control Number: 18/430,586 Page 14 Art Unit: 2128 Application/Control Number: 18/430,586 Page 15 Art Unit: 2128 Application/Control Number: 18/430,586 Page 16 Art Unit: 2128 Application/Control Number: 18/430,586 Page 17 Art Unit: 2128 Application/Control Number: 18/430,586 Page 18 Art Unit: 2128 Application/Control Number: 18/430,586 Page 19 Art Unit: 2128 Application/Control Number: 18/430,586 Page 20 Art Unit: 2128 Application/Control Number: 18/430,586 Page 21 Art Unit: 2128 Application/Control Number: 18/430,586 Page 22 Art Unit: 2128 Application/Control Number: 18/430,586 Page 23 Art Unit: 2128 Application/Control Number: 18/430,586 Page 24 Art Unit: 2128 Application/Control Number: 18/430,586 Page 25 Art Unit: 2128 Application/Control Number: 18/430,586 Page 26 Art Unit: 2128 Application/Control Number: 18/430,586 Page 27 Art Unit: 2128 Application/Control Number: 18/430,586 Page 28 Art Unit: 2128
Prosecution Insights