DETAILED ACTION
This action is in response the communications filed on 08/05/2025 in which claims 1-4, 9, 17 and 23-25 are amended, claims 18-22 have been canceled and therefore claims 1-17 and 23-25 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-17 and 23-25 rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Shen ("ReasoNet: Learning to Stop Reading in Machine Comprehension" 20170813)
In regard to claims 1 and 23, Shen teaches: A computer-implemented neural network based memory system, comprising: a memory configured to receive and store representations of knowledge items, wherein the memory comprises a set of memory slots each to store a representation of a respective knowledge item; (Shen, p. 1048 "Memory: [a memory] The external memory is denoted as M. It is a list of word vectors, M = {mi} i=1..D, where mi is a fixed dimensional vector. [a set of memory slots] For example, in the Graph Reachability, mi is the vector representation of each word [memory to receive store a representation of a respective knowledge item] in the graph description encoded by a bidirectional-RNN."; p. 1049 "In the machine reading comprehension tasks, a training dataset is a collection of triplets of query q, passage p, and answer a. Say <qn, pn, an> is the n-th training instance. The first step is to extract memory M from pn by mapping each symbolic in the passage to a contextual representation given by the concatenation of forward and backward RNN hidden states, i.e., mk = [pn→k, pn←|pn|-k+1] [memory to receive store a representation of a respective knowledge item]...")
PNG
media_image1.png
618
1172
media_image1.png
Greyscale
an iterative memory retrieval system configured to iteratively generate a memory system output by, at each of a succession of time steps, combining a current query derived from an input query with data retrieved from the memory at a previous time step, (Shen, p. 1048 "Internal State: [an iterative memory retrieval system] The internal state is denoted as s which is a vector representation of the question state. Typically, the initial state s1 is the last-word vector representation of query by an RNN... The sequence of internal states are modeled by an RNN: st+1 = RNN (st,xt; θs)... [iteratively generate... at each of a succession of time steps]"; p. 1048 "Attention: The attention vector is generated based on the current internal state st and the external memory M: xt = fatt (st, M; θx). [combining a current query St with data retrieved from the memory M]"; p. 1049 "initial state s1 from qn by assigning s1 = [qn→|qn|, qn←1] [derived from an input query] "; see Figure 1: St -> St+1 -> St+2 [iteratively generate a memory system output by, at each of a succession of time steps])
wherein the iterative memory retrieval system comprises a soft attention subsystem configured to determine from the current query a set of soft attention values, one for each of the set of memory slots, and (Shen, p. 1048 "Attention: [a soft attention subsystem] The attention vector is generated based on the current internal state st and the external memory M: xt = fatt (st, M; θx)."; p. 1050 "M = (Mquery, Mdoc), where Mquery and Mdoc are extracted from query bidirectional-GRU encoder and passage bidirectional-GRU encoder respectively. We choose projected cosine similarity function as the attention module. The attention score a_doc_ti on memory m_doc_i given the state st is computed as...follows: a_doc_ti = softmax ... cos(W_doc_1 m_doc_i, W_doc_2 st)... [determine (from st) the attention score a_ti] W_doc_1 and W_doc_2 are weight vectors associated with m_doc_i and st, respectively, and are joint trained in the ReasonNet... Similarly, the attention vector on query x_query_t = Σ a_query_ti m_query_i... [determine (from st) the attention score a_doc_ti/a_query_ti, i.e., determine (from the current query) a set of soft attention values] [ai for each mi, one for each of the set of memory slots mi] "; st represents the state of the query) to determine a set of weights for the set of memory slots from a combination of the set of soft attention values; (Shen, p. 1048 " Thus, the attention vector on passage is given by x_doc_t = Σ... a_doc_ti m_doc_i. Similarly, the attention vector on query x_query_t = Σ a_query_ti m_query_i... [determine x_doc_t/x_query_t from a combination of attention score a_doc_ti/a_query_ti, determine a set of weights from a combination of the set of soft attention values] The final attention vector is the concatenation of the query attention vector and the passage attention vector xt = (x_query_t, x_doc_t).")
an output system to determine the memory system output from a query result determined by applying the current query to the memory at a final time step; and (Shen, p. 1048 "Answer: [the output system] The action of answer module is triggered when the termination gate variable is true: at ~ p(·|fa(st;a)). [determine the memory system output at from a query result st]"; p. 1048 "Attention: The attention vector is generated based on the current internal state st and the external memory M: xt = fatt (st, M; θx)... [applying the current query to the memory]"; p. 1049 "Output: Termination Step T [at a final step], Answer aT"; p. 1048"The internal state is denoted as s which is a vector representation of the question state."; the hidden states/units of RNN is the vector representation of the query, s1 is the initial state of the query, and s_t (or s_t+2 in Fig. 1) is [the query result], and st+2 in Fig. 1 or termination step T both can be an example of [a final time step])
a controller to control a number of time steps performed by the iterative memory retrieval system until the final time step, (Shen, p. 1048 "Termination Gate: [a controller] The termination gate generates a random variable according to the current internal state; tt ~ p(·|ftg(st;tg)))... If tt is true, the ReasoNet stops, and the answer module executes at time step t; [control a number of time steps performed by the iterative memory retrieval system]otherwise the ReasoNet generates... to update the next internal state st+1"; see Figure 1: termination with t, t+1, t+2 [a number of time steps] and t+2 in Fig. 1 or termination step T both can be an example of [the final time step])
wherein the controller comprises a controller neural network subsystem (Shen, p. 1049 "The learnable parameters θ of the ReasoNet are… the termination gate network θtg...[a controller neural network subsystem] ") configured to receive observations from the iterative memory retrieval system and (p. 1048 "Termination Gate: [the controller] The termination gate generates a random variable according to the current internal state; [receive observations from the iterative memory retrieval system] tt ~ p(·|ftg(st;tg))). tt is a binary random variable. If tt is true, the ReasoNet stops, and the answer module executes at time step t ; otherwise the ReasoNet...")
wherein the observations at each time step comprises a measure of a change in the set of weights between a current time step and the previous time step. (Shen, p. 1048 "Termination Gate: The termination gate generates a random variable according to the current internal state [the observations]; tt ~ p(·|ftg(st;tg)))... If tt is true, the ReasoNet stops, and the answer module executes at time step t ; otherwise the ReasoNet generates an attention vector xt+1, [a measure of a change in the set of weights (e.g. xt -> xt+1) between a current time step and the previous time step]…"; p. 1048 "Internal State: The internal state is denoted as s which is a vector representation of the question state. Typically, the initial state s1 is the last-word vector representation of query by an RNN... The sequence of internal states are modeled by an RNN: st+1 = RNN (st, xt; θs). "; st+2 = RNN (st+1, xt+1; θs) = RNN (RNN (st, xt; θs), xt+1; θs), [the observations at each time step comprises a measure of a change in the set of weights (xt -> xt+1)]; Recurrent Neural Network (RNN) bases its internal state (or hidden state) on previous states)
Claim 23 recites substantially the same limitation as claim 1, therefore the rejection applied to claim 1 also apply to claim 23. In addition, Shen teaches: One or more non-transitory computer readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement: (Shen, p. 4 "Models are trained on GTX TitanX 12GB.")
In regard to claims 2 and 24, Shen teaches: wherein the iterative memory retrieval system further comprises:
a query update subsystem to apply the set of weights to values derived from the representations of the knowledge items in the memory slots to determine the query result, wherein the current query is defined by the input query at an initial time step and depends on the query result from the previous time step thereafter. (Shen, p. 1048 "The sequence of internal states are modeled by an RNN: st+1 = RNN (st,xt; θs)…"; p. 1049 "Algorithm 1… Input: Memory M; Initial state s1;... Output: Termination Step T, Answer aT… 4 Update internal state [a query update subsystem] st+1 = RNN (st,xt;θs); [apply xt to determine st+1, apply the set of weights to values to determine the query result]"; xt is derived from mi (m_doc_i and m_query_i) [derived from the representations of the knowledge items in the memory slots])
In regard to claims 3 and 25, Shen teaches: wherein the controller neural network subsystem has a halting control output, (Shen, p. 1049 "The learnable parameters θ of the ReasoNet are… the termination gate network θtg...[the controller neural network subsystem] "; p. 1048 "Termination Gate: [the controller] The termination gate generates a random variable according to the current internal state; tt ~ p(·|ftg(st;tg))). tt is a binary random variable. If tt is true, the ReasoNet stops, [tt, a halting control output] and the answer module executes at time step t ; otherwise the ReasoNet..."; if tt = 1 then halt, else if tt = 0, no halt)
wherein the controller is configured to halt the iterative memory retrieval system, using the halting control output, to control the number of time steps performed until the final time step. (Shen, p. 1048 "Termination Gate: [the controller] The termination gate generates a random variable according to the current internal state; tt ~ p(·|ftg(st;tg)))... If tt is true, the ReasoNet stops, [using the halting control output tt] and the answer module executes at time step t; [to control the number of time steps performed] otherwise the ReasoNet generates... to update the next internal state st+1"; see Figure 1: termination with t, t+1, t+2 [the number of time steps] and t+2 in Fig. 1 or termination step T both can be an example of [the final time step])
In regard to claim 4, Shen teaches: wherein the observations at each time step comprise one or more of: the current query at the current time step; the current query at the previous time step; and a count of a number of time steps taken. (Shen, p. 1048 "Termination Gate: The termination gate generates a random variable according to the current internal state; [the observations st at each time step t]; p. 1049 "Algorithm 1… Input: Memory M; Initial state s1;... Output: Termination Step T, Answer aT… 3 Generate attention vector xt... 4 Update internal state s_t+1 = RNN (st,xt;θs); 5 Set t = t+1 [a count of a number of time steps taken]..."; st represents the state of the query, therefore st is [the current query at the current time step] which is generated based on st-1 [the current query at the previous time step])
In regard to claim 5, Shen teaches: wherein the controller comprises a reinforcement learning controller neural network subsystem to define a probability of halting the iterative memory retrieval system for the halting control output. (Shen, p. 1047 "With the use of reinforcement learning, ReasoNets can dynamically determine whether to continue the comprehension process after digesting intermediate results, or to terminate reading when it concludes that existing information is adequate to produce an answer..."; p. 1049 "The learnable parameters θ of the ReasoNet are… the termination gate network θtg... [a reinforcement learning controller neural network subsystem]"; p. 1048 "Termination Gate: [the controller] The termination gate generates a random variable according to the current internal state; tt ~ p(·|ftg(st;tg))). tt is a binary random variable. If tt is true, the ReasoNet stops, [define a probability of halting/stopping, tt = 1 or 0] and... the answer module executes at time step t ; otherwise the ReasoNet..."; p. 1051 "The corresponding termination probability and the sum of attention scores.."; p. 1054 "... The termination step is chosen with the maximum termination probability p(k)..., where ti is the termination probability at step i."; probability ∈ [0, 1])
In regard to claim 6, Shen teaches: further comprising a training engine to train the reinforcement learning controller neural network subsystem using a reinforcement learning technique with a loss function dependent upon a count of a number of time steps taken until the final time step. (Shen, p. 1049 "The ReasoNet performs an answer action aT at the T-th step, which implies that the termination gate variables t1:T = (t1 = 0, t2 = 0,... tT-1 = 0, tT = 1) The ReasoNet learns a stochastic policy π... with parameters θ to get a distribution of termination actions, to continue reading or to stop... The expected reward for an instance is defined as: J(θ) = Eπ (t1:T, aT; θ) [Σ rt]... [loss function dependent on t1:T, a count of a number of time steps taken until the final time step] J can be maximized by directly applying gradient based optimization methods.")
In regard to claim 7, Shen teaches: wherein the reinforcement learning controller neural network subsystem is configured to estimate a time-discounted return resulting from halting the iterative memory retrieval system at a time step, and wherein the loss function is dependent upon the time-discounted return. (Shen, p. 1049 "The expected reward for an instance is defined as: J(θ) = Eπ (t1:T, aT; θ) [Σ rt]... [loss function dependent on a time-discounted return rt at a time step t] J can be maximized by directly applying gradient based optimization methods."; In reinforcement learning, the discounted return represents the total future reward an agent expects to receive, but with future rewards discounted by a factor that decreases with time)
In regard to claim 8, Shen teaches: wherein the reinforcement learning technique is a policy gradient-based reinforcement learning technique and wherein the loss function is further dependent upon a value estimate generated by the reinforcement learning controller neural network subsystem for the time step. (Shen, p. 1049 "The expected reward for an instance is defined as: J(θ) = Eπ (t1:T, aT; θ) [Σ rt]... [loss function dependent on a value estimate aT or rt generated by RL] J can be maximized by directly applying gradient based optimization methods. The gradient of J is given by: ∇θJ(θ)... Motivated by the REINFORCE algorithm [31], we compute ∇θJ(θ)... [a policy gradient-based reinforcement learning technique]... Each episode generates actions and a reward from the last step: <(t1:T, aT), rT>...")
In regard to claim 9, Shen teaches: wherein the iterative memory retrieval system is configured to determine a key-value pair representing each of the knowledge items, (Shen, p. 1051 "We use bidirectional-GRU encoders to extract the query representation Mquery and the passage representation Mdoc, given a query and a passage. We compute the similarity matrix between each word in the query and each word in the passage.[a key-value pair representing each of the knowledge items]")
wherein the soft attention subsystem is configured to determine a similarity measure between the current query and the key for each memory slot to determine the set of soft attention values, and (Shen, p. 1052 "The similarity matrix is denoted as S ∈ R TxJ, where tand J are the number of words in the passage and query, respectively, and S_tj = wsT[M_doc:t; M_query:j; M_doc:t ○ M_query [a similarity measure between the current query and the key for each memory slot to determine the set of soft attention values]] ∈ R... ○ denotes the elementwise multiplication")
wherein the query update subsystem is configured to apply the set of weights to the values representing the knowledge items in each of the memory slots to determine the query result. (Shen, p. 1048 "The sequence of internal states are modeled by an RNN: st+1 = RNN (st,xt; θs)…"; p. 1049 "Algorithm 1… Input: Memory M; Initial state s1;... Output: Termination Step T, Answer aT… 4 Update internal state [the query update subsystem] st+1 = RNN (st,xt;θs); [apply xt to determine st+1, apply the set of weights to values to determine the query result]"; xt is derived from mi (m_doc_i and m_query_i), i.e. xt are [the values representing the knowledge items in each of the memory slots])
In regard to claim 10, Shen teaches: wherein the iterative memory retrieval system is configured to apply respective key and value projection matrices to the representation of the knowledge item in a memory slot to determine the key-value pair representing the knowledge item in the memory slot. (Shen, p. 1051 "We use bidirectional-GRU encoders to extract the query representation Mquery and the passage representation Mdoc, [the representation of the knowledge item] given a query and a passage. We compute the similarity matrix between each word in the query and each word in the passage. The similarity matrix is denoted as S ∈ R TxJ, where tand J are the number of words in the passage and query, respectively, and S_tj = wsT[M_doc:t; M_query:j; M_doc:t ○ M_query] [respective key and value projection matrices] ∈ R... ○ denotes the elementwise multiplication...We then compute the context-to-query attention and query-to-context attention from the similarity matrix S [apply S to M_doc, i.e. apply respective key and value projection matrices to the representation of the knowledge item] by following recent co-attention work [20] to obtain the query-aware passage representation G. We feed G to a 128-dimensional bidirectional GRU to obtain the memory M = bidirectional-GRU(G), where M ∈ 256xT. [S_tj or M = bi-GRU(G), to determine the key-value pair representing the knowledge item in the memory slot]")
In regard to claim 11, Shen teaches: wherein the iterative memory retrieval system is configured to apply a query projection matrix to the input query to provide an encoded query, (Shen, p. 1049 "The learnable parameters θ of the ReasoNet are the embedding matrices θW..."; p. 1050 "Bi-GRU Encoder: We apply bidirectional GRU for encoding query and passage into vector representations. [encode the input query to provide an encoded query]")
wherein the encoded query comprises the current query defined by the input query at the initial time step. (Shen, p. 1048 "Internal State: The internal state is denoted as s which is a vector representation of the question state. Typically, the initial state s1 is the last-word vector representation of query [the current query defined by the input query at the initial time step] by an RNN."; p. 1049 "initial state s1 from qn by assigning s1 = [qn→|qn|, qn←1]"; initially the current query is the input query)
In regard to claim 12, Shen teaches: wherein the soft attention subsystem comprises a soft attention neural network to process the set of soft attention values to determine the set of weights for the set of memory slots. (Shen, p. 1048 "Attention: [a soft attention subsystem] The attention vector is generated based on the current internal state st and the external memory M: xt = fatt (st, M; θx)."; p. 1049 "The learnable parameters θ of the ReasoNet are the embedding matrices θw, attention network θx, [a soft attention neural network] the state RNN network θs, the answer action network θa, and the termination gate network θtg..."; p. 1050 " The attention score a_doc_ti on memory m_doc_i given the state st is computed as follows... Thus, the attention vector on passage is given by x_doc_t = Σ... a_doc_ti m_doc_i. Similarly, the attention vector on query x_query_t = Σ a_query_ti m_query_i... [process attention score a_ti (a_doc_ti/a_query_ti) to determine xt (x_doc_t/x_query_t), i.e. process the set of soft attention values to determine the set of weights] The final attention vector is the concatenation of the query attention vector and the passage attention vector xt = (x_query_t, x_doc_t).")
In regard to claim 13, Shen teaches: wherein the output system comprises an output neural network to process the query result to generate the memory system output. (Shen, p. 1048 "Internal State: The internal state is denoted as s which is a vector representation of the question state."; p. 1048 "Answer: [the output system] The action of answer module is triggered when the termination gate variable is true: at ~ p(·|fa(st;a)). [process internal state st to generate answer at]"; p. 1049 "Algorithm 1… Input: Memory M; Initial state s1;... Output: Termination Step T, Answer aT… 4 Update internal state s_t+1 = RNN (st,xt;θs); 6 Generate answer at ~ p(·|fa(st;θa)) [process internal state st to generate answer at, i.e. process the query result to generate the memory system output] 7 Return T = t and aT= at;" p. 1049 "The learnable parameters θ of the ReasoNet are... the answer action network θa [an output neural network]"; the hidden states/units of RNN is the vector representation of the query, s1 is the initial state of the query, and s_t (or s_t+2 in Fig. 1) is [the query result])
In regard to claim 14, Shen teaches: further comprising an encoder neural network subsystem to encode knowledge item data for the knowledge items into the representations of the knowledge items. (Shen, p. 1050 "Bi-GRU Encoder: We apply bidirectional GRU for encoding query and passage into vector representations."; p. 1051 "Bi-GRU Encoder: We apply bidirectional GRU for encoding query and passage [encode knowledge item data for the knowledge items] into vector representations. [the representations of the knowledge items]")
In regard to claim 15, Shen teaches: wherein the encoder neural network subsystem comprises a convolutional neural network. (Shen, p. 1051 "we adopt one layer 100-dimensional convolutional neural network on character-level with a width size of 5 and each character encoded as an 8-dimensional vector following the work [20].")
In regard to claim 16, Shen teaches: wherein the encoder neural network subsystem comprises a recurrent neural network. (Shen, p. 1051 "Bi-GRU Encoder: We apply bidirectional GRU [a recurrent neural network] for encoding query and passage into vector representations."; GRU is an RNN)
In regard to claim 17, Shen teaches: wherein the knowledge items comprise one or more of image items, digitized sound items, text data items, or graph data items. (Shen, p. 1047 "ReasoNets achieve superior performance in machine comprehension datasets, including unstructured CNN and Daily Mail datasets, the Stanford SQuAD dataset, and a structured Graph Reachability dataset."; p. 1051 "... a test example 69e1f777e41bf67d5a22b7c69ae76f0ae873cf43.story [text data items] from the CNN dataset"; p. 1051 "SQuAD is a machine comprehension dataset on 536 Wikipedia articles [text data items], with more than 100,000 questions."; p. 6 "Table 4: Small and large random graph [graph data items] in the Graph Reachability dataset.")
PNG
media_image2.png
402
304
media_image2.png
Greyscale
PNG
media_image3.png
252
500
media_image3.png
Greyscale
Response to Arguments
Applicant's amendments with respect to the claim and drawing objections have been fully considered and are sufficient to overcome the objections. The objections have been withdrawn.
Applicant's arguments with respect to the rejection of the claims under 35 U.S.C. 102 have been fully considered but they are not persuasive:
Applicant argues: (see p. 9) … However… The attention vector, as described in Shen, therefore is a query attention vector and a passage attention vector, each being a set of weights determined using attention scores computed for the query and for the passage respectively. Accordingly, it follows that the attention vector in Shen is not "data retrieved from the memory at a previous time step."
Examiner answers: The examiner does not correlate “the attention vector” to “data retrieved from the memory.” Instead, “the external memory M” in xt = fatt (st, M; θx) corresponds to “data retrieved from the memory.”
Applicant argues: (see p. 11) Furthermore, the Action cited Shen as teaching the feature "wherein the observations at each time step comprise a measure of a change in the set of weights between a current time step and the previous time step" as previously recited by claim 4 that is now being incorporated into amended claim 1… However, the random variable tt that the termination gate described in Shen generates is not "the observations at each time step" as recited by amended claim 1. Rather, "tt is a binary random variable." Moreover, the attention vectors are not "observations at each time step" that "comprise a measure of a change in the set of weights between a current time step and the previous time step" as the attention vector is not received by a controller neural network of a controller that controls "…
Examiner answers: The examiner does not correlate “the variable tt” to “the observations at each time step.” Instead, “the internal state St” corresponds to “the observations at each time step.” Further, The examiner does not correlate “the attention vector” to “the observations at each time step.” Instead, “the internal state St” corresponds to “the observations at each time step.”
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SU-TING CHUANG whose telephone number is (408)918-7519. The examiner can normally be reached Monday - Thursday 8-5 PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew J. Jung can be reached on (571) 270-3779. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.C./Examiner, Art Unit 2146
/ANDREW J JUNG/Supervisory Patent Examiner, Art Unit 2146