DETAILED ACTION
This action is in response to the communications filed on 06/10/2026 in which claims 1-20 are amended and claims 1-20 are pending.
--
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claims 1, 8 and 15 recite the limitation “train a model using the first set of the plurality of input features.” There is insufficient antecedent basis for “the plurality of input features” in the claim. For examination purposes examiner has interpreted “the plurality of input features” to be “the first plurality of input features.”
Claims 1, 8 and 15 recite the limitation “determine a reward based on… the composite interpretability of the first set of the first plurality of features.” There is insufficient antecedent basis for “the first plurality of features” in the claim. For examination purposes examiner has interpreted “the first plurality of features” to be “the first plurality of input features.”
Claims 2-7, 9-14 and 16-20 are also rejected due to their dependency on a rejected claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
-
Claims 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more
Step 1: Claims 1-7 recite a system. Claims 8-14 recite a method. Claims 15-20 recite a non-transitory medium Therefore, claims 1-7 are directed to a machine, claims 8-14 are directed to a process, and claims 15-20 are directed to a manufacture.
With respect to claims 1, 8 and 15:
2A Prong 1: The claim recites a judicial exception.
determine an interpretability of each of the first plurality of input features based on a domain ontology and on symbolic rules associated with entities of the domain ontology (mental process – evaluation or judgement, determine an interpretability of each of the features based on a domain ontology and on symbolic rules)
determine a first set of the first plurality of input features which were determined as interpretable (mental process – evaluation or judgement, determine a first set of the features which were determined as interpretable)
determine a composite interpretability of the first set of the first plurality of input features based on the interpretability of each of the first set of the first plurality of input features (mental process – evaluation or judgement, determine a composite interpretability of the first set of the features based on the interpretability)
determine a performance of the trained model (mental process – evaluation or judgement, determine/evaluate a performance of a model)
determine a reward based on the performance of the trained model and the composite interpretability of the first set of the first plurality of features (mental process – evaluation or judgement, determine a reward based on the performance of the trained model and the composite interpretability)
determine a reward based on the performance of the trained model and the composite interpretability of the first set of the first plurality of features (mental process – evaluation or judgement, determine a reward based on the performance of the model and the composite interpretability)
2A Prong 2: The judicial exception is not integrated into a practical application.
(claim 1) A system comprising: a memory storing processor-executable program code; and at least one processing unit to execute the processor-executable program code to cause the system to: (claim 15) A non-transitory medium storing executable program code executable by at least one processing unit of a computing system to cause the computing system to: (mere instructions to apply an exception, (2) Whether the claim invokes computers - MPEP 2106.05(f); generic computer components)
generate a first plurality of input features using a learning network (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; using a learning network to generate features)
train a model using the first set of the plurality of input features (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; train a model using features)
receive the reward at the learning network (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting)
generate a second plurality of input features using the learning network based on the reward (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; using the learning network to generate features based on the reward)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
(claim 1) A system comprising: a memory storing processor-executable program code; and at least one processing unit to execute the processor-executable program code to cause the system to: (claim 15) A non-transitory medium storing executable program code executable by at least one processing unit of a computing system to cause the computing system to: (mere instructions to apply an exception, (2) Whether the claim invokes computers - MPEP 2106.05(f); generic computer components)
generate a first plurality of input features using a learning network (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; using a learning network to generate features)
train a model using the first set of the plurality of input features (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; train a model using features)
receive the reward at the learning network (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting, and WURC: receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 - MPEP 2106.05(d)(II)(i))
generate a second plurality of input features using the learning network based on the reward (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; using the learning network to generate features based on the reward)
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claims 2, 9 and 16:
2A Prong 1: The claim recites a judicial exception.
wherein determination of an interpretability of each of the first plurality of input features comprises (mental process – evaluation or judgement, determine an interpretability of each of the first plurality of features)
annotation of each of the first plurality of input features based on the entities of the domain ontology (mental process – evaluation or judgement, annotate each of the first plurality of features based on the entities of the domain ontology)
With respect to claims 3, 10 and 17:
2A Prong 1: The claim recites a judicial exception.
wherein determination of an interpretability of each of the first plurality of input features comprises (mental process – evaluation or judgement, determine an interpretability of each of the first plurality of features)
2A Prong 2: The judicial exception is not integrated into a practical application.
executing symbolic reasoning by applying the symbolic rules to the annotated first plurality of input features (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; applying the rules to the features)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
executing symbolic reasoning by applying the symbolic rules to the annotated first plurality of input features (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; applying the rules to the features)
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claims 4, 11 and 18:
2A Prong 2: The judicial exception is not integrated into a practical application.
wherein the symbolic reasoning comprises subsumption and instance checking (a particular technological environment or field of use – MPEP 2106.05(h))
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
wherein the symbolic reasoning comprises subsumption and instance checking (a particular technological environment or field of use – MPEP 2106.05(h))
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claims 5, 12 and 19:
2A Prong 2: The judicial exception is not integrated into a practical application.
wherein the model is trained using the first set of the first plurality of input features and a second set of the first plurality of input features which were not determined as non-interpretable (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; training a model using the first set of features and a second set of features)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
wherein the model is trained using the first set of the first plurality of input features and a second set of the first plurality of input features which were not determined as non-interpretable (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; training a model using the first set of features and a second set of features)
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claims 6 and 13:
2A Prong 1: The claim recites a judicial exception.
wherein determination of an interpretability of each of the first plurality of input features comprises (mental process – evaluation or judgement, determine an interpretability of each of the first plurality of features)
2A Prong 2: The judicial exception is not integrated into a practical application.
executing symbolic reasoning by applying the symbolic rules to the first plurality of input features (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; applying the rules to the features)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
executing symbolic reasoning by applying the symbolic rules to the first plurality of input features (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; applying the rules to the features)
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claims 7 and 14:
2A Prong 2: The judicial exception is not integrated into a practical application.
wherein the symbolic reasoning comprises subsumption and instance checking (a particular technological environment or field of use – MPEP 2106.05(h))
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
wherein the symbolic reasoning comprises subsumption and instance checking (a particular technological environment or field of use – MPEP 2106.05(h))
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claim 20:
2A Prong 1: The claim recites a judicial exception.
wherein determination of an interpretability of each of the first plurality of input features comprises (mental process – evaluation or judgement, determine an interpretability of each of the first plurality of features)
2A Prong 2: The judicial exception is not integrated into a practical application.
execution of subsumption and instance checking on the first plurality of input features using the symbolic rules (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; execution of subsumption and instance checking on the features)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
execution of subsumption and instance checking on the first plurality of input features using the symbolic rules (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; execution of subsumption and instance checking on the features)
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 6, 8-10, 13 and 15-17 rejected under 35 U.S.C. 103 as being unpatentable over Lyu ("SDRL: Interpretable and Data-Efficient Deep Reinforcement Learning Leveraging Symbolic Planning" 2019) in view of Ma ("Learning Symbolic Rules for Interpretable Deep Reinforcement Learning" 20210316) in view of Dash (US 20240135205 A1, filed on 2022-10-11)
In regard to claims 1, 8 and 15, Lyu teaches: generate a first plurality of input features using a learning network; (Lyu, p. 2970, Abstract "This framework features a planner – controller – meta-controller architecture, which takes charge of subtask scheduling, data-driven subtask learning, and subtask evaluation, respectively."; p. 2971, Introduction "we propose a Symbolic Deep Reinforcement Learning (SDRL) framework that features a planner–controller–meta-controller architecture, i.e., 1. A planner uses prior symbolic knowledge to perform long-term planning by a sequence of symbolic actions (subtasks) [the first set of the plurality of input features] that achieve its intrinsic goal;"; p. 2795, Setup "Our experiment setup follows the DQN controller architecture (Kulkarni et al. 2016) with double-Q learning... The architecture of the deep neural networks is shown in Table 1."; see Fig. 1, meta-controller and symbolic planner act as a learning network that generates subtasks [generate a first plurality of input features, symbolic plans (a sequence of subgoals, subtasks, or symbolic states and actions)])
PNG
media_image1.png
254
600
media_image1.png
Greyscale
… train a model using the first set of the plurality of input features; (Lyu, p. 2970, Abstract "This framework features a planner – controller [a model] – meta-controller architecture, which takes charge of subtask scheduling, data-driven subtask learning [train a model], and subtask evaluation, respectively."; p. 2971, Introduction "we propose a Symbolic Deep Reinforcement Learning (SDRL) framework that features a planner–controller–meta-controller architecture, i.e., 1. A planner uses prior symbolic knowledge to perform long-term planning by a sequence of symbolic actions (subtasks) [the first set of the plurality of input features] that achieve its intrinsic goal; 2. A controller uses DRL algorithms to learn [train a model] the sub-policy for each subtask [using the first set of the plurality of input features] based on intrinsic rewards;")
determine a performance of the trained model; (Lyu, p. 2971, Introduction "A meta-controller learns on extrinsic rewards by measuring the training performance of controllers [determine a performance of the trained model] and propose new intrinsic goals to the planner."; p. 2972, Integrating Symbolic Planning with Reinforcement Learning "In particular, the meta-controller is introduced to bridge the gap of planning over symbolic states and DRL over pixel images, by learning at the task level using extrinsic reward derived from training performance of DRL. [determine a performance of the trained model]")
determine a reward based on the performance of the trained model and the composite interpretability of the first set of the first plurality of features; (Lyu, p. 2971, Introduction "A meta-controller learns on extrinsic rewards [a reward] by measuring the training performance of controllers [the performance of the trained model] and propose new intrinsic goals to the planner."; p.2972 "We use an out-of-box symbolic planner to generate and improve plans, with symbolic transitions automatically mapped to subtasks, leading to a more interpretable and expressive representation."; p. 2972, Integrating Symbolic Planning with Reinforcement Learning "In particular, the meta-controller is introduced to bridge the gap of planning over symbolic states and DRL over pixel images, by learning at the task level using extrinsic reward derived from training performance of DRL. [a reward based on the performance of the trained model]" p. 2974, Rewards "We further define extrinsic reward [a reward] for selecting subtask g at symbolic state s as r_e(s, g) = f(ε) where f is a function about ε, a criterion that measures the competence of the learned sub-policy [the performance of the trained model] for each subtask. [the composite interpretability of the first set of the first plurality of features]"; p. 2973, SDRL Framework "A symbolic planner generates high-level plans, i.e., a sequence of subtasks, to meet its intrinsic goal."; see prior art Ma teaches neural-symbolic reasoning in the meta-controller that generates subtasks with composite interpretability, therefore the subtasks provided to the controller are based on 'the composite interpretability of the first set of the first plurality of features,' and therefore extrinsic rewards are determined based on the composite interpretability)
receive the reward at the learning network; and (Lyu, p. 2973, SDRL Framework "When the sub-policy is learned and reliably achieves the subtask, the extrinsic reward is equivalent to the environmental reward. Using extrinsic rewards, meta-controller performs R-learning that reflects the long-term average reward and gains the reward of selecting each subtask."; p. 2974, Planning and Learning "Meta-controller performs R-learning (Line 18) based on extrinsic rewards for the symbolic transitions"; see Fig. 1, extrinsic rewards is provided to the meta-controller [the learning network])
generate a second plurality of input features using the learning network based on the reward. (Lyu, p. 2974, Planning and Learning "The loop continues until the symbolic plan Π* cannot be further improved."; when the loop continues, meta-controller [the learning network] generates an improved symbolic plans (a sequence of subgoals, subtasks, or symbolic states and actions) [a second plurality of input features] based on the extrinsic reward [the reward])
Ma teaches NSRL performing neuro-symbolic reasoning as a meta-controller in the HRL to receive symbolic states and generate a subtask. (Ma, p. 6, 5.1.2 "We implement our architecture NSRL and HDQN with an option-based hierarchical reinforcement learning framework similar to SDRL. This framework is split into two levels, meta controller (high level) and action controller (low level). The meta controller [a learning network] assigns a task [generate a subtask, symbolic plans (a sequence of subgoals, subtasks, or symbolic states and actions)] to be achieved by the action controller. The only difference between these agents is the way to induce a policy in the high level... while NSRL performs neuro-symbolic reasoning. In terms of the low level, all the agents reuse the controller architecture in Kulkarni et al. [2016]..."; also see Ma2, Fig. 1) Because both Lyu and Ma teach Hierarchical Reinforcement Learning, the prior arts are combined here to address the claimed limitations.
Lyu does not teach, but Ma teaches: determine an interpretability of each of the first plurality of input features (Ma, p. 1, 1 Introduction "we investigate an approach that represents states and actions using first-order logic (FOL) and makes sequential decisions via neural-logic reasoning... this framework features a reasoning module based on neural attention networks, which performs relational reasoning on symbolic states and induces the RL policy."; p. 2, 3 Preliminary "Interpretable rules described by First-Order Logic are first introduced, then the basics of Reinforcement Learning (RL) are briefly recalled... A rule also called clause can be written as follows: α ← α1 ^ α2,... ^ αn"; p. 3, 4.1 System Framework "the symbolic states from the environment are firstly transformed into a matrix P... matrix P and the attention weights are sent to the reasoning module to perform reasoning on existing symbolic knowledge... we denote the predicate matrix at each step as P(1), P(2), P(3), P(4), which are the results of the multiplications of Sφ and the symbolic matrix P. [determine an interpretability] Then, we sequentially multiply these matrices to generate logical rules of different lengths.; Symbolic states are input features and are transformed into a matrix P. The first calculation results based on the matrix P and predicate attention weights in the reasoning module are the interpretability of input features.) based on a domain ontology and on symbolic rules associated with entities of the domain ontology; (Ma, p. 3, 4.2 Reasoning Module "Consider a knowledge graph, [ontology] where objects are represented as nodes and relations are edges. [entities of the domain ontology] Multi-hop reasoning on such a graph mainly focuses on searching chain-like logical rules [symbolic rules] of the following form: query (x, x') ← R1 (x, z1) ^ R2 (z1, z2)... ^ Rn (z_n-1, x'). (1)"; p. 5, 5 Experiments "we evaluate our approach on two domains, i.e., Montezuma’s Revenge and BlocksWorld Manipulation"; p. 8 "Table 4... Domain... Blocks World... Montezuma’s Revenge"; in light of spec. [0034] Domain ontology 160 may be considered a knowledge base defining a hierarchy of n logical entities and [0046])
determine a first set of the first plurality of input features which were determined as interpretable; (Ma, p. 3, 4.1 System Framework "the symbolic states from the environment are firstly transformed into a matrix P... matrix P and the attention weights are sent to the reasoning module to perform reasoning on existing symbolic knowledge... we denote the predicate matrix at each step as P(1), P(2), P(3), P(4), which are the results of the multiplications of Sφ and the symbolic matrix P. Then, we sequentially multiply these matrices to generate logical rules of different lengths. [determine a first set of the first plurality of input features which were determined as interpretable]")
determine a composite interpretability of the first set of the first plurality of input features based on the interpretability of each of the first set of the first plurality of input features; (Ma, p. 3, 4.1 System Framework "Next, we apply path attention weights Sy on these rules to generate the reasoning results [determine a composite interpretability]"; p. 8, 5.3 Interpretable Policy "The results of the product are multiplied by the attention weights of the path of the corresponding length. We interpret these results as the confidence of the corresponding logical rules."; applying path attention weight to a first result in the reasoning module to generate a composite result)
PNG
media_image2.png
538
960
media_image2.png
Greyscale
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified Lyu to incorporate the teachings of Ma by including symbolic logic into DRL, and including NSRL performing neuro-symbolic reasoning as a meta-controller in the HRL to generate a subtask. Doing so would achieve competitive performance. (Ma, p. 1, Abstract "we propose a Neural Symbolic Reinforcement Learning framework by introducing symbolic logic into DRL... interpretability is achieved by extracting the logical rules learned by the reasoning module in a symbolic rule space. The experimental results show that our framework has better interpretability, along with competing performance in comparison to state-of-the-art approaches.")
Lyu and Ma do not teach, but Dash teaches: A system comprising: a memory storing processor-executable program code; and at least one processing unit to execute the processor-executable program code to cause the system to: (Dash, [0116] Processor set 510 includes one, or more, computer processors of any type now known or to be developed in the future... Cache 521 is memory that is located in the processor chip package(s) and is typically used for data or code that should be available for rapid access by the threads or cores running on processor set 510.")
Claims 8 and 15 recite substantially the same limitation as claim 1, therefore the rejection applied to claim 1 also apply to claims 8 and 15. In addition, Dash teaches: (claim 15) A non-transitory medium storing executable program code executable by at least one processing unit of a computing system to cause the computing system to: (Dash, [0116] Processor set 510 includes one, or more, computer processors of any type now known or to be developed in the future... Cache 521 is memory that is located in the processor chip package(s) and is typically used for data or code that should be available for rapid access by the threads or cores running on processor set 510.")
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified Lyu and Ma to incorporate the teachings of Dash by including the hardware implementation and the labeling function with the logic rules. Doing so would allow to infer missing facts or quickly identify new rules. (Dash, [0005] "One approach for KGC is to learn first-order logic rules that use known facts to imply other known facts, and then use these to infer missing facts."; [0051] "The rule generation is made efficient and feasible... to quickly identify, through a linear programming problem solving process of iteratively adding new rule clauses and corresponding rules.
In regard to claims 2, 9 and 16, Lyu does not teach, but Ma teaches: wherein determination of an interpretability of each of the first plurality of input features comprises: (Ma, p. 1, 1 Introduction "we investigate an approach that represents states and actions using first-order logic (FOL) and makes sequential decisions via neural-logic reasoning... this framework features a reasoning module based on neural attention networks, which performs relational reasoning on symbolic states and induces the RL policy."; p. 2, 3 Preliminary "Interpretable rules described by First-Order Logic are first introduced, then the basics of Reinforcement Learning (RL) are briefly recalled... A rule also called clause can be written as follows: α ← α1 ^ α2,... ^ αn"; p. 3, 4.1 System Framework "the symbolic states from the environment are firstly transformed into a matrix P... matrix P and the attention weights are sent to the reasoning module to perform reasoning on existing symbolic knowledge... we denote the predicate matrix at each step as P(1), P(2), P(3), P(4), which are the results of the multiplications of Sφ and the symbolic matrix P. [determine an interpretability] Then, we sequentially multiply these matrices to generate logical rules of different lengths.; Symbolic states are input features and are transformed into a matrix P. The first calculation results based on the matrix P and predicate attention weights in the reasoning module are the interpretability of input features.)
Lyu and Ma do not teach, but Dash teaches: annotation of each of the first plurality of input features based on the entities of the domain ontology. (Dash, [0004] "A 'fact' in the knowledge graph [the domain ontology] may be represented as a tuple data structure, such as a triplet of the form (a, r, b) where a and b are nodes, and r is a binary relation labeling a directed edge from a to b indicating that r(a, b) is true. [annotation of each of the first plurality of features (a and b)] As an example, consider a KG where the nodes correspond to distinct cities, states, and countries and the relations are one of capital_of, shares_border_with, or part_of. A fact (a, part_of, b) in such a graph represents a directed edge from a to b labeled by part_of, implying that a is part of b. [annotation]"; [0087] "a set of n binary relations R defined over the domain V"; [0052])
The rationale for combining the teachings of Lyu, Ma and Dash is the same as set forth in the rejection of claim 1.
In regard to claims 3, 10 and 17, Lyu does not teach, but Ma teaches: wherein determination of an interpretability of each of the first plurality of input features comprises: executing symbolic reasoning by applying the symbolic rules to the annotated first plurality of input features. (Ma, p. 3, 4.2 Reasoning Module "Multi-hop reasoning on such a graph mainly focuses on searching chain-like logical rules [symbolic reasoning, symbolic rules] of the following form: query (x, x') ← R1 (x, z1) ^ R2 (z1, z2)... ^ Rn (z_n-1, x'). (1) [applying the symbolic rules]... entry (i, j) is 1 if Pk(xi, xj) holds, i.e., entity xi and x j are connected by edge Pk [annotated, e.g. ^ (conjunction)] in the knowledge graph")
In regard to claims 6 and 13, Lyu does not teach, but Ma teaches: wherein determination of an interpretability of each of the first plurality of input features comprises: executing symbolic reasoning by applying the symbolic rules to the first plurality of input features. (Ma, p. 3, 4.2 Reasoning Module "Multi-hop reasoning on such a graph mainly focuses on searching chain-like logical rules [symbolic reasoning, symbolic rules] of the following form: query (x, x') ← R1 (x, z1) ^ R2 (z1, z2)... ^ Rn (z_n-1, x'). (1) [applying the symbolic rules]... entry (i, j) is 1 if Pk(xi, xj) holds, i.e., entity xi and x j are connected by edge Pk [annotated, e.g. ^ (conjunction)] in the knowledge graph")
Claims 4, 7, 11, 14, 18 and 20 rejected under 35 U.S.C. 103 as being unpatentable over Lyu, Ma and Dash as applied to claims 1, 8 and 15, and in further view of Russell ("Inference in First-Order Logic" 20021114)
In regard to claims 4, 11 and 18, Lyu, Ma and Dash do not teach, but Russell teaches: wherein the symbolic reasoning comprises subsumption and instance checking. (Russell, p. 273, Inference rules for quantifiers "Let us begin with universal quantifiers. Suppose our knowledge base contains the standard folkloric axiom stating that all greedy kings are evil: for all x King(x) ^ Greedy(x) => Evil(x)… The rule of Universal Instantiation [instance checking] (UI for short) says that we can infer any sentence obtained by substituting a ground term (a term without variables) for the variable. 1 To write out the inference rule formally, we use the notion of substitutions [subsumption] introduced in Section 8.3... The corresponding Existential Instantiation rule [instance checking] for the existential quantifier is slightly more complicated...."; p. 4, A first-order inference rule, "The inference that John is evil works like this: find some x such that x is a king and x is greedy, and then infer that this x is evil. More generally, if there is some substitution θ that makes the premise of the implication identical to sentences already in the knowledge base, then we can assert the conclusion of the implication, after applying θ. In this case, the substitution {x/John} achieves that aim... for any sentence p (whose variables are assumed to be universally quantified) and for any substitution θ, p |= SUBST(θ; p). [subsumption]")
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified Lyu, Ma and Dash to incorporate the teachings of Russell by including inference in first-order logic rules. Doing so would result in use the most efficient method that can accommodate the facts and axioms that need to be expressed. (Russell, p. 272 "Section 9.1 introduces inference rules for quantifiers and shows how to reduce first-order inference to propositional inference, albeit at great expense... In general, one tries to use the most efficient method that can accommodate the facts and axioms that need to be expressed.")
In regard to claims 7 and 14, Lyu, Ma and Dash do not teach, but Russell teaches: wherein the symbolic reasoning comprises subsumption and instance checking. (Russell, p. 273, Inference rules for quantifiers "Let us begin with universal quantifiers. Suppose our knowledge base contains the standard folkloric axiom stating that all greedy kings are evil: for all x King(x) ^ Greedy(x) => Evil(x)… The rule of Universal Instantiation [instance checking] (UI for short) says that we can infer any sentence obtained by substituting a ground term (a term without variables) for the variable. 1 To write out the inference rule formally, we use the notion of substitutions [subsumption] introduced in Section 8.3... The corresponding Existential Instantiation rule [instance checking] for the existential quantifier is slightly more complicated...."; p. 4, A first-order inference rule, "The inference that John is evil works like this: find some x such that x is a king and x is greedy, and then infer that this x is evil. More generally, if there is some substitution θ that makes the premise of the implication identical to sentences already in the knowledge base, then we can assert the conclusion of the implication, after applying θ. In this case, the substitution {x/John} achieves that aim... for any sentence p (whose variables are assumed to be universally quantified) and for any substitution θ, p |= SUBST(θ; p). [subsumption]")
The rationale for combining the teachings of Lyu, Ma, Dash and Russell is the same as set forth in the rejection of claim 4.
In regard to claim 20, Lyu, Ma and Dash do not teach, but Russell teaches: execution of subsumption and instance checking on the first plurality of features using the symbolic rules. (Russell, p. 273, Inference rules for quantifiers "Let us begin with universal quantifiers. Suppose our knowledge base contains the standard folkloric axiom stating that all greedy kings are evil: for all x King(x) ^ Greedy(x) => Evil(x)… The rule of Universal Instantiation [instance checking] (UI for short) says that we can infer any sentence obtained by substituting a ground term (a term without variables) for the variable. 1 To write out the inference rule formally, we use the notion of substitutions [subsumption] introduced in Section 8.3... The corresponding Existential Instantiation rule [instance checking] for the existential quantifier is slightly more complicated...."; p. 4, A first-order inference rule, "The inference that John is evil works like this: find some x such that x is a king and x is greedy, and then infer that this x is evil. More generally, if there is some substitution θ that makes the premise of the implication identical to sentences already in the knowledge base, then we can assert the conclusion of the implication, after applying θ. In this case, the substitution {x/John} achieves that aim... for any sentence p (whose variables are assumed to be universally quantified) and for any substitution θ, p |= SUBST(θ; p). [subsumption]")
The rationale for combining the teachings of Lyu, Ma, Dash and Russell is the same as set forth in the rejection of claim 4.
Claims 5, 12 and 19 rejected under 35 U.S.C. 103 as being unpatentable over Lyu, Ma and Dash as applied to claims 1, 8 and 15, and in further view of Lee ("Attaining interpretability in reinforcement learning via hierarchical primitive composition" 20211015)
In regard to claims 5, 12 and 19, Lyu, Ma and Dash do not teach, but Lee teaches: wherein the model is trained using the first set of the first plurality of input features and a second set of the first plurality of input features which were not determined as non-interpretable. (Lee, p. 2, I. Introduction "HPC can represent the current purpose of the agent by intents and subgoals associated to each of the primitives in a human-interpretable manner, i.e., XAI capability"; p. 4 C. Model Construction "(Fig. 2a). Subgoals are seen as a directing state, e.g., it can be a goal pose of the reaching primitive, a position of a certain object to be grasped, or a target velocity of a mobile robot to be achieved. As long as the primitives require these targets as a state, the subgoal can replace these features. Simultaneously, the same state sl is also given to the state siever. Since the state space from the compound MDP is the union of all the state spaces of the primitives (*), this layer sieves and distributes the given state to the primitives accordingly, denoted as sli. The distributed state sli [original state, a second set of the first plurality of input features which were not determined as non-interpretable.] is then modified with the given subgoal glim [subgoals, the first set of the first plurality of input features which were determined as interpretable] to form a subgoal-aware state. The dimension of the state which indicates the target of the primitive is replaced with the subgoal (Fig. 2b). Given the modified states..."; see Fig. 2, the mode at level l-1 is given the compound state of the original non-interpretable state s and interpretable
PNG
media_image3.png
792
1222
media_image3.png
Greyscale
subgoal g)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified Lyu, Ma and Dash to incorporate the teachings of Lee by including interpretable subgoals with the state spaces of the primitives. Doing so would enhance sample efficiency and guarantee stability at optimization process. (Lee, p. 8, VI. Discussion and Conclusion "It can represent the intention of the agent by intents and subgoals distributed to each of the primitives in a human-interpretable manner, while reusing a pretrained primitives enhance sample efficiency and guarantees stability at optimization process.")
Response to Arguments
Applicant's amendments with respect to the claim objections have been fully considered and are sufficient to overcome the objections. The objections have been withdrawn.
Applicant's amendments with respect to 35 U.S.C. § 112 have been fully considered, but 35 U.S.C. § 112 are maintained.
Applicant's arguments with respect to the rejection of the claims under 35 U.S.C. 101 have been fully considered but they are not persuasive:
Argument: (p. 8) The Office Action alleges that the as-filed additional limitation do not integrate the judicial exception into a practical application… Applicant submits that the above-listed "additional limitations" cannot be considered "insignificant extra-solution activity" or "high level recitations". Rather, the additional limitations describe core features of iterative automated feature-engineering as described in the present application.
Response: Applicant’s amended limitations do not integrate the judicial exception into a practical application because the claim as whole reciting a learning network performing the generate/train/receive/generate steps is mere instructions to apply an exception, i.e. merely a generic computer component as a tool to carry out the underlying abstract ideas. The claim does not recite any model architecture being improved or any other technical improvement. Automated feature-engineering via iterative feature refinement or feature generation is the abstract idea being automated, therefore, it does not transform the claim to be eligible.
Argument: (p. 10-11) Applicant submits that the as-filed specification "provides sufficient details such that one of ordinary skill in the art would recognize the claimed invention as providing an improvement" in technology. For example, the as-filed Specification indicates "[s]ome embodiments provide a scalable solution to automate feature engineering that is more likely to obtain features with statistical significance and which also remain interpretable by domain experts". Specification, para. [0024]. Moreover, a required by the October 2019 Update: Subject Matter Eligibility, the claims include the components or steps that provide the improvements described in the specification. In this regard, the following claim limitations improve automated feature engineering by training a learning network to generate input features based on the performance of a learning model which is trained using generated input features and a composite interpretability of the generated input features:
Response: Paragraph [0024] describes a ‘scalable’ solution of the feature engineering, which is not enough for providing an improvement. The claim is suggested to recite specific training steps along with specific architecture that produce improvements. Further, Applicant’s argument that the claim “training a leaning network to generate input features…” does not correspond to the claim language. The claim recites the “model” that is trained is a different claimed element from “the learning network.” Reciting generic model training in the claim does not provide an improvement.
Applicant's arguments with respect to the rejection of the claims under 35 U.S.C. 103 have been fully considered but they are not persuasive or moot:
Argument: (p. 13) Ma does not describe interpretability of input features. Accordingly, Ma also fails to disclose the use of a domain ontology and on symbolic rules associated with entities of the domain ontology to determine the interpretability of input features.
Response: Examiner respectfully disagrees. Ma teaches NSRL performing neuro-symbolic reasoning as a meta-controller in the HRL to receive symbolic states and generate a subtask. Symbolic states are input features and are transformed into a matrix P. The first calculation results based on the matrix P and predicate attention weights in the reasoning module are the interpretability of input features.
Argument: (p. 13) Ma also fails to disclose or to suggest determination of a reward based on the
performance of a trained model and on a composite interpretability of a first set of input features, and generation of a second plurality of input features using the learning network based on the reward.
Response: the arguments do not apply to the references (Lyu) being used in the current rejection.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kulkarni ("Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation" 20160531) teaches (Kulkarni, p. 4 "As shown in Figure 1, the agent uses a two-stage hierarchy consisting of a controller and a meta-controller."; p. 5 "Separate deep-Q networks are used inside the meta-controller and controller.")
PNG
media_image4.png
554
1312
media_image4.png
Greyscale
PNG
media_image5.png
626
1156
media_image5.png
Greyscale
Ma2 ("Interpretable Reinforcement Learning With Neural Symbolic Logic" 2021) teaches Hierarchical Reinforcement Learning (HRL) with meta-controller and controller in Fig. 1.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SU-TING CHUANG whose telephone number is (408)918-7519. The examiner can normally be reached Monday - Thursday 8-5 PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.C./Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights