DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Application is a continuation of Provisional Application No. 63/377,502, filed
on September 28, 2022.
Response to Arguments
Applicant's arguments filed 05/22/2025 regarding the rejection under 35 USC 101 have
been fully considered but they are not persuasive.
Applicant argues, see especially page 10-11, that claims 1, 9, and 17, are patent eligible because “First, under Step 2A, Prong One, claims 1, 9, and 17 are not directed to an abstract idea. In the Office Action, the Examiner asserts the limitations of "generating, ...a predicted agent action ..." "executing a second agent action ..." recite the abstract idea of "certain methods of organizing human activities but for recitation of generic computer components". Applicant respectfully disagrees. The claims are directed to specific technical operations performed by a specific neural network architecture, for example, "a recurrent neural network based agent model" and "a recurrent neural network based intervention model" that simulate agent-intervention interactions through hidden state updates, policy optimizations, and parameter training. (Specification, [0027], [0028].) These are not activities that can be performed mentally or through human organization; they require the specific computational architecture and iterative training processes, i.e., "training the neural network based intervention model by updating parameters of the neural network based intervention model based on a first expected return computed based on incurred rewards over a plurality of time steps", of neural networks operating on probability distributions, hidden states, and gradient- based parameter updates. In particular, during the Interview, the Examiner noted "generating, ..., a predicted agent action ..." and "executing a second agent action ..." as "a certain methods of organizing human activity but for recitation of generic computer components". Applicant respectfully disagrees and submits the claim in fact recites "agent actions for a plurality of neural network based agents" comprising" generating, by a recurrent neural network based agent model implemented on one or more processors and configured to simulate the neural network based agent, a predicted agent action at a second time step conditioned on the first agent action, and the first intervention at the first time step; generating, by a recurrent neural network based intervention model implemented on the one or more processors and configured to simulate interventions against the neural network based agent, a second intervention at the second time step according to the intervention policy and conditioned on the first agent action, the first intervention, and the predicted agent action", which are not organized human activities. Unlike managing personal behavior or relationships between these limitations describe computational processes executed by specific neural network architectures that generate predictions and interventions through mathematical operations on hidden states and probability distributions-processes that have no human analog and cannot be performed through human organization. The recited "recurrent neural network based agent model" and "recurrent neural network based intervention model" are not stand-ins for human actors but are distinct technical constructs that operate through iterative state updates and policy optimizations fundamentally beyond the scope of human mental processes or interpersonal activities.” Examiner respectfully disagrees. The applicant argues “Unlike managing personal behavior or relationships between these limitations describe computational processes executed by specific neural network architectures that generate predictions and interventions through mathematical operations on hidden states and probability distributions-processes that have no human analog and cannot be performed through human organization”. However the process of intervening to correct a behavior or steer an agent to a different result is the Certain Method of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People of a Teacher-Student relationship where the teacher manages the personal behavior of the student through interactions. The Examiner points to MPEP 2106.04(a)(2)(II)(C), where the court analyze what is considered "managing personal behavior or relationships or interactions between people", such as "An example of a claim reciting social activities is Voter Verified, Inc. V. Election Systems & Software, LLC, 887 F.3d 1376, 126 USPQ2d 1498 (Fed. Cir. 2018). The social activity at issue in Voter Verified was voting. The patentee claimed "[a] method for voting providing for self-verification of a ballot comprising the steps of" presenting an election ballot for voting, accepting input of the votes, storing the votes, printing out the votes, comparing the printed votes to votes stored in the computer, and determining whether the printed ballot is acceptable. 887 F.3d at 1384-85, 126 USPQ2d at 1503-04. The Federal Circuit found that the claims were directed to the abstract idea of "voting, verifying the vote, and submitting the vote for tabulation", which is a "fundamental activity that forms the basis of our democracy" and has been performed by humans for hundreds of years. 887 F.3d at 1385-86, 126 USPQ2d at 1504-05." Related to this case, similar to the Court's reasoning, the fundamental activity of a teacher intervening in a students actions to steer them toward the intended goal has been performed by humans for hundreds or thousands of year as an more expertise person guides a student through interventions in order to help them learn.
Applicant argues, see especially page 11-12, that claims 1, 9, and 17, are patent eligible because “Second, under Step 2A, Prong Two, even if considered as reciting an abstract idea (which Applicant does not concede), claim 1 as a whole integrates the alleged abstract idea into a practical application of improving the hardware efficiency and adaptability of AI agent systems, and thus is eligible under Step 2A, Prong Two. In particular, the 2025 Memo requires the Examiner to consult the specification to determine whether the disclosed invention improves technology or a technical field, and evaluate the claim to ensure it reflects the disclosed improvement. (2025 Memo, 4). Here, the Specification describes a specific technical problem: " execution of the designed mechanism [of AI agents] can be costly" for different types of real-world applications (Specification, [0003]). In response, the claimed invention provides a neural network model-based meta-learning framework "adapted to out-of-distribution agents with different learning strategies and reward functions, thereby reducing the cost of learning." (Specification, [0020]). Specifically, the framework achieves "a cost-effective mechanism that is K-shot adaptable with only partial information about the agents," and "even for learning agent having out-of-distribution actions or unseen explore-exploit behaviors, the principal can learn with only a few interactions, reducing the cost for real-world experiment." (Specification, [0024]). Therefore, the amended claims are eligible as an improvement to AI agent technology. Amended claim 1 reflects "an improvement" to a problem in computer technology relating to the costly and inefficient training of mechanism design systems. The Specification explains that "mechanism design studies how to design a gaming system, e.g., the reward functions and environment rules, that are implemented by a set of intelligent agents" and that "the execution of the designed mechanism can be costly." (Specification, [0003], [0019]). Claim 1 addresses this by reciting a specific technical solution: using a "neural network based agent model" that generates "a predicted agent action at a second time step conditioned on the first agent action, and the first intervention at the first time step," combined with a "neural network based intervention model" that generates "a second intervention at the second time step according to the intervention policy and conditioned on the first agent action, the first intervention, and the predicted agent action." This specific architecture enables the intervention model to "learn to appropriately incentivize the agent model while minimizing the total cost of intervening." (Specification, [0107]). The training process, which updates "parameters of the neural network based intervention model based on a first expected return computed based on incurred rewards over a plurality of time steps," enables the framework to achieve "strong few-shot generalization in the real world." (Specification, [0024]).” Examiner respectfully disagrees. The applicant argues that “Here, the Specification describes a specific technical problem: " execution of the designed mechanism [of AI agents] can be costly" for different types of real-world applications (Specification, [0003]). In response, the claimed invention provides a neural network model-based meta-learning framework "adapted to out-of-distribution agents with different learning strategies and reward functions, thereby reducing the cost of learning." (Specification, [0020]).”. However, the claims in light of the specification do not provide enough details about the costs of running these mechanisms for one of ordinary skill in the art to recognize the improvement. It is not clear from the specification or the claims what costs are being reduced and how these costs are reduced by the alleged improvement.
Applicant argues, see especially page 10-11, that claims 1, 9, and 17, are patent eligible because “Third, various case law and/or PTAB decisions have ruled similar subject matter relating to improving neural networks or machine learning systems as eligible. For example, Applicant respectfully points the Examiner to the latest decision in Appeal 2024-000567, in which Director John A. Squires decided a method claim of "training a machine learning model" is eligible. (Decision on Appeal 2024-000567, App. 16/319,040). Specifically, Director Squires found the claim of training a machine learning model reflects an improvement to "effectively learn new tasks in succession whilst protecting knowledge about previous tasks" and thus allowed AI systems to "use less of their storage capacity" as described in the specification (Id. at 9). In particular, Director Squires emphasized that the panel "essentially equated any machine learning with an unpatentable 'algorithm' and the remaining additional elements as "generic computer components,' without adequate explanation," and urged that "Examiners and panels should not evaluate claims at such a high level of generality." (Id.). Applicant respectfully submits the pending claims, which reflect an improvement to AI agents with cost-efficient few-shot adaptability to out-of-distribution agents with unseen learning behaviors as described in the specification, are thus eligible subject matter at least under Step 2A Prong Two. Thus, the claim as a whole is directed to a technological improvement, not merely the application of generic machine learning in a new context. The mere presence of terms like "agent" or "intervention" in claim 1 does not render the claim abstract or generic; rather, it is the specific manner in which the "recurrent neural network based agent model" and the "recurrent neural network based intervention model" are configured to simulate agent-intervention interactions and are trained through the specific expected return optimization that makes the invention patent-eligible under § 101.” Examiner respectfully disagrees. The claims in light of the specification do not provide enough details about the costs of running these mechanisms for one of ordinary skill in the art to recognize the improvement. It is not clear from the specification or the claims what costs are being reduced and how these costs are reduced by the alleged improvement. In the decision in Appeal 2024-000567 it is made clear that the improvement is explicitly stated in enough detail in the specification for one of ordinary skill in the art to recognize the improvement in the claims. However, in this case it is not clear from the specification and the claims how and what cost is lowered through this alleged improvement.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea (mental process) without significantly more.
Regarding claim 1, in Step 1 of the 101 analyses set forth in MPEP 2106, the claim recites A method for predicting agent actions for a plurality of neural network based agents using agent-intervention simulation. A method is one of the four statutory categories.
In Step 2a Prong 1 of the 101 analyses set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a certain method of organizing human activity and mental process but for recitation of generic computer components:
generating, …1, a predicted agent action at a second time step conditioned on the first agent action, and the first intervention at the first time step; generating, …2, a second intervention at the second time step according to the intervention policy and conditioned on the first agent action, the first intervention, and the predicted agent action; (The predicting and intervening in an agents actions based on a policy represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A teacher is able to predict the actions of students and intervene in order to correct the lesson being learned in order to teach the student.)
executing a second agent action according to an agent policy at the second time step that incurs a reward that is based on the second intervention at the second time step; and (The agent taking an action based the interventions represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A student that takes an action based on a teacher’s intervention and receiving a reward represents actions of teaching.)
performing a mechanism design based on the agent-intervention simulation; (A person can mentally perform mechanism design based on a simulation by a process of simply evaluating the agent-intervention simulation, and making a judgement on how the mechanism design should be designed MPEP 2106.04(a)(2)).
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a certain method of organizing human activity but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. According, the claim “recites” an abstract idea.
In Step 2a Prong 2 of the 101 analyses set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
obtaining a first agent action performed by a neural network based agent of the plurality of neural network based agents at a first time step and a first intervention that is generated according to an intervention policy at the first time step; (Adding insignificant extra-solution activity (mere data gathering) to the judicial exception (MPEP 2106.05(g))).
…1 by a recurrent neural network based agent model implemented on one or more processors and configured to simulate the neural network based agent … (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))).
…2 by a recurrent neural network based agent model implemented on the one or more processors and configured to simulate interventions against the neural network based agent … (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))).
training the neural network based intervention model by updating parameters of the neural network based intervention model based on a first expected return computed based on incurred rewards over a plurality of time steps (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))).
and displaying, to a user, the mechanism design on a user interface (UI). (In step 2A, prong 2, this recites insignificant extra solution activity of mere data Output, which is not indicative of integration into a practical application (MPEP 2106.05(g)). In step 2B, this recites receiving or sending data over a network which is a well-understood, routine and conventional activity, which is not indicative of significantly more.)
Since the claim does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In Step 2b of the 101 analyses set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
obtaining a first agent action performed by a neural network based agent of the plurality of neural network based agents at a first time step and a first intervention that is generated according to an intervention policy at the first time step; (Adding insignificant extra-solution activity (mere data gathering) to the judicial exception (MPEP 2106.05(g)) Furthermore, the additional element is directed to receiving or transmitting data over a network / performing repetitive calculations / electronic recordkeeping / storing and retrieving information in memory / electronically scanning or extracting data from a physical document, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).).
…1 by a recurrent neural network based agent model implemented on one or more processors and configured to simulate the neural network based agent … (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))) Furthermore, the additional element is directed to application of a computer tool (machine learning model), which is not indicative of significantly more (MPEP 2106.05(f)).)
…2 by a recurrent neural network based agent model implemented on the one or more processors and configured to simulate interventions against the neural network based agent … (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))) Furthermore, the additional element is directed to application of a computer tool (machine learning model), which is not indicative of significantly more (MPEP 2106.05(f)).)
training the neural network based intervention model by updating parameters of the neural network based intervention model based on a first expected return computed based on incurred rewards over a plurality of time steps. (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))) Furthermore, the additional element is directed to application of a computer tool (machine learning model), which is not indicative of significantly more (MPEP 2106.05(f)).)
and displaying, to a user, the mechanism design on a user interface (UI). (In step 2A, prong 2, this recites insignificant extra solution activity of mere data Output, which is not indicative of integration into a practical application (MPEP 2106.05(g)). In step 2B, this recites receiving or sending data over a network which is a well-understood, routine and conventional activity, which is not indicative of significantly more.) Furthermore, the additional element is directed to receiving or transmitting data over a network / performing repetitive calculations / electronic recordkeeping / storing and retrieving information in memory / electronically scanning or extracting data from a physical document, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).).
Regarding claim 2 it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 2 recites wherein the first expected return is computed based on the incurred rewards and intervention costs associated with the interventions over the plurality of time steps. (In step 2A, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h))). In step 2B, generally linking the use of the judicial exception to a particular technological environment is not indicative of significantly more.)
Regarding claim 3 it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 3 recites updating the agent policy by maximizing a second expected return computed based on incurred rewards including the incurred reward at the second time step, prior to the updating of the parameters of the neural network based intervention model. (In step 2A, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h))). In step 2B, generally linking the use of the judicial exception to a particular technological environment is not indicative of significantly more.)
Regarding claim 4 it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 4 recites wherein the second agent action is determined by sampling the second agent action according to the agent policy at the second time step. (In step 2A, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h))). In step 2B, generally linking the use of the judicial exception to a particular technological environment is not indicative of significantly more.)
Regarding claim 5 it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 5 recites wherein the generating of the second intervention at the second time step includes: generating, by the neural network based intervention model, a distribution over interventions; and sampling the second intervention according to the generated distribution. (Creating a distribution and sampling from it is directed to a mathematical calculation and thus is not patent eligible (MPEP 2106).)
Regarding claim 6 it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 6 recites wherein the updating of the parameters of the neural network based intervention model is performed at an end of the plurality of time steps including the first time step and the second time step. (In step 2A prong 2, merely training a generic machine learning operation constitutes “applying” the machine learning operation (MPEP 2106.05(f)). In step 2B, merely applying a generic machine learning operation is not indicative of significantly more.)
Regarding claim 7 it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 7 recites generating, by the neural network based agent model, (In step 2A prong 2, merely utilizing a generic machine learning operation constitutes “applying” the machine learning operation (MPEP 2106.05(f)). In step 2B, merely applying a generic machine learning operation is not indicative of significantly more.) a second predicted agent action at a fourth time step conditioned on a third agent action at a third time step, and a third intervention at the third time step, the third time step being after the plurality of time steps; (The predicting and intervening in an agents actions based on a policy represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A teacher is able to predict the actions of students and intervene in order to correct the lesson being learned in order to teach the student.) generating, by a neural network based intervention model, (In step 2A prong 2, merely utilizing a generic machine learning operation constitutes “applying” the machine learning operation (MPEP 2106.05(f)). In step 2B, merely applying a generic machine learning operation is not indicative of significantly more.) a fourth intervention at the fourth time step according to the intervention policy after the plurality of time steps and conditioned on the third agent action, the third intervention, and the second predicted agent action; (The predicting and intervening in an agents actions based on a policy represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A teacher is able to predict the actions of students and intervene in order to correct the lesson being learned in order to teach the student.) executing a fourth agent action at the fourth time step that incurs a reward that is based on the fourth intervention at the fourth time step; (The agent taking an action based the interventions represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A student that takes an action based on a teacher’s intervention and receiving a reward represents actions of teaching.) collecting a rollout including the fourth agent action, the fourth intervention, and an intervention distribution after the plurality of time steps; and (In step 2A, prong 2, this recites insignificant extra solution activity of mere data gathering, which is not indicative of integration into a practical application (MPEP 2106.05(g)). In step 2B, this recites receiving data over a network which is a well-understood, routine and conventional activity, which is not indicative of significantly more.) training the neural network based intervention model by updating parameters of the neural network based intervention model based on collected rollouts over a second plurality of time steps. (In step 2A prong 2, merely training a generic machine learning operation constitutes “applying” the machine learning operation (MPEP 2106.05(f)). In step 2B, merely applying a generic machine learning operation is not indicative of significantly more.)
Regarding claim 8 it is dependent upon claim 7, and thereby incorporates the limitations of, and corresponding analysis applied to claim 7. Further, claim 8 recites training the neural network based agent model by maximizing a log-likelihood of expected agent actions over the first or second plurality of time steps. (Maximizing a log-likelyhood function is directed to a mathematical calculation and thus is not patent eligible (MPEP 2106).)
Regarding claim 9, in Step 1 of the 101 analyses set forth in MPEP 2106, the claim recites A system for predicting agent actions for a plurality of neural network based agents according to an intervention input, the system comprising. A system is one of the four statutory categories.
In Step 2a Prong 1 of the 101 analyses set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a represents certain methods of organizing human activity but for recitation of generic computer components:
generating, …1, a predicted agent action at a second time step conditioned on the first agent action, and the first intervention at the first time step; generating, …2, a second intervention at the second time step according to the intervention policy and conditioned on the first agent action, the first intervention, and the predicted agent action; (The predicting and intervening in an agents actions based on a policy represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A teacher is able to predict the actions of students and intervene in order to correct the lesson being learned in order to teach the student.)
executing a second agent action according to an agent policy at the second time step that incurs a reward that is based on the second intervention at the second time step; and (The agent taking an action based the interventions represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A student that takes an action based on a teacher’s intervention and receiving a reward represents actions of teaching.)
performing a mechanism design based on the agent-intervention simulation; (A person can mentally perform mechanism design based on a simulation by a process of simply evaluating the agent-intervention simulation, and making a judgement on how the mechanism design should be designed MPEP 2106.04(a)(2)).
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a certain methods of organizing human activity but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. According, the claim “recites” an abstract idea.
In Step 2a Prong 2 of the 101 analyses set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
a memory that stores a neural network based agent model and a neural network based intervention model, and a plurality of processor executable instructions; (Generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h))).
a communication interface that receives a first agent action performed by a neural network based agent of the plurality of neural network based agents at a first time step and a first intervention that is generated according to an intervention policy at the first time step; and (Adding insignificant extra-solution activity (mere data gathering) to the judicial exception (MPEP 2106.05(g))).
one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising: (Generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h))).
…1 by a recurrent neural network based agent model implemented on one or more processors and configured to simulate the neural network based agent … (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))).
…2 by a recurrent neural network based agent model implemented on the one or more processors and configured to simulate interventions against the neural network based agent … (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))).
training the neural network based intervention model by updating parameters of the neural network based intervention model based on a first expected return computed based on incurred rewards over a plurality of time steps. (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))).
and displaying, to a user, the mechanism design on a user interface (UI). (In step 2A, prong 2, this recites insignificant extra solution activity of mere data Output, which is not indicative of integration into a practical application (MPEP 2106.05(g)). In step 2B, this recites receiving or sending data over a network which is a well-understood, routine and conventional activity, which is not indicative of significantly more.)
Since the claim does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In Step 2b of the 101 analyses set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
a memory that stores a neural network based agent model and a neural network based intervention model, and a plurality of processor executable instructions; (Generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h)) Furthermore, the additional element is directed to generally linking the use of the judicial exception to a particular technological environment or field of use, which is not indicative of significantly more.).
a communication interface that receives a first agent action performed by a neural network based agent of the plurality of neural network based agents at a first time step and a first intervention that is generated according to an intervention policy at the first time step; and (Adding insignificant extra-solution activity (mere data gathering) to the judicial exception (MPEP 2106.05(g)) Furthermore, the additional element is directed to receiving or transmitting data over a network / performing repetitive calculations / electronic recordkeeping / storing and retrieving information in memory / electronically scanning or extracting data from a physical document, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).).
one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising: (Generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h)) Furthermore, the additional element is directed to generally linking the use of the judicial exception to a particular technological environment or field of use, which is not indicative of significantly more.).
…1 by a recurrent neural network based agent model implemented on one or more processors and configured to simulate the neural network based agent … (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))) Furthermore, the additional element is directed to application of a computer tool (machine learning model), which is not indicative of significantly more (MPEP 2106.05(f)).)
…2 by a recurrent neural network based agent model implemented on the one or more processors and configured to simulate interventions against the neural network based agent … (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))) Furthermore, the additional element is directed to application of a computer tool (machine learning model), which is not indicative of significantly more (MPEP 2106.05(f)).)
training the neural network based intervention model by updating parameters of the neural network based intervention model based on a first expected return computed based on incurred rewards over a plurality of time steps. (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))) Furthermore, the additional element is directed to application of a computer tool (machine learning model), which is not indicative of significantly more (MPEP 2106.05(f)).)
and displaying, to a user, the mechanism design on a user interface (UI). (In step 2A, prong 2, this recites insignificant extra solution activity of mere data Output, which is not indicative of integration into a practical application (MPEP 2106.05(g)). In step 2B, this recites receiving or sending data over a network which is a well-understood, routine and conventional activity, which is not indicative of significantly more.) Furthermore, the additional element is directed to receiving or transmitting data over a network / performing repetitive calculations / electronic recordkeeping / storing and retrieving information in memory / electronically scanning or extracting data from a physical document, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).).
Regarding claim 10 it is dependent upon claim 9, and thereby incorporates the limitations of, and corresponding analysis applied to claim 10. Further, claim 0 recites wherein the first expected return is computed based on the incurred rewards and intervention costs associated with the interventions over the plurality of time steps. (In step 2A, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h))). In step 2B, generally linking the use of the judicial exception to a particular technological environment is not indicative of significantly more.)
Regarding claim 11 it is dependent upon claim 9, and thereby incorporates the limitations of, and corresponding analysis applied to claim 9. Further, claim 11 recites updating the agent policy by maximizing a second expected return computed based on incurred rewards including the incurred reward at the second time step, prior to the updating of the parameters of the neural network based intervention model. (In step 2A, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h))). In step 2B, generally linking the use of the judicial exception to a particular technological environment is not indicative of significantly more.)
Regarding claim 12 it is dependent upon claim 9, and thereby incorporates the limitations of, and corresponding analysis applied to claim 9. Further, claim 12 recites wherein the second agent action is determined by sampling the second agent action according to the agent policy at the second time step. (In step 2A, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h))). In step 2B, generally linking the use of the judicial exception to a particular technological environment is not indicative of significantly more.)
Regarding claim 13 it is dependent upon claim 9, and thereby incorporates the limitations of, and corresponding analysis applied to claim 9. Further, claim 13 recites wherein the generating of the second intervention at the second time step includes: generating, by the neural network based intervention model, a distribution over interventions; and sampling the second intervention according to the generated distribution. (Creating a distribution and sampling from it is directed to a mathematical calculation and thus is not patent eligible (MPEP 2106).)
Regarding claim 14 it is dependent upon claim 9, and thereby incorporates the limitations of, and corresponding analysis applied to claim 9. Further, claim 14 recites wherein the updating of the parameters of the neural network based intervention model is performed at an end of the plurality of time steps including the first time step and the second time step. (In step 2A prong 2, merely training a generic machine learning operation constitutes “applying” the machine learning operation (MPEP 2106.05(f)). In step 2B, merely applying a generic machine learning operation is not indicative of significantly more.)
Regarding claim 15 it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 15 recites generating, by the neural network based agent model, (In step 2A prong 2, merely utilizing a generic machine learning operation constitutes “applying” the machine learning operation (MPEP 2106.05(f)). In step 2B, merely applying a generic machine learning operation is not indicative of significantly more.) a second predicted agent action at a fourth time step conditioned on a third agent action at a third time step, and a third intervention at the third time step, the third time step being after the plurality of time steps; (The predicting and intervening in an agents actions based on a policy represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A teacher is able to predict the actions of students and intervene in order to correct the lesson being learned in order to teach the student.) generating, by a neural network based intervention model, (In step 2A prong 2, merely utilizing a generic machine learning operation constitutes “applying” the machine learning operation (MPEP 2106.05(f)). In step 2B, merely applying a generic machine learning operation is not indicative of significantly more.) a fourth intervention at the fourth time step according to the intervention policy after the plurality of time steps and conditioned on the third agent action, the third intervention, and the second predicted agent action; (The predicting and intervening in an agents actions based on a policy represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A teacher is able to predict the actions of students and intervene in order to correct the lesson being learned in order to teach the student.) executing a fourth agent action at the fourth time step that incurs a reward that is based on the fourth intervention at the fourth time step; (The agent taking an action based the interventions represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A student that takes an action based on a teacher’s intervention and receiving a reward represents actions of teaching.) collecting a rollout including the fourth agent action, the fourth intervention, and an intervention distribution after the plurality of time steps; and (In step 2A, prong 2, this recites insignificant extra solution activity of mere data gathering, which is not indicative of integration into a practical application (MPEP 2106.05(g)). In step 2B, this recites receiving data over a network which is a well-understood, routine and conventional activity, which is not indicative of significantly more.) training the neural network based intervention model by updating parameters of the neural network based intervention model based on collected rollouts over a second plurality of time steps. (In step 2A prong 2, merely training a generic machine learning operation constitutes “applying” the machine learning operation (MPEP 2106.05(f)). In step 2B, merely applying a generic machine learning operation is not indicative of significantly more.)
Regarding claim 16 it is dependent upon claim 15, and thereby incorporates the limitations of, and corresponding analysis applied to claim 7. Further, claim 8 recites training the neural network based agent model by maximizing a log-likelihood of expected agent actions over the first or second plurality of time steps. (Maximizing a log-likelyhood function is directed to a mathematical calculation and thus is not patent eligible (MPEP 2106).)
Regarding claim 17, in Step 1 of the 101 analyses set forth in MPEP 2106, the claim recites A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising. A non- transitory machine-readable medium is a manufacture which is one of the four statutory categories.
In Step 2a Prong 1 of the 101 analyses set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a certain methods of organizing human activity but for recitation of generic computer components:
generating, …1, a predicted agent action at a second time step conditioned on the first agent action, and the first intervention at the first time step; generating, …2, a second intervention at the second time step according to the intervention policy and conditioned on the first agent action, the first intervention, and the predicted agent action; (The predicting and intervening in an agents actions based on a policy represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A teacher is able to predict the actions of students and intervene in order to correct the lesson being learned in order to teach the student.)
executing a second agent action according to an agent policy at the second time step that incurs a reward that is based on the second intervention at the second time step; and (The agent taking an action based the interventions represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A student that takes an action based on a teacher’s intervention and receiving a reward represents actions of teaching.)
performing a mechanism design based on the agent-intervention simulation; (A person can mentally perform mechanism design based on a simulation by a process of simply evaluating the agent-intervention simulation, and making a judgement on how the mechanism design should be designed MPEP 2106.04(a)(2)).
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a certain methods of organizing human activity but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. According, the claim “recites” an abstract idea.
In Step 2a Prong 2 of the 101 analyses set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
obtaining a first agent action performed by a neural network based agent of the plurality of neural network based agents at a first time step and a first intervention that is generated according to an intervention policy at the first time step; (Adding insignificant extra-solution activity (mere data gathering) to the judicial exception (MPEP 2106.05(g))).
…1 by a recurrent neural network based agent model implemented on one or more processors and configured to simulate the neural network based agent… (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))).
…2 by a recurrent neural network based agent model implemented on the one or more processors and configured to simulate interventions against the neural network based agent … (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))).
training the neural network based intervention model by updating parameters of the neural network based intervention model based on a first expected return computed based on incurred rewards over a plurality of time steps. (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))).
and displaying, to a user, the mechanism design on a user interface (UI). (In step 2A, prong 2, this recites insignificant extra solution activity of mere data Output, which is not indicative of integration into a practical application (MPEP 2106.05(g)). In step 2B, this recites receiving or sending data over a network which is a well-understood, routine and conventional activity, which is not indicative of significantly more.)
Since the claim does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In Step 2b of the 101 analyses set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
obtaining a first agent action performed by a neural network based agent of the plurality of neural network based agents at a first time step and a first intervention that is generated according to an intervention policy at the first time step; (Adding insignificant extra-solution activity (mere data gathering) to the judicial exception (MPEP 2106.05(g)) Furthermore, the additional element is directed to receiving or transmitting data over a network / performing repetitive calculations / electronic recordkeeping / storing and retrieving information in memory / electronically scanning or extracting data from a physical document, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).).
…1 by a recurrent neural network based agent model implemented on one or more processors and configured to simulate the neural network based agent … (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))) Furthermore, the additional element is directed to application of a computer tool (machine learning model), which is not indicative of significantly more (MPEP 2106.05(f)).)
…2 by a recurrent neural network based agent model implemented on the one or more processors and configured to simulate interventions against the neural network based agent … (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))) Furthermore, the additional element is directed to application of a computer tool (machine learning model), which is not indicative of significantly more (MPEP 2106.05(f)).)
training the neural network based intervention model by updating parameters of the neural network based intervention model based on a first expected return computed based on incurred rewards over a plurality of time steps. (Merely utilizing a generic machine learning model constitutes “applying” the machine learning model (MPEP 2106.05(f))) Furthermore, the additional element is directed to application of a computer tool (machine learning model), which is not indicative of significantly more (MPEP 2106.05(f)).)
and displaying, to a user, the mechanism design on a user interface (UI). (In step 2A, prong 2, this recites insignificant extra solution activity of mere data Output, which is not indicative of integration into a practical application (MPEP 2106.05(g)). In step 2B, this recites receiving or sending data over a network which is a well-understood, routine and conventional activity, which is not indicative of significantly more.) Furthermore, the additional element is directed to receiving or transmitting data over a network / performing repetitive calculations / electronic recordkeeping / storing and retrieving information in memory / electronically scanning or extracting data from a physical document, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).).
Regarding claim 18 it is dependent upon claim 17, and thereby incorporates the limitations of, and corresponding analysis applied to claim 17. Further, claim 18 recites wherein the first expected return is computed based on the incurred rewards and intervention costs associated with the interventions over the plurality of time steps. (In step 2A, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h))). In step 2B, generally linking the use of the judicial exception to a particular technological environment is not indicative of significantly more.)
Regarding claim 19 it is dependent upon claim 17, and thereby incorporates the limitations of, and corresponding analysis applied to claim 17. Further, claim 19 recites updating the agent policy by maximizing a second expected return computed based on incurred rewards including the incurred reward at the second time step, prior to the updating of the parameters of the neural network based intervention model. (In step 2A, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h))). In step 2B, generally linking the use of the judicial exception to a particular technological environment is not indicative of significantly more.)
Regarding claim 20 it is dependent upon claim 17, and thereby incorporates the limitations of, and corresponding analysis applied to claim 17. Further, claim 20 recites generating, by the neural network based agent model, (In step 2A prong 2, merely utilizing a generic machine learning operation constitutes “applying” the machine learning operation (MPEP 2106.05(f)). In step 2B, merely applying a generic machine learning operation is not indicative of significantly more.) a second predicted agent action at a fourth time step conditioned on a third agent action at a third time step, and a third intervention at the third time step, the third time step being after the plurality of time steps; (The predicting and intervening in an agents actions based on a policy represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A teacher is able to predict the actions of students and intervene in order to correct the lesson being learned in order to teach the student.) generating, by a neural network based intervention model, (In step 2A prong 2, merely utilizing a generic machine learning operation constitutes “applying” the machine learning operation (MPEP 2106.05(f)). In step 2B, merely applying a generic machine learning operation is not indicative of significantly more.) a fourth intervention at the fourth time step according to the intervention policy after the plurality of time steps and conditioned on the third agent action, the third intervention, and the second predicted agent action; (The predicting and intervening in an agents actions based on a policy represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A teacher is able to predict the actions of students and intervene in order to correct the lesson being learned in order to teach the student.) executing a fourth agent action at the fourth time step that incurs a reward that is based on the fourth intervention at the fourth time step; (The agent taking an action based the interventions represents Certain Methods Of Organizing Human Activity of Managing Personal Behavior or Relationships or Interactions Between People grouping of abstract ideas (MPEP 2106.04(a)(2)). A student that takes an action based on a teacher’s intervention and receiving a reward represents actions of teaching.) collecting a rollout including the fourth agent action, the fourth intervention, and an intervention distribution after the plurality of time steps; and (In step 2A, prong 2, this recites insignificant extra solution activity of mere data gathering, which is not indicative of integration into a practical application (MPEP 2106.05(g)). In step 2B, this recites receiving data over a network which is a well-understood, routine and conventional activity, which is not indicative of significantly more.) training the neural network based intervention model by updating parameters of the neural network based intervention model based on collected rollouts over a second plurality of time steps. (In step 2A prong 2, merely training a generic machine learning operation constitutes “applying” the machine learning operation (MPEP 2106.05(f)). In step 2B, merely applying a generic machine learning operation is not indicative of significantly more.)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 9-14, and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Guest et al. Pub. No.: US 20240424245 A1 in view of Fachantidis et al. “Learning to Teach Reinforcement Learning Agents” 12/06/2017
Regarding claim 1 Guest teaches A method for predicting agent actions for a plurality of neural network based agents using agent-intervention simulation, the method comprising: obtaining a first agent action …1 at a first time step and a first intervention that is generated according to an intervention policy at the first time step; (Guest, paragraph 0127, teaches the use of a user’s (i.e. agent) actions at a time step and an intervention model that uses a policy to determine when to intervene in the agents actions. ) generating, by a recurrent neural network based agent model implemented on one or more processors and configured to simulate the neural network based agent, a predicted agent action at a second time step conditioned on the first agent action, and the first intervention at the first time step; (Guest, paragraph 0099-0101, teaches a likelihood engine that is used to predict the next action that a user (i.e. agent) will take based on the previous action that the user had taken. The likelihood engine is a neural network.) generating, by a recurrent neural network based intervention model implemented on the one or more processors and configured to simulate interventions against the neural network based agent,, a second intervention at the second time step according to the intervention policy and conditioned on the first agent action, the first intervention, and the predicted agent action; (Guest, paragraph 0102-0103, teaches the use of an intervention engine that gets the predicted action from the likelihood engine and determines if an intervention is needed and what intervention is needed at the time based on its policy. Guest, paragraph 0056, teaches the intervention model being a neural network.) …2 that is based on the second intervention at the second time step; and (Guest, paragraph 0103, teaches an intervention effectiveness engine that after the execution of an intervention and the action that is taken by the user (i.e. agent) with that intervention it then determines how effective that intervention was in getting the desired outcome this represents a reward in that a reward is a score that determines how effective or good an action was.) training the neural network based intervention model by updating parameters of the neural network based intervention model based on a first expected return computed based on incurred rewards over a plurality of time steps …3 (Guest, paragraph 104 and 0139-145, teaches the training and adjusting of the intervention model based on the outcome of the intervention based on the action taken by the user in response to the intervention and the intervention effectiveness over time.) …3
Guest does not teach …1 performed by a neural network based agent of the plurality of neural network based agents… However, Fachantidis in analogous art teaches this limitation (Fachantids. Page 25, section 3.1 teaches a student model neural network agent that performs tasks in an environment and is capable of receiving outside advice (i.e. interventions) from another agent.)
Further Guest does not teach …2 executing a second agent action according to an agent policy at the second time step that incurs a reward … However, Fachantidis in analogous art teaches this limitation (Fachantidis, page 35, Section 5.3, and algorithm 1, teaches the agent taking a second action after receiving advice (i.e. intervention) from the teacher model and incurring a reward based on the action and the advice that was given to the student model.)
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine Fachantidis’ teaching of a neural network agent that is able to receive and act on interventions with Guest’s teaching of a meta learning system. The motivation to do so would be to be able to generate more examples for the intervention model to train on and to have the intervention model be able to interact with machine learning based agents.
The combination of Guest and Fachantidis does not teach …3 performing a mechanism design based on the agent-intervention simulation; and displaying, to a user, the mechanism design on a user interface (UI). However, Wels in analogous art teaches this limitation (Wels, paragraph 0029, teaches the Machine learning displaying results and the model structure to a user interface in.)
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine Weis’ teaching of a user interface the displays a machine learning model and its results with the combination of Guest and Fachantidis teaching of a meta learning system. The motivation to do so would be to allow users to visualize the neural network structure and the results that it produces.
Regarding claim 2 The combination of Guest, Fachantidis and Wels teaches The method of claim 1, wherein the first expected return is computed based on the incurred rewards and intervention costs associated with the interventions over the plurality of time steps. (Guest, paragraph 0103-0104 and 0139-0145, teaches the use of an intervention engine that integrates the use of a loss function to determine the costs of the intervention and an intervention effectiveness model that determines how effective an intervention was in getting the desired outcome this represents a reward in that a reward is a score that determines how effective or good an action was.)
Regarding claim 3 The combination of Guest, Fachantidis and Wels teaches The method of claim 1, further comprising: updating the agent policy by maximizing a second expected return computed based on incurred rewards including the incurred reward at the second time step, prior to the updating of the parameters of the neural network based intervention model. (Guest, paragraph 0095-0104 and 0139-0145, teaches the system of the intervention model and how it is trained over multiple iterations at different points in time (i.e. second time step) the model will learn what interventions will work best at each different point in time in order to achieve a desired result. Further it teaches the use of an intervention engine that integrates the use of a loss function to determine the costs of the intervention and an intervention effectiveness model that determines how effective an intervention was in getting the desired outcome this represents a reward in that a reward is a score that determines how effective or good an action was.)
Regarding claim 4 The combination of Guest, Fachantidis and Wels teaches The method of claim 1, wherein the second agent action is determined by sampling the second agent action according to the agent policy at the second time step. (Guest, paragraph 0093, 0127-0130, teaches the use of a user and user information that allows the system to determine what behaviors and actions are taking place before and after an intervention has been implemented, using this history of behaviors and actions the system is able to predict a user’s behavior after an intervention and determine what that second action will be based on that prior data (i.e. agent policy).)
Regarding claim 5 The combination of Guest, Fachantidis and Wels teaches The method of claim 1, wherein the generating of the second intervention at the second time step includes: generating, by the neural network based intervention model, a distribution over interventions; and sampling the second intervention according to the generated distribution. (Guest, paragraph 0102-0104, teaches the use of an intervention engine (i.e. neural network intervention model) that takes in a distribution of the likelihood of the behaviors of a user based on the possible interventions and allows for the intervention engine to determine the best intervention to sample based on the list of interventions.)
Regarding claim 6 The combination of Guest, Fachantidis and Wels teaches The method of claim 1, wherein the updating of the parameters of the neural network based intervention model is performed at an end of the plurality of time steps including the first time step and the second time step. (Guest, paragraph 0104, 0117-0119, FIG 5, teaches the adjusting of the intervention engine (i.e. intervention model) being performed after a plurality of time steps being performed.)
Regarding claim 9 Guest teaches A system for predicting agent actions for a plurality of neural network based agents using agent-intervention simulation, the system comprising: a memory that stores a neural network based agent model and a neural network based intervention model, and a plurality of processor executable instructions; a communication interface that receives a first agent action …1 at a first time step and a first intervention that is generated according to an intervention policy at the first time step; and one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising: (Guest, paragraph 0127, teaches the use of a user’s (i.e. agent) actions at a time step and an intervention model that uses a policy to determine when to intervene in the agents actions. ) generating, by a recurrent neural network based agent model implemented on one or more processors and configured to simulate the neural network based agent, a predicted agent action at a second time step conditioned on the first agent action, and the first intervention at the first time step; (Guest, paragraph 0099-0101, teaches a likelihood engine that is used to predict the next action that a user (i.e. agent) will take based on the previous action that the user had taken. The likelihood engine is a neural network.) generating, by a neural network based intervention model, a second intervention at the second time step according to the intervention policy and conditioned on the first agent action, the first intervention, and the predicted agent action; (Guest, paragraph 0102-0103, teaches the use of an intervention engine that gets the predicted action from the likelihood engine and determines if an intervention is needed and what intervention is needed at the time based on its policy. Guest, paragraph 0056, teaches the intervention model being a neural network.) executing a second agent action according to an agent policy at the second time step that incurs a reward that is based on the second intervention at the second time step; and (Guest, paragraph 0103, teaches an intervention effectiveness engine that after the execution of an intervention and the action that is taken by the user (i.e. agent) with that intervention it then determines how effective that intervention was in getting the desired outcome this represents a reward in that a reward is a score that determines how effective or good an action was.) training the neural network based intervention model by updating parameters of the neural network based intervention model based on a first expected return computed based on incurred rewards over a plurality of time steps …3 (Guest, paragraph 104 and 0139-145, teaches the training and adjusting of the intervention model based on the outcome of the intervention based on the action taken by the user in response to the intervention and the intervention effectiveness over time.)
Guest does not teach …1 performed by a neural network based agent of the plurality of neural network based agents… However, Fachantidis in analogous art teaches this limitation (Fachantids. Page 25, section 3.1 teaches a student model neural network agent that performs tasks in an environment and is capable of receiving outside advice (i.e. interventions) from another agent.)
Further Guest does not teach …2 executing a second agent action according to an agent policy at the second time step that incurs a reward … However, Fachantidis in analogous art teaches this limitation (Fachantidis, page 35, Section 5.3, and algorithm 1, teaches the agent taking a second action after receiving advice (i.e. intervention) from the teacher model and incurring a reward based on the action and the advice that was given to the student model.)
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine Fachantidis’ teaching of a neural network agent that is able to receive and act on interventions with Guest’s teaching of a meta learning system. The motivation to do so would be to be able to generate more examples for the intervention model to train on and to have the intervention model be able to interact with machine learning based agents.
The combination of Guest and Fachantidis does not teach …3 performing a mechanism design based on the agent-intervention simulation; and displaying, to a user, the mechanism design on a user interface (UI). However, Wels in analogous art teaches this limitation (Wels, paragraph 0029, teaches the Machine learning displaying results and the model structure to a user interface in.)
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine Weis’ teaching of a user interface the displays a machine learning model and its results with the combination of Guest and Fachantidis teaching of a meta learning system. The motivation to do so would be to allow users to visualize the neural network structure and the results that it produces.
Regarding claim 10 The combination of Guest, Fachantidis and Wels teaches The system of claim 9, wherein the first expected return is computed based on the incurred rewards and intervention costs associated with the interventions over the plurality of time steps. (Guest, paragraph 0103-0104 and 0139-0145, teaches the use of an intervention engine that integrates the use of a loss function to determine the costs of the intervention and an intervention effectiveness model that determines how effective an intervention was in getting the desired outcome this represents a reward in that a reward is a score that determines how effective or good an action was.)
Regarding claim 11 The combination of Guest, Fachantidis and Wels teaches The system of claim 9, further comprising: updating the agent policy by maximizing a second expected return computed based on incurred rewards including the incurred reward at the second time step, prior to the updating of the parameters of the neural network based intervention model. (Guest, paragraph 0095-0104 and 0139-0145, teaches the system of the intervention model and how it is trained over multiple iterations at different points in time (i.e. second time step) the model will learn what interventions will work best at each different point in time in order to achieve a desired result. Further it teaches the use of an intervention engine that integrates the use of a loss function to determine the costs of the intervention and an intervention effectiveness model that determines how effective an intervention was in getting the desired outcome this represents a reward in that a reward is a score that determines how effective or good an action was.)
Regarding claim 12 The combination of Guest, Fachantidis and Wels teaches The system of claim 9, wherein the second agent action is determined by sampling the second agent action according to the agent policy at the second time step. (Guest, paragraph 0093, 0127-0130, teaches the use of a user and user information that allows the system to determine what behaviors and actions are taking place before and after an intervention has been implemented, using this history of behaviors and actions the system is able to predict a user’s behavior after an intervention and determine what that second action will be based on that prior data (i.e. agent policy).)
Regarding claim 13 The combination of Guest, Fachantidis and Wels teaches The system of claim 9, wherein the generating of the second intervention at the second time step includes: generating, by the neural network based intervention model, a distribution over interventions; and sampling the second intervention according to the generated distribution. (Guest, paragraph 0102-0104, teaches the use of an intervention engine (i.e. neural network intervention model) that takes in a distribution of the likelihood of the behaviors of a user based on the possible interventions and allows for the intervention engine to determine the best intervention to sample based on the list of interventions.)
Regarding claim 14 The combination of Guest, Fachantidis and Wels teaches The system of claim 9, wherein the updating of the parameters of the neural network based intervention model is performed at an end of the plurality of time steps including the first time step and the second time step. (Guest, paragraph 0104, 0117-0119, FIG 5, teaches the adjusting of the intervention engine (i.e. intervention model) being performed after a plurality of time steps being performed.)
Regarding claim 17 Guest teaches A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising: obtaining a first agent action …1 at a first time step and a first intervention that is generated according to an intervention policy at the first time step; (Guest, paragraph 0127, teaches the use of a user’s (i.e. agent) actions at a time step and an intervention model that uses a policy to determine when to intervene in the agents actions. ) generating, by a recurrent neural network based agent model implemented on one or more processors and configured to simulate the neural network based agent, a predicted agent action at a second time step conditioned on the first agent action, and the first intervention at the first time step; (Guest, paragraph 0099-0101, teaches a likelihood engine that is used to predict the next action that a user (i.e. agent) will take based on the previous action that the user had taken. The likelihood engine is a neural network.) generating, by a recurrent neural network based intervention model implemented on the one or more processors and configured to simulate interventions against the neural network based agent,, a second intervention at the second time step according to the intervention policy and conditioned on the first agent action, the first intervention, and the predicted agent action; (Guest, paragraph 0102-0103, teaches the use of an intervention engine that gets the predicted action from the likelihood engine and determines if an intervention is needed and what intervention is needed at the time based on its policy. Guest, paragraph 0056, teaches the intervention model being a neural network.) …2 that is based on the second intervention at the second time step; and (Guest, paragraph 0103, teaches an intervention effectiveness engine that after the execution of an intervention and the action that is taken by the user (i.e. agent) with that intervention it then determines how effective that intervention was in getting the desired outcome this represents a reward in that a reward is a score that determines how effective or good an action was.) training the neural network based intervention model by updating parameters of the neural network based intervention model based on a first expected return computed based on incurred rewards over a plurality of time steps …3 (Guest, paragraph 104 and 0139-145, teaches the training and adjusting of the intervention model based on the outcome of the intervention based on the action taken by the user in response to the intervention and the intervention effectiveness over time.)
Guest does not teach …1 performed by a neural network based agent of the plurality of neural network based agents… However, Fachantidis in analogous art teaches this limitation (Fachantids. Page 25, section 3.1 teaches a student model neural network agent that performs tasks in an environment and is capable of receiving outside advice (i.e. interventions) from another agent.)
Further Guest does not teach …2 executing a second agent action according to an agent policy at the second time step that incurs a reward … However, Fachantidis in analogous art teaches this limitation (Fachantidis, page 35, Section 5.3, and algorithm 1, teaches the agent taking a second action after receiving advice (i.e. intervention) from the teacher model and incurring a reward based on the action and the advice that was given to the student model.)
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine Fachantidis’ teaching of a neural network agent that is able to receive and act on interventions with Guest’s teaching of a meta learning system. The motivation to do so would be to be able to generate more examples for the intervention model to train on and to have the intervention model be able to interact with machine learning based agents.
The combination of Guest and Fachantidis does not teach …3 performing a mechanism design based on the agent-intervention simulation; and displaying, to a user, the mechanism design on a user interface (UI). However, Wels in analogous art teaches this limitation (Wels, paragraph 0029, teaches the Machine learning displaying results and the model structure to a user interface in.)
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine Weis’ teaching of a user interface the displays a machine learning model and its results with the combination of Guest and Fachantidis teaching of a meta learning system. The motivation to do so would be to allow users to visualize the neural network structure and the results that it produces.
Regarding claim 18 The combination of Guest, Fachantidis and Wels teaches The non-transitory machine-readable medium of claim 17, wherein the first expected return is computed based on the incurred rewards and intervention costs associated with the interventions over the plurality of time steps. (Guest, paragraph 0103-0104 and 0139-0145, teaches the use of an intervention engine that integrates the use of a loss function to determine the costs of the intervention and an intervention effectiveness model that determines how effective an intervention was in getting the desired outcome this represents a reward in that a reward is a score that determines how effective or good an action was.)
Regarding claim 19 The combination of Guest, Fachantidis and Wels teaches The non-transitory machine-readable medium of claim 17, further comprising: updating the agent policy by maximizing a second expected return computed based on incurred rewards including the incurred reward at the second time step, prior to the updating of the parameters of the neural network based intervention model. (Guest, paragraph 0095-0104 and 0139-0145, teaches the system of the intervention model and how it is trained over multiple iterations at different points in time (i.e. second time step) the model will learn what interventions will work best at each different point in time in order to achieve a desired result. Further it teaches the use of an intervention engine that integrates the use of a loss function to determine the costs of the intervention and an intervention effectiveness model that determines how effective an intervention was in getting the desired outcome this represents a reward in that a reward is a score that determines how effective or good an action was.)
Claims 7-8, 15-16 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Guest et al. Pub. No.: US 20240424245 A1 in view of Fachantidis et al. “Learning to Teach Reinforcement Learning Agents” 12/06/2017 in further view of Oliva et al. “Graph Neural Networks for Relational Inductive Bias in Vision-based Deep Reinforcement Learning of Robot Control” 03/11/2022.
Regarding claim 7 The combination of Guest, Fachantidis and Wels teaches The method of claim 1, further comprising, after training the neural network based intervention model: generating, by the neural network based agent model, a second predicted agent action at a fourth time step conditioned on a third agent action at a third time step, and a third intervention at the third time step, the third time step being after the plurality of time steps; (Guest, paragraph 0099-0101, teaches a likelihood engine that is used to predict the next action that a user (i.e. agent) will take based on the previous action that the user had taken. The likelihood engine is a neural network. It further teaches the ability for multiple iterations to generate a plurality of predictions at a plurality of time steps.)generating, by a neural network based intervention model, a fourth intervention at the fourth time step according to the intervention policy after the plurality of time steps and conditioned on the third agent action, the third intervention, and the second predicted agent action; (Guest, paragraph 0102-0103, teaches the use of an intervention engine that gets the predicted action from the likelihood engine and determines if an intervention is needed and what intervention is needed at the time based on its policy. Guest, paragraph 0056, teaches the intervention model being a neural network. It further teaches the ability for multiple iterations to generate a plurality of interventions based on a plurality of predicted actions at a plurality of time steps.)executing a fourth agent action at the fourth time step that incurs a reward that is based on the fourth intervention at the fourth time step; (Guest, paragraph 0103, teaches an intervention effectiveness engine that after the execution of an intervention and the action that is taken by the user (i.e. agent) with that intervention it then determines how effective that intervention was in getting the desired outcome this represents a reward in that a reward is a score that determines how effective or good an action was. It further teaches the ability for multiple iterations to generate a plurality of interventions based on a plurality of predicted actions at a plurality of time steps.)…1including the fourth agent action, the fourth intervention, and an intervention distribution after the plurality of time steps; and (Guest, paragraph 0102-0104, teaches the use of an intervention engine (i.e. neural network intervention model) that takes in a distribution of the likelihood of the behaviors of a user that is determined using information about the user actions and the interventions used. Further it teaches the ability for a plurality of agent actions to take place.) training the neural network based intervention model by updating parameters of the neural network based intervention model …2 over a second plurality of time steps. (Guest, paragraph 104 and 0139-145, teaches the training and adjusting of the intervention model based on the outcome of the intervention based on the action taken by the user in response to the intervention and the intervention effectiveness over time.)
Guest does not teach …1 collecting a rollout… …2based on collected rollouts… However, Oliva teaches this limitation in analogous art (Oliva, page 5-6, section III-D, teaches the use of rollouts as a means of collecting and storing data in relation to training a neural network.)
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine Oliva’s teaching of collecting rollouts of data for training neural networks with Guest’s teaching of a neural network intervention model. The motivation to do so would be to collect the intentions and actions used over a plurality of timesteps and maintaining information about the steps that were taken to get to the result.
Regarding claim 8 The combination of Guest, Fachantidis, Wels and Oliva teaches The method of claim 7, further comprising training the neural network based agent model by maximizing a log-likelihood of expected agent actions over the first or second plurality of time steps. (Guest, paragraph 0143-144, teaches using cross entropy loss using user (i.e. agent) actions at different times (i.e. negative log likelihood) which is minimized meaning that the log-likelihood is maximized.)
Regarding claim 15 The combination of Guest, Fachantidis and Wels teaches The system of claim 9, wherein the operations further comprise, after training the neural network based intervention model: generating, by the neural network based agent model, a second predicted agent action at a fourth time step conditioned on a third agent action at a third time step, and a third intervention at the third time step, the third time step being after the plurality of time steps; (Guest, paragraph 0099-0101, teaches a likelihood engine that is used to predict the next action that a user (i.e. agent) will take based on the previous action that the user had taken. The likelihood engine is a neural network. It further teaches the ability for multiple iterations to generate a plurality of predictions at a plurality of time steps.)generating, by a neural network based intervention model, a fourth intervention at the fourth time step according to the intervention policy after the plurality of time steps and conditioned on the third agent action, the third intervention, and the second predicted agent action; (Guest, paragraph 0102-0103, teaches the use of an intervention engine that gets the predicted action from the likelihood engine and determines if an intervention is needed and what intervention is needed at the time based on its policy. Guest, paragraph 0056, teaches the intervention model being a neural network. It further teaches the ability for multiple iterations to generate a plurality of interventions based on a plurality of predicted actions at a plurality of time steps.)executing a fourth agent action at the fourth time step that incurs a reward that is based on the fourth intervention at the fourth time step; (Guest, paragraph 0103, teaches an intervention effectiveness engine that after the execution of an intervention and the action that is taken by the user (i.e. agent) with that intervention it then determines how effective that intervention was in getting the desired outcome this represents a reward in that a reward is a score that determines how effective or good an action was. It further teaches the ability for multiple iterations to generate a plurality of interventions based on a plurality of predicted actions at a plurality of time steps.)…1including the fourth agent action, the fourth intervention, and an intervention distribution after the plurality of time steps; and (Guest, paragraph 0102-0104, teaches the use of an intervention engine (i.e. neural network intervention model) that takes in a distribution of the likelihood of the behaviors of a user that is determined using information about the user actions and the interventions used. Further it teaches the ability for a plurality of agent actions to take place.)training the neural network based intervention model by updating parameters of the neural network based intervention model …2 over a second plurality of time steps. (Guest, paragraph 104 and 0139-145, teaches the training and adjusting of the intervention model based on the outcome of the intervention based on the action taken by the user in response to the intervention and the intervention effectiveness over time.)
Guest does not teach …1 collecting a rollout… …2based on collected rollouts… However, Oliva teaches this limitation in analogous art (Oliva, page 5-6, section III-D, teaches the use of rollouts as a means of collecting and storing data in relation to training a neural network.)
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine Oliva’s teaching of collecting rollouts of data for training neural networks with Guest’s teaching of a neural network intervention model. The motivation to do so would be to collect the intentions and actions used over a plurality of timesteps and maintaining information about the steps that were taken to get to the result.
Regarding claim 16 The combination of Guest, Fachantidis, Wels, and Oliva teaches The method of claim 15, further comprising training the neural network based agent model by maximizing a log-likelihood of expected agent actions over the first or second plurality of time steps. (Guest, paragraph 0143-144, teaches using cross entropy loss using user (i.e. agent) actions at different times (i.e. negative log likelihood) which is minimized meaning that the log-likelihood is maximized.)
Regarding claim 20 The combination of Guest, Fachantidis and Wels teaches The non-transitory machine-readable medium of claim 17, further comprising, after training the neural network based intervention model: generating, by the neural network based agent model, a second predicted agent action at a fourth time step conditioned on a third agent action at a third time step, and a third intervention at the third time step, the third time step being after the plurality of time steps; (Guest, paragraph 0099-0101, teaches a likelihood engine that is used to predict the next action that a user (i.e. agent) will take based on the previous action that the user had taken. The likelihood engine is a neural network. It further teaches the ability for multiple iterations to generate a plurality of predictions at a plurality of time steps.)generating, by a neural network based intervention model, a fourth intervention at the fourth time step according to the intervention policy after the plurality of time steps and conditioned on the third agent action, the third intervention, and the second predicted agent action; (Guest, paragraph 0102-0103, teaches the use of an intervention engine that gets the predicted action from the likelihood engine and determines if an intervention is needed and what intervention is needed at the time based on its policy. Guest, paragraph 0056, teaches the intervention model being a neural network. It further teaches the ability for multiple iterations to generate a plurality of interventions based on a plurality of predicted actions at a plurality of time steps.)executing a fourth agent action at the fourth time step that incurs a reward that is based on the fourth intervention at the fourth time step; (Guest, paragraph 0103, teaches an intervention effectiveness engine that after the execution of an intervention and the action that is taken by the user (i.e. agent) with that intervention it then determines how effective that intervention was in getting the desired outcome this represents a reward in that a reward is a score that determines how effective or good an action was. It further teaches the ability for multiple iterations to generate a plurality of interventions based on a plurality of predicted actions at a plurality of time steps.)…1including the fourth agent action, the fourth intervention, and an intervention distribution after the plurality of time steps; and (Guest, paragraph 0102-0104, teaches the use of an intervention engine (i.e. neural network intervention model) that takes in a distribution of the likelihood of the behaviors of a user that is determined using information about the user actions and the interventions used. Further it teaches the ability for a plurality of agent actions to take place.) training the neural network based intervention model by updating parameters of the neural network based intervention model …2 over a second plurality of time steps. (Guest, paragraph 104 and 0139-145, teaches the training and adjusting of the intervention model based on the outcome of the intervention based on the action taken by the user in response to the intervention and the intervention effectiveness over time.)
Guest does not teach …1 collecting a rollout… …2based on collected rollouts… However, Oliva teaches this limitation in analogous art (Oliva, page 5-6, section III-D, teaches the use of rollouts as a means of collecting and storing data in relation to training a neural network.)
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine Oliva’s teaching of collecting rollouts of data for training neural networks with Guest’s teaching of a neural network intervention model. The motivation to do so would be to collect the intentions and actions used over a plurality of timesteps and maintaining information about the steps that were taken to get to the result.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THOMAS B LANE whose telephone number is (571)272-1872. The examiner can normally be reached M-Th: 7:20am-5:20pm; F: Out of Office.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MARIELA REYES can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THOMAS BERNARD LANE/Examiner, Art Unit 2142
/HAIMEI JIANG/Primary Examiner, Art Unit 2142
Prosecution Insights