DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed 3/3/2026 has been entered. Claims 1-4, 10-11, 13, and 18-20 have been amended. Claims 5-6, 12, and 14-15 have been canceled. Claims 1-4, 7-11, 13, and 16-20 are pending in the application.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 17 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends.
Dependent claim 17 recites "The method of claim 10, wherein the optimal action is an action expected to result in a maximum future reward." Independent claim 10, as amended in the response filed March 3, 2026, already recites in its body: "…training a model, by the computing system and using the selected instances of experience data, to predict an optimal action to take in a reinforcement learning model, wherein the optimal action is an action expected to result in a maximum future reward; and…" Because the wherein clause of claim 17 is identical to a limitation already present in claim 10, claim 17 fails to specify a further limitation of the subject matter claimed, as required by 35 U.S.C. 112(d).
Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-4, 7-11, 13, and 16-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50 (“2019 PEG”).
Claim 1
Step 1: The claim recites “A computing system comprising”; therefore, it is directed to the statutory category of machines.
Step 2A Prong 1: The claim recites, inter alia:
generate a plurality of instances of episode data, each comprising a sequence of instances of experience data: These limitations recite a mentally performable process with aid of pen and paper of using observation and judgement to generate a plurality of instances of episode data, each comprising a sequence of instances of observed experience data.
store each of the plurality of instances of episode data in a buffer: These limitations recite a mentally performable process with aid of pen and paper of using observation and judgement to annotate each of the plurality of instances of episode data in a designated buffer region of the paper.
compile statistics associated with each of the plurality of instances of episode data: These limitations recite a mentally performable process with aid of pen and paper of using judgement to annotate statistics associated with each of the generated plurality of instances of episode data.
identify episode data associated with an episode in which a reward was achieved: These limitations recite a mentally performable process, with the aid of pen and paper, of using observation and judgement to identify, among the annotated episode data, episode data associated with an episode in which a reward was observed to be achieved.
select, based on the statistics and by prioritizing instances from the episode in which the reward was achieved, a subset of instances of the experience data from the buffer: These limitations recite a mentally performable process, with the aid of pen and paper, of using judgement to selectively annotate a subset grouping of instances of the experience data from the designated buffer region of the paper, based on the annotated statistics and by giving priority to instances from the episode in which the reward was observed to be achieved.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
A computing system comprising a storage device and processing circuitry having access to the storage device, wherein the processing circuitry is configured to: These additional limitations are recited at a high level of generality and merely represent generic computer machinery performing in their ordinary capacity to implement the underlying judicial exception. See MPEP 2106.05(f).
train a model, using the subset of instances of experience data, to predict an optimal action to take in a reinforcement learning model: These additional elements recite only the idea of training a model using the subset of instances of experience data to predict an optimal action to take in a reinforcement learning model without details on how this is accomplished. The claim omits any details of what type of model is trained, e.g. a CNN, a RNN, etc., and there is no details as to whether/how the trained model is related/couple to/the same as a reinforcement learning model taking action. No details or particularities defined for how training is accomplished using the subset of instances of experience data, e.g. is the particular subset used for validation and/or is the training method supervised/unsupervised/hybrid/etc. as “a model” could be exclusive from “a reinforcement learning model” but for predicting an optimal action. Therefore, these limitations represent no more than mere instructions to implement the abstract idea and is equivalent to adding the words “apply it” to the recited judicial exception. See MPEP 2106.05(f).
and control another system, by applying the trained model, to cause the other system to perform an operation: These additional elements recite, at a high level of generality, applying the result of the abstract idea (the trained model) to control an unspecified “other system” to perform an unspecified “operation.” No particular system, technology, or operation is recited. The limitation merely links the abstract idea to a general field of use and/or appends the words “apply it”. See MPEP 2106.05(f), (h).
Thus, the way in which the additional elements use or interact with the judicial exception when analyzed with the claim as a whole do not integrate the judicial exception into a practical application.
Improvement consideration (MPEP 2106.05(a); Ex Parte Desjardins): The specification asserts a benefit; that selecting experience data based on episode-level and/or epoch-level attributes “may be more effective at improving the predictive skill” of the agent and may “more quickly and efficiently improve the skill … in predicting an optimal action,” with reduced reward variance. However, a claim integrates an exception via a technological improvement only where the claim itself recites the components or steps that provide the improvement described in the specification; the improvement may not be imported from the specification into a claim that does not recite it. Here, the claim recites only a generic selection criterion (“by prioritizing instances from the episode in which the reward was achieved”) and the result (“train a model … to predict an optimal action”). The specific mechanism the specification credits with the improvement; the use of episode-level and epoch-level attributes to select experience data, and the attendant variance reduction; is not recited in the claim. Moreover, the prioritized-selection step is itself the identified abstract idea and is therefore subsumed in the judicial exception, rather than an improvement to how the computer or model functions in operation. Accordingly, the claim does not reflect a technological improvement, and the exception is not integrated into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include invoking computers or other machinery to apply the underlying judicial exception and generically linking the judicial exception to a general field of use/technological environment/a recitation of the words “apply it” (or an equivalent). Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP 2106.05.
Claim 2
Step 1: a machine, as in claim 1.
Step 2A Prong 1: The claim recites, inter alia:
wherein to compile statistics associated with each of the plurality of instances of episode data, compile statistics about a reward achieved during each episode: These limitations recite furthering the mentally performable process with aid of pen and paper of using judgement to annotate statistics associated with each of the generated plurality of instances of episode data by further annotating statistics about an observed reward achieved during each observed episode.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
the processing circuitry is further configured to: These additional limitations are recited at a high level of generality and merely represent generic computer machinery performing in their ordinary capacity to implement the underlying judicial exception. See MPEP 2106.05(f).
Thus, the way in which the additional elements use or interact with the judicial exception when analyzed with the claim as a whole do not integrate the judicial exception into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include invoking computers or other machinery to apply the underlying judicial exception. Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP 2106.05.
Claim 3
Step 1: a machine, as in claim 2.
Step 2A Prong 1: The claim recites, inter alia:
wherein the episode is a first episode and the reward is a first reward, and wherein to select the subset of instances of the experience data: identify episode data associated with a second episode in which a second reward was achieved: These limitations recite furthering the mentally performable process of using observation and judgement to identify episode data associated with a second observed episode in which a second reward was observed to be achieved to select the subset of instances of the experience data observed.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
the processing circuitry is further configured to: These additional limitations are recited at a high level of generality and merely represent generic computer machinery performing in their ordinary capacity to implement the underlying judicial exception. See MPEP 2106.05(f).
Thus, the way in which the additional elements use or interact with the judicial exception when analyzed with the claim as a whole do not integrate the judicial exception into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include invoking computers or other machinery to apply the underlying judicial exception. Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP 2106.05.
Claim 4
Step 1: a machine, as in claim 3.
Step 2A Prong 1: The claim recites, inter alia:
select, by prioritizing instances from the second episode in which the second reward was achieved, additional instances of experience data: These limitations recite a mentally performable process of using judgement and observation to select, by prioritizing instances from the second episode in which the second reward was achieved, additional instances of experience data.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
wherein to train the model, the processing circuitry is further configured to: train the model further based on the additional instances of experience data: These additional elements recite only the idea of training a model further based on the additional instances of experience data processed by the judicial exception with generically recited processing circuitry without reflecting specific mechanisms the specification credits with the improvement; the use of episode-level and epoch-level attributes to select experience data, and the attendant variance reduction. Therefore, these limitations represent no more than mere instructions to implement the abstract idea and is equivalent to adding the words “apply it” to the recited judicial exception. See MPEP 2106.05(f) and Ex Parte Desjardins.
Thus, the way in which the additional elements use or interact with the judicial exception when analyzed with the claim as a whole do not integrate the judicial exception into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include a recitation of the words “apply it” (or an equivalent) and generally linking the use of a judicial exception to a particular technological environment or field of use. Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP 2106.05.
Claim 7
Step 1: a machine, as in claim 1.
Step 2A Prong 1: The claim recites, inter alia:
wherein to compile statistics associated with each of the plurality of instances of episode data: These limitations recite the mentally performable process with aid of pen and paper of using judgement to annotate statistics associated with each of the generated plurality of instances of episode data.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
the processing circuitry is further configured to: These additional limitations are recited at a high level of generality and merely represent generic computer machinery performing in their ordinary capacity to implement the underlying judicial exception. See MPEP 2106.05(f).
compile statistics about at least one of: a reward achieved during an episode; a timeframe associated with an episode; a step count associated with an episode; and error information associated with an episode: These additional elements are recited at a high level of generality and amount to generally linking the use of the judicial exception, e.g. compile statistics associated with each of the plurality of instances of episode data, to a particular technological environment or field of use. Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself and cannot integrate a judicial exception into a practical application. See MPEP 2106.05(h).
Thus, the way in which the additional elements use or interact with the judicial exception when analyzed with the claim as a whole do not integrate the judicial exception into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include invoking computers or other machinery to apply the underlying judicial exception and generally linking the use of the judicial exception to a particular field of use or technological environment. Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP 2106.05.
Claim 8
Step 1: a machine, as in claim 1.
Step 2A Prong 1: The claim recites the same abstract ideas as claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
wherein the optimal action is an action expected to result in a maximum future reward: These additional elements recite only furthers the idea of training a model using the subset of instances of experience data to predict an optimal action to take in a reinforcement learning model, as in claim 1, wherein the optimal action is an action expected to result in a maximum future reward, without details on how this is accomplished. The claim omits any details of what type of model is trained, e.g. a CNN, a RNN, etc., and there is no details as to whether/how the trained model is related/couple to/the same as a reinforcement learning model taking action. No details or particularities defined for how training is accomplished using the subset of instances of experience data, e.g. is the particular subset used for validation and/or is the training method supervised/unsupervised/hybrid/etc. as “a model” could be exclusive from “a reinforcement learning model” but for predicting an optimal action. Therefore, these limitations represent no more than mere instructions to implement the abstract idea and is equivalent to adding the words “apply it” to the recited judicial exception. See MPEP 2106.05(f).
Thus, the way in which the additional elements use or interact with the judicial exception when analyzed with the claim as a whole do not integrate the judicial exception into a practical application. Step 2B: The additional elements from Step 2A Prong 2 include a recitation of the words “apply it” (or an equivalent). Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP 2106.05.
Claim 9
Step 1: a machine, as in claim 1.
Step 2A Prong 1: The claim recites the same abstract ideas as claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
wherein to train the model, the processing circuitry is further configured to: train a neural network: These additional elements recite only the idea of training a model that is a neural network with generically recited processing circuitry without details on how this is accomplished. The claim omits any details of what type of neural network model is trained, e.g. a CNN, a RNN, etc. No details or particularities defined for how training is accomplished, e.g. training using supervised/unsupervised/hybrid/etc. methods. Therefore, these limitations represent no more than mere instructions to implement the abstract idea and is equivalent to adding the words “apply it” to the recited judicial exception. See MPEP 2106.05(f).
Thus, the way in which the additional elements use or interact with the judicial exception when analyzed with the claim as a whole do not integrate the judicial exception into a practical application. Step 2B: The additional elements from Step 2A Prong 2 include a recitation of the words “apply it” (or an equivalent). Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP 2106.05.
Claims 10-11 and 16-18
Step 1: These claims are directed to “A method comprising:”; therefore, these claims are directed to the statutory category of a process.
Step 2A Prong 1: These claims recite the same abstract ideas as in claims 1-2 and 7-9, respectively.
Step 2A Prong 2: The judicial exceptions recited in these claims are not integrated into a practical application. The analysis at this step mirrors that of claims 1-2 and 7-9, respectively.
Step 2B: These claims do not contain significantly more than the judicial exception. The analysis at this step mirrors that of claims 1-2 and 7-9, respectively.
Claim 13
Step 1: a process, as in claim 10.
Step 2A Prong 1: The claim recites, inter alia:
wherein the episode is a first episode and the reward is a first reward, and selecting, by prioritizing instances from a second episode in which a second reward was achieved, additional instances of experience data: These limitations recite a mentally performable process of using observation to observe wherein the episode is a first episode and the reward is a first reward and followed by judgement to select, by prioritizing instances from a second episode in which a second reward was achieved, additional instances of experience data.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
wherein training the model includes: training the model further based on the additional instances of experience data: These additional elements recite only the idea of training a model further based on the additional instances of experience data provided by the judicial exception without reflecting specific mechanisms the specification credits with the improvement; the use of episode-level and epoch-level attributes to select experience data, and the attendant variance reduction. Therefore, these limitations represent no more than mere instructions to implement the abstract idea and is equivalent to adding the words “apply it” to the recited judicial exception. See MPEP 2106.05(f) and Ex Parte Desjardins.
Thus, the way in which the additional elements use or interact with the judicial exception when analyzed with the claim as a whole do not integrate the judicial exception into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include a recitation of the words “apply it” (or an equivalent). Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP 2106.05.
Claims 19-20
Step 1: These claims are directed to “A non-transitory computer-readable medium comprising instructions that, when executed, configure processing circuitry of a computing system to perform operations comprising:”; therefore, these claims are directed to the statutory category of an article of manufacture.
Step 2A Prong 1: Claim 19 recites the same abstract ideas as in claim 1 and claim 8. Claim 20 recites the same abstract ideas as in claims 2..
Step 2A Prong 2: The judicial exceptions recited in these claims are not integrated into a practical application. The only substantive difference between claims 19-20 and claims 1-2 and 8 is that claims 19-20 are directed to “A non-transitory computer-readable medium comprising instructions that, when executed, configure processing circuitry of a computing system to perform operations comprising:”. However, mere recitation that a judicial exception is to be performed using generic computer equipment in their ordinary capacity, i.e. a non-transitory computer-readable medium comprising instructions that, when executed, configure processing circuitry of a computing system to perform operations, cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). With that exception, the analysis at this step are substantially the same as that of claims 1-2 and 8.
Step 2B: These claims do not contain significantly more than the judicial exception. The only substantive difference between claims 19-20 and claims 1-2 and 8 is that claims 19-20 are directed to “A non-transitory computer-readable medium comprising instructions that, when executed, configure processing circuitry of a computing system to perform operations comprising:”. However, mere recitation that a judicial exception is to be performed using generic computer equipment in their ordinary capacity, a non-transitory computer-readable medium comprising instructions that, when executed, configure processing circuitry of a computing system to perform operations, cannot amount to significantly more than the judicial exception. See MPEP 2106.05(f). With that exception, the analysis at this step are substantially the same as that of claims 1-2 and 8.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-11, 13, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Oh et al. (hereinafter Oh) "Self-Imitation Learning" (2018), in view of Levine et al. (hereinafter Levine), US 2019/0232488 A1.
Regarding independent claim 1, Oh teaches A computing system comprising a storage device and processing circuitry having access to the storage device, wherein the processing circuitry is configured to (page 3878, Section 1, the self-imitation-learning algorithm stores experiences in a replay buffer and an episode buffer (a storage device) and runs an agent that executes the algorithm (processing circuitry having access to the storage device)):
generate a plurality of instances of episode data, each comprising a sequence of instances of experience data (page 3880, Algorithm 1, at each step the agent executes a transition (an instance of experience data) that is stored into the episode buffer, and each terminated episode is the resulting sequence of such transitions (episode data), with repeated iterations producing a plurality of episodes);
store each of the plurality of instances of episode data in a buffer (page 3880, Algorithm 1, upon episode termination the transitions and their computed returns are written to the replay buffer (a buffer));
compile statistics associated with each of the plurality of instances of episode data (page 3880, Algorithm 1, on each episode's termination Oh computes the discounted episode return for every step in the episode (statistics), a statistic compiled per episode and attached to each stored instance);
identify episode data associated with an episode in which a reward was achieved (page 3878, Section 1, Oh learns to imitate state-action pairs in the replay buffer only when the return in the past episode is greater than the agent's value estimate, thereby identifying the experience data belonging to past episodes whose accumulated reward, the return, was high (an episode in which a reward was achieved));
select, based on the statistics and by prioritizing instances from the episode in which the reward was achieved, a subset of instances of the experience data from the buffer (page 3879, Related Work and page 3880, Section 3, Oh samples a mini-batch of transitions (a subset of instances of the experience data) from the replay buffer (the buffer) with probability proportional to the clipped advantage derived from the full-episode return (the statistics), so the sampling is based on the per-episode statistic and prioritizes transitions drawn from high-return episodes (instances from the episode in which the reward was achieved));
train a model, using the subset of instances of experience data, to predict an optimal action to take in a reinforcement learning model (page 3881, Section 4.3 to page 3882, Oh updates the policy and value parameters (a model) on the sampled subset via the self-imitation-learning gradient and updates the policy and the value directly towards the optimal policy and the optimal value, so that the trained model predicts the optimal action to take (an optimal action to take in a reinforcement learning model)).
Oh does not expressly teach control another system, by applying the trained model, to cause the other system to perform an operation.
However, Levine teaches control another system, by applying the trained model, to cause the other system to perform an operation ([0005], a robot (another system) utilizes the trained policy neural network (the trained model) by applying a current state at each control cycle and implementing control commands to effectuate the indicated action (to cause the other system to perform an operation); [0058], the action implemented on the robot is the action output by the policy network for the current state).
Because Oh and Levine are analogous art and within the same field of endeavor, specifically training reinforcement-learning agents by learning from stored experiences, they address the same problem solving area of training a policy that selects optimal actions to maximize reward and putting that trained policy to use, accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine Levine's deployment of a trained reinforcement-learning policy to control a robot with Oh's self-imitation-learning training pipeline, with a reasonable expectation of success, such that the policy trained by Oh on episode-return-prioritized experience data is applied at each control cycle to control a separate system and cause it to perform an operation, to teach control another system, by applying the trained model, to cause the other system to perform an operation. This modification would have been motivated by the desire to put the trained policy to its intended practical use of selecting and executing actions on a controlled system, as Levine teaches that a trained policy network is utilized by a robot to implement control commands that effectuate actions (Levine: [0005]).
Regarding dependent claim 2, Oh, in view of Levine, teach the computing system of claim 1, wherein to compile statistics associated with each of the plurality of instances of episode data, the processing circuitry is further configured to: compile statistics about a reward achieved during each episode (see Oh, page 3880, Algorithm 1, the statistic compiled per episode is the episode return, the discounted sum of the rewards achieved during that episode (statistics about a reward achieved during each episode)).
Regarding dependent claim 3, Oh, in view of Levine, teach the computing system of claim 2, wherein the episode is a first episode and the reward is a first reward, and wherein to select the subset of instances of the experience data, the processing circuitry is further configured to: identify episode data associated with a second episode in which a second reward was achieved (see Oh, page 3880, Algorithm 1, the replay buffer accumulates transitions and returns from multiple successive episodes, so the system identifies high-return transitions across more than one past episode, including a second episode having an achieved second return (a second episode in which a second reward was achieved))).
Regarding dependent claim 4, Oh, in view of Levine, teach the computing system of claim 3, wherein to train the model, the processing circuitry is further configured to: select, by prioritizing instances from the second episode in which the second reward was achieved, additional instances of experience data (see Oh, page 3880, Algorithm 1, across the self-imitation updates the system repeatedly samples additional prioritized mini-batches, including transitions from later episodes whose return is high (additional instances of experience data)), and train the model further based on the additional instances of experience data (see Oh, page 3881, Section 4.3, the policy and value parameters continue to be updated on those additional sampled instances (train the model further based on the additional instances of experience data))).
Regarding dependent claim 7, Oh, in view of Levine, teach the computing system of claim 1, wherein to compile statistics associated with each of the plurality of instances of episode data, the processing circuitry is further configured to compile statistics about at least one of: a reward achieved during an episode; a timeframe associated with an episode; a step count associated with an episode; and error information associated with an episode (see Oh, page 3880, Algorithm 1, the recited alternative group requires that only one member be taught, and Oh compiles the reward achieved during an episode as the discounted episode return (a reward achieved during an episode), satisfying the first alternative)).
Regarding dependent claim 8, Oh, in view of Levine, teach the computing system of claim 1, wherein the optimal action is an action expected to result in a maximum future reward (see Oh, page 3880, Section 4.1 to page 3882, Section 4.3, the optimal policy maximizes the entropy-regularized discounted sum of future rewards, so the optimal action is the action expected to result in a maximum future reward (an action expected to result in a maximum future reward))).
Regarding dependent claim 9, Oh, in view of Levine, teach the computing system of claim 1, wherein to train the model, the processing circuitry is further configured to: train a neural network (see Oh, page 3882, Section 5.1, the model trained is a deep neural network, for Atari a three-layer convolutional neural network (a neural network))).
Regarding independent claim 10, Oh, in view of Levine, teach a method comprising the generating, storing, compiling-statistics, identifying, selecting, training, and controlling steps reciting the method corresponding of the operations of the system of claim 1, taught for the same reasons set forth for claim 1 with the same combination and the motivation to combine. In addition, Oh further teaches wherein the optimal action is an action expected to result in a maximum future reward (page 3880, Section 4.1 to page 3882, Section 4.3, the optimal policy maximizes the entropy-regularized discounted sum of future rewards, so the optimal action is the action expected to result in a maximum future reward (an action expected to result in a maximum future reward)).
Regarding dependent claims 11, 13, and 16-18, these are method claims that recite substantially the same subject matter as the system of claims 2, 3+4, and 7-9, respectively. Thus, claims 11, 13, and 16-18 are rejected for the same reasons as claims 2, 3+4, and 7-9.
Regarding independent claim 19, Oh, in view of Levine, teach a non-transitory computer-readable media comprising instructions that, when executed, configure processing circuitry of a computing system to perform operations comprising (page 3878, Section 1, the self-imitation-learning algorithm and its stored experiences are executed by a computing system (a computing system), necessarily embodied as instructions on non-transitory computer-readable media (non-transitory computer-readable media) that configure the agent's processing circuitry (processing circuitry)); the recited operations are the computer-readable-media claim of the operations of the system of claim 1, taught for the same reasons set forth for claim 1 with the same combination and the motivation to combine. In addition, Oh further teaches wherein the optimal action is an action expected to result in a maximum future reward (page 3880, Section 4.1 to page 3882, Section 4.3, the optimal policy maximizes the entropy-regularized discounted sum of future rewards, so the optimal action is the action expected to result in a maximum future reward (an action expected to result in a maximum future reward)).
Regarding dependent claim 20, this is a non-transitory computer-readable media claim that recites substantially the same subject matter as the system of claim 2. Thus, claim 20 is rejected for the same reason as claim 2.
Response to Arguments
Applicant’s claim amendments and Remarks filed 3/3/2026 with respect to the rejections under 35 U.S.C. 101 have been fully considered but they are not persuasive.
Applicant argues (Remarks, page 8) that amended claim 1 cannot be considered an abstract idea because it recites "specific interactions between computing systems, where one computing system controls another system," and, citing the August 4, 2025 memorandum on subject matter eligibility, that the claim recites inherently computer-based processes such as "train[ing] a model" and "control[ling] another system" that a human mind is not equipped to perform.
Examiner respectfully disagrees, the limitations identified as the abstract idea at Step 2A Prong One are the data-evaluation steps of generating a plurality of instances of episode data, storing each in a buffer, compiling statistics, identifying episode data associated with an episode in which a reward was achieved, and selecting, by prioritizing instances from that episode, a subset of instances of the experience data. As set forth in the rejection above, each of these steps is a process of observation and judgement that can be performed in the human mind or with the aid of pen and paper, and the amendment adding the episode-identification criterion to the selecting step does not change its character as such an evaluative judgement. The operations Applicant identifies as beyond the capacity of the mind, namely "train a model" and "control another system," are not relied upon as the abstract idea; they are additional elements evaluated at Step 2A Prong Two and Step 2B, where, as explained above, generic model training and a generic instruction to control an unspecified other system to perform an unspecified operation amount to no more than "apply it" and to linking the exception to a field of use. The "cannot practically be performed in the human mind" inquiry of the August 4, 2025 memorandum is therefore directed to limitations other than those identified as the mental process, and it does not undermine the Prong One finding. Claim 1 continues to recite a judicial exception.
Applicant argues (Remarks, page 2) that the recitation "control another system, by applying the trained model, to cause the other system to perform an operation" integrates the exception into a practical application, analogizing to Example 45 of the October 2019 Patent Eligibility Guidance Update, in which a single clause directing one system to control another to perform an operation was said to confer eligibility.
Examiner respectfully disagrees, the claim found eligible in Example 45 recited controlling a particular physical apparatus, an injection molding machine, to perform a particular physical operation, sending a control signal to open the mold and eject the molded part, upon a specific condition being reached; eligibility turned on that particular, real-world control of a particular machine. Amended claim 1 recites no such particularity. As set forth in the rejection above, the limitation applies the result of the abstract idea (the trained model) to control an unspecified "other system" to perform an unspecified "operation," and no particular system, technology, or operation is recited; the limitation therefore merely links the abstract idea to a general field of use and/or appends the words "apply it." See MPEP 2106.05(f), (h). As further explained in the improvement consideration above (MPEP 2106.05(a); Ex Parte Desjardins), the benefit asserted in the specification cannot integrate the exception, because the claim recites only a generic selection criterion ("by prioritizing instances from the episode in which the reward was achieved") and the result ("train a model ... to predict an optimal action"), not the episode-level and epoch-level mechanism, and attendant variance reduction, that the specification credits with the improvement. The exception is not integrated into a practical application.
Applicant argues (Remarks, page 3) that independent claims 10 and 19, and the dependent claims, recite patent-eligible subject matter for reasons analogous to claim 1.
Examiner respectfully disagrees, this argument is not persuasive for the reasons given above with respect to claim 1. Claims 10 and 19 recite the same data-evaluation abstract idea and the same generic training and control additional elements, analyzed above; the dependent claims add only further mental evaluation criteria, such as compiling statistics about a reward, timeframe, step count, or error, and identifying or prioritizing a second rewarded episode, or generic neural-network training, none of which integrates the exception or supplies an inventive concept, as set forth in the rejection. The rejection under 35 U.S.C. 101 is maintained.
Applicant’s claim amendments and Remarks filed 3/3/2026 with respect to the rejections under 35 U.S.C. 102 have been fully considered but they are moot as pending claims are now rejected under a new ground of rejection, necessitated by the claim amendments, under 35 U.S.C. 103 as unpatentable over Oh in view of Levine. To the extent Applicant argues (Remarks, page 3) that Oh provides no apparent reason for modification to include the amended features, the rejection now articulates a reason with rational underpinning to combine the references, as set forth above: it would have been obvious to apply Oh's trained self-imitation-learning policy to control a separate system, as Levine teaches a trained policy network being utilized by a robot to implement control commands that effectuate actions (Levine, [0005], [0058]), in order to put the trained policy to its intended practical use of selecting and executing actions on a controlled system. See KSR Int'l Co. v. Teleflex Inc., 550 U.S. 398 (2007); MPEP 2143.01.
Applicant argues (Remarks, pages 3-4) that Oh samples transitions "using the clipped advantage (R - V(s))+ as priority" and "prioritizes sampling individual transitions rather than looking at the episode as a whole," using the prioritized experience replay approach, such that Oh does not "select, based on the statistics and by prioritizing instances from the episode in which the reward was achieved, a subset of instances of the experience data from the buffer."
Examiner respectfully disagrees, as set forth in the rejection above, Oh's prioritization is keyed to a per-episode statistic rather than to a transition-local error. Oh learns to imitate state-action pairs in the replay buffer only when the return in the past episode is greater than the agent's value estimate (Oh, page 3878, Section 1), and Oh samples transitions from the replay buffer with probability proportional to the clipped advantage derived from the full-episode return (Oh, page 3879, Related Work, and page 3880, Section 3). Because the priority of each sampled transition is computed from the full-episode return R, prioritizing those transitions necessarily prioritizes the instances belonging to the episodes in which a reward, a high return, was achieved, which teaches the recited selecting by prioritizing instances from the episode in which the reward was achieved. Applicant's distinction between prioritizing "individual transitions" and looking at "the episode as a whole" is not commensurate with the claim language: the claim itself selects "a subset of instances of the experience data," that is, individual transitions, and requires only that the selection prioritize instances drawn from the rewarded episode, which Oh's full-episode-return-based priority does. That Oh employs prioritized experience replay as the sampling vehicle does not change that the priority value is the episode-return statistic. The argument therefore does not distinguish the claims from the rejection of record.
Applicant argues (Remarks, page 4) that independent claims 10 and 19, as amended, and the remaining dependent claims are likewise patentable over Oh.
Examiner respectfully disagrees, as set forth above, claims 10 and 19 recite the method and computer-readable-media counterparts of claim 1 and are taught by Oh in view of Levine for the same reasons and on the same citations set forth for claim 1; the dependent claims are addressed individually in the rejection above, and no limitation of any dependent claim is separately argued and claims 1-4, 7-11, 13, and 16-20 stand rejected under 35 U.S.C. 103 over Oh in view of Levine.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KUANG FU CHEN whose telephone number is (571)272-1393. The examiner can normally be reached M-F 9:00-5:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached on (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KC CHEN/Primary Patent Examiner, Art Unit 2143
Prosecution Insights