DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is in response to a Request of Continued Examination filed on August 7th, 2026. Claims 1-5, 8-12, 15-19, and 21-23 are pending in the current application, with claims 1, 4, 8, 11, 15, and 18 being currently amended, and claims 21-23 being newly added limitations.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on August 7th, 2026, has been entered.
Claim Rejections - 35 USC § 112
Claim 21-23 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “higher” in claims 21-23 is a relative term which renders the claim indefinite. The term “higher” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The use of the term renders the scope of an “uncertainty value” indefinite.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5, 8-12, 15-19, and 21-23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1, Under Step 1 of the Subject Matter Eligibility Test of Products and Processes, the claim is directed towards a process which is one of the four statutory categories.
Next, under a Step 2A Prong 1 Analysis, the claim recites the following limitations, which are interpreted to be, under the broadest reasonable interpretation, abstract ideas:
sampling states over a whole state space (mental process, observation)
determining when sampled states are out of distribution out of distribution being a state that is not within the dataset (D) (mental process, evaluation)
assigning probability weights to the sampled states that are out of distribution, the probability weights being greater than probability weights assigned to the in-distribution states (mental process, judgement)
Updating the Q-function to generate a fitted Q-function solving an optimization problem with a minimization term and a maximization term, wherein the minimization term comprises an expectation of Q-values over states sampled from a distribution
(
ⅆ
ϕ
)
that assigns probabilities to states outside the dataset (D) and over actions sampled from the policy, and penalizes an overall expected reward when a present state is out of distribution (mathematical concept)
the maximization term comprises an expectation of Q-values over states sampled from a marginal distribution
(
d
π
β
)
of states in the dataset (D) and over actions sampled from the policy (mathematical concept)
Therefore, we have to examine the claim under Step 2A prong 2, which considers the additional elements within the claim. The claim’s additional elements are:
performing offline reinforcement learning to iteratively update an agent that possesses a policy and a Q-function based on a dataset (D), the dataset (D) having in- distribution states
updating the policy according to an existing reinforcement learning algorithm.
controlling a robotic device using the policy
“performing offline reinforcement learning to iteratively update an agent that possesses a policy and a Q-function based on a dataset, the dataset (D) having in- distribution states”, and “updating the policy according to an existing reinforcement learning algorithm” merely indicates the field of use and technological environment in which to apply the judicial exception, “generally linking” it, i.e. the abstract idea generically performing offline RL learning to update an agent using a Q-function based on a dataset with in-distribution states and out-of-distribution states. (See MPEP 2106.05(h)) The limitation, “controlling a robotic device using the policy”, is considered to be mere instructions to apply a judicial exception, as it applies the judicial exception in no meaningful way, as it merely states that the abstract idea should be applied to achieve a desired result. (See MPEP2106.05(f)(3)) Therefore, these additional elements do not integrate the abstract idea into a practical application. The claim is directed to an abstract idea.
Under a Step 2B analysis, the claim’s addition elements do not amount to significantly
more than the judicial exception as explained above in Step 2A prong 2. Therefore, the claim is ineligible.
Regarding claim 8, Under Step 1 of the Subject Matter Eligibility Test of Products and Processes, the claim is directed towards a system which is considered a machine, which is one of the four statutory categories.
Next, under a Step 2A Prong 1 Analysis, the claim recites the following limitations, which are interpreted to be, under the broadest reasonable interpretation, abstract ideas:
sample states over a whole state space (mental process, observation)
determine when sampled states are out of distribution out of distribution being a state that is not within the dataset (D) (mental process, evaluation)
assign probability weights to the sampled states that are out of distribution, the probability weights being greater than probability weights assigned to the in-distribution states (mental process, judgement)
Update the Q-function to generate a fitted Q-function solving an optimization problem with a minimization term and a maximization term, wherein the minimization term comprises an expectation of Q-values over states sampled from a distribution
(
ⅆ
ϕ
)
that assigns probabilities to states outside the dataset (D) and over actions sampled from the policy, and penalizes an overall expected reward when a present state is out of distribution (mathematical concept)
the maximization term comprises an expectation of Q-values over states sampled from a marginal distribution
(
d
π
β
)
of states in the dataset (D) and over actions sampled from the policy (mathematical concept)
Therefore, we have to examine the claim under Step 2A prong 2, which considers the additional elements within the claim. The claim’s additional elements are:
A processor
A memory in communication with the processor and storing an offline learning module
performing offline reinforcement learning to iteratively update an agent that possesses a policy and a Q-function based on a dataset (D), the dataset (D) having in- distribution states
and update the policy according to an existing reinforcement learning algorithm.
controlling a robotic device using the policy
The processor, memory, “performing offline reinforcement learning to iteratively update an agent that possesses a policy and a Q-function based on a dataset (D), the dataset (D) having in- distribution states”, and to “update the policy according to an existing reinforcement learning algorithm“ are limitations that, as drafted, merely indicates the field of use and technological environment in which to apply the judicial exception, “generally linking” the limitations to the judicial exception. (See MPEP 2106.05(h)) The limitation, “control a robotic device using the policy”, is considered to be mere instructions to apply a judicial exception, as it applies the judicial exception in no meaningful way, as it merely states that the abstract idea should be applied to achieve a desired result. (See MPEP2106.05(f)(3)) Therefore, these additional elements do not integrate the abstract idea into a practical application. The claim is directed to an abstract idea.
Under a Step 2B analysis, the claim’s addition elements do not amount to significantly
more than the judicial exception as explained above in Step 2A prong 2. Therefore, the claim is ineligible.
Regarding claim 15, Under Step 1 of the Subject Matter Eligibility Test of Products and Processes, the claim is directed towards a non-transitory computer-readable medium which is considered a manufacture, which is one of the four statutory categories.
Next, under a Step 2A Prong 1 Analysis, the claim recites the following limitations, which are interpreted to be, under the broadest reasonable interpretation, abstract ideas:
sample states over a whole state space (mental process, observation)
determine when sampled states are out of distribution out of distribution being a state that is not within the dataset (D) (mental process, evaluation)
assign probability weights to the sampled states that are out of distribution, the probability weights being greater than probability weights assigned to the in-distribution states (mental process, judgement)
Update the Q-function to generate a fitted Q-function solving an optimization problem with a minimization term and a maximization term, wherein the minimization term comprises an expectation of Q-values over states sampled from a distribution
(
ⅆ
ϕ
)
that assigns probabilities to states outside the dataset (D) and over actions sampled from the policy, and penalizes an overall expected reward when a present state is out of distribution (mathematical concept)
the maximization term comprises an expectation of Q-values over states sampled from a marginal distribution
(
d
π
β
)
of states in the dataset (D) and over actions sampled from the policy (mathematical concept)
Therefore, we have to examine the claim under Step 2A prong 2, which considers the additional elements within the claim. The claim’s additional elements are:
A non-transitory computer-readable medium storing instructions
A processor
performing offline reinforcement learning to iteratively update an agent that possesses a policy and a Q-function based on a dataset (D), the dataset (D) having in-distribution
update the policy according to an existing reinforcement learning algorithm.
and control a robotic device using the policy
The non-transitory computer-readable medium storing instructions, processor, “performing offline reinforcement learning to iteratively update an agent that possesses a policy and a Q-function based on a dataset (D), the dataset (D) having in- distribution states”, and to “update the policy according to an existing reinforcement learning algorithm“ are limitations that, as drafted, merely indicates the field of use and technological environment in which to apply the judicial exception, “generally linking” the limitations to the judicial exception. (See MPEP 2106.05(h)) “control a robotic device using the policy”, is considered to be mere instructions to apply a judicial exception, as it applies the judicial exception in no meaningful way, as it merely states that the abstract idea should be applied to achieve a desired result. (See MPEP2106.05(f)(3)) Therefore, these additional elements do not integrate the abstract idea into a practical application. The claim is directed to an abstract idea.
Under a Step 2B analysis, the claim’s addition elements do not amount to significantly
more than the judicial exception as explained above in Step 2A prong 2. Therefore, the claim is ineligible.
Regarding claims 2, 9, 16, the claims recite “the fitted Q-function is a conservative Q-function that lower-bounds an actual Q-function corresponding to an underlying Markov Decision Process in the dataset (D).” As drafted, the limitation merely indicates the field of use and technological environment of the judicial exception, “generally linking” the limitation to the judicial exception. (See MPEP 2106.05(h)) Therefore, the claims are not eligible under U.S.C. 101 for the same reasons as set forth in the rejection of claims 1, 8, and 15.
Regarding claims 3, 10, 17, the claims recite “training a bag of dynamics models (P1,P2,...,P) according to transitions within a dataset ((D)) to output an uncertainty estimation model that indicates when the present state is the out of distribution state. As drafted, the limitation is considered to be mere instructions to apply the judicial exception. (See MPEP 2106.05(f)) Therefore, the claims are not eligible under U.S.C. 101 for the same reasons as set forth in the rejection of claims 1, 8, and 15.
Regarding claims 4, 11, 18, the claims recite the minimizing term is expressed as:
PNG
media_image1.png
40
193
media_image1.png
Greyscale
wherein s is the present state, a is the action,
π
^
k
the policy, Q is the Q-value, and
ⅆ
ϕ
is a distribution that assigns probabilities to states outside the dataset (D); and the maximizing term is expressed as
PNG
media_image2.png
33
205
media_image2.png
Greyscale
wherein s is the present state, a is the action,
π
^
k
the policy, Q is the Q-value, and
ⅆ
ϕ
is a marginal distribution of states in the dataset (D). The limitation as drafted is considered to be a “mathematical concept”, which is an abstract idea. Therefore, the claims are not eligible under U.S.C. 101 for the same reasons as set forth in the rejection of claims 1, 8, and 15.
Regarding claims 5, 12, 19, the claims recite the fitted Q-function further includes a regularization term. The limitation as drafted is considered to be a “mathematical concept”, which is an abstract idea. Therefore, the claims are not eligible under U.S.C. 101 for the same reasons as set forth in the rejection of claims 4, 11, and 18.
Regarding claims 21, 22, and 23, the claims recite “determining, using the uncertainty estimation model, an uncertainty value for each sampled state of a set of the sampled states, wherein a higher uncertainty value indicates that a corresponding sampled state is more likely to be an unseen state; generating a discrete distribution over the set of the sampled states, the discrete distribution assigning, to each sampled state of the set, a probability that is based on the uncertainty value of the sampled state relative to a sum of the uncertainty values of the set of the sampled states; and constructing the distribution (dp) based at least in part on the discrete distribution. “Determining… an uncertainty value for each sampled state of a set of the sampled states, wherein a higher uncertainty value indicates that a corresponding sampled state is more likely to be an unseen state”, and “constructing the distribution (dp) based at least in part on the discrete distribution” are limitations that, as drafted, are interpreted to be, under the broadest reasonable interpretation, “mental processes”, which is a grouping of abstract idea. “Generating a discrete distribution over the set of the sampled states, the discrete distribution assigning, to each sampled state of the set, a probability that is based on the uncertainty value of the sampled state relative to a sum of the uncertainty values of the set of the sampled states” is interpreted to be, under the broadest reasonable interpretation, a “mathematical concept”, which is a grouping of abstract idea. Lastly, “using the uncertainty estimation model” is interpreted to be mere instructions to apply a judicial exception, as it instructs to use the uncertainty estimation model to perform the abstract idea. (See MPEP 2106.05(f)) Therefore, the claims are not eligible under U.S.C. 101 for the same reasons as set forth in the rejection of claims 3, 10, and 17.
Allowable Subject Matter
Claims 1-5, 8-12, 15-19 and 21-23 are allowable under prior art, but require further amendment to be considered eligible subject matter under 101.
Response to Arguments
Applicant's arguments filed August 7th, 2026, have been fully considered but they are not fully persuasive. The arguments and amendment regarding the 103 rejections in the previous action are persuasive. As such, in this action, the 103 rejections have been withdrawn. The applicant argues in substance:
Argument 1: The claims are directed towards an improvement in the function of a machine learning model, integrating the recited judicial exception into a practical application.
The examiner respectfully disagrees. As explained in a prior action, the claims do not appear to recite any improvement in the functionality of a machine learning model, but rather the claim language is seemingly directed towards the improvement of an abstract idea; the abstract idea then being applied to the field of machine learning. When examined, as a whole, the claims are directed towards an abstract idea.
Argument 2: Controlling a robotic device using a policy is not insignificant extra-solution activity, and is unsupported by a Berkheimer analysis
Under step 2A prong 2 of the Subject Matter Eligibility Test for Products and Processes, a new rationale for the limitation “controlling a robotic device using a policy” has been given in the current action. The rationale in the previous action was improper and has been corrected in this action.
As explained in this action, the limitation “controlling a robotic device using a policy” is interpreted to be mere instruction to apply a judicial exception, as the limitation generically recites an effect of the judicial exception, and amounts to a claim that is merely adds the words "apply it" to the judicial exception. (See MPEP 2106.05(f)(3))
Argument 3: The dependent claims 3, 10, 17, and 21-23 further integrate the alleged abstract idea into a practical application.
The examiner respectfully disagrees. As explained above, claims 3, 10, and 17 merely indicated the field of use and technological environment. As such, the claims do not integrate the abstract idea into a practical application, but rather “generally links” training models and an uncertainty estimation model to the abstract idea. Furthermore, the further limiting claims 21, 22, and 23, do not recite any further limitations that would make the claims patent eligible, as explained in the rejection above. As such, the 101 rejections are maintained.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Tyler E Iles whose telephone number is (571)272-5442. The examiner can normally be reached 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.E.I./ Patent Examiner, Art Unit 2122
/KAKALI CHAKI/ Supervisory Patent Examiner, Art Unit 2122
