Prosecution Insights
Last updated: November 04, 2026
Application No. 18/516,125

SYSTEMS AND METHODS FOR EXECUTING CONFIDENCE-AWARE REINFORCEMENT LEARNING

Non-Final OA §101§103§112
Filed
Nov 21, 2023
Examiner
BLAIN, SHAWN MICHAEL
Art Unit
2143
Tech Center
2100 — Computer Architecture & Software
Assignee
Pascal Poupart
OA Round
1 (Non-Final)
Grant Probability
Favorable
1-2
OA Rounds

Examiner Intelligence

Grants only 0% of cases
0%
Career Allowance Rate
0 granted / 0 resolved
-55.0% vs TC avg
Minimal +0% lift
Without
With
+0.0%
Interview Lift
resolved cases with interview
Typical timeline
Avg Prosecution
4 currently pending
Career history
4
Total Applications
across all art units
This examiner has no resolved cases yet (career too new); statute-level performance unavailable. The Grant Probability card shows Tech Center averages instead.

Office Action

§101 §103 §112
Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is in response to the original filing on 11/21/2023. Claims 1 – 20 are pending and have been considered below. Claim Rejections - 35 USC § 112 The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. Claims 1 – 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 1 recites the limitation "the expert entities" in line 7, and claim 11 recites the limitation "the expert entities" in line 8. There is insufficient antecedent basis for this limitation in these claims. Appropriate correction is required. Claims 2 - 10 and 12 -20 are rejected under 35 USC 112(b) because of their dependencies with the parent claims. Claim Rejections - 35 USC § 101 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1 – 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50 (“2019 PEG”). Claim 1 Step 1: This claim recites “A computer-implemented …method comprising:”; it is directed to the statutory category of a process. Step 2A Prong 1: This claim recites, inter alia: generating a main constraint for the set of expert trajectories, the main constraint being conditioned on a pre-determined confidence level, the pre-determined confidence level being indicative of a probability that the main constraint is at least as constraining as the expert constraint, the main constraint comprising one or more rules limiting the actions that are executable by the AI model; determining a target policy among a plurality of policies, the target policy complying with the main constraint; These limitations recite a mentally performable process that encompasses performing a mental process in a computer environment, in which an evaluation has taken place for judgment of a main constraint conditioned on a pre-determined confidence level and the target policy. (See MPEP 2106.04(a)(2)(III)(C)(2)) Thus, this claim recites a judicial exception. Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of this claim are as follows: accessing a set of expert trajectories, each expert trajectory comprising a sequence of expert state-action pairs, These additional elements does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering similar to obtaining information about transactions. (see MPEP 2106.05(g)) a given one of the expert trajectories including information about a given state of the environment and a corresponding action that is to be executed in response to the given state, These additional elements does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception, wherein generically linking a mentally performable process to the information of respective a given one of the expert trajectories the technical environment. (see MPEP 2106.05(h)) the expert entities complying with an expert constraint that is unknown; These additional elements does not integrate the abstract idea into a practical application because it merely recites to "apply it", the particularity or generality of the application of the judicial exception. (see MPEP 2106.05(f)) A computer-implemented method for executing confidence-aware reinforcement learning for an Artificial Intelligence (AI) model for subsequent deployment of that AI model in an environment, the method comprising: and executing the target policy by the AI model. These additional elements does not integrate the abstract idea into a practical application because it merely recites to "apply it", on a generic computer. (see MPEP 2106.05(f)) Thus, the way in which the additional elements use or interact with the judicial exception when analyzed with this claim as a whole do not integrate the judicial exception into a practical application. Step 2B: The additional elements from Step 2A Prong 2 include insignificant extra-solution activities of data gathering recited by “accessing a set of expert trajectories, each expert trajectory comprising a sequence of expert state-action pairs,”. This is well understood, routine, and conventional activity similar to storing and retrieving information in memory as described in MPEP 2106.05(d)(II)(iv). Additional elements further include linking the judicial exception a particular field of use, invoking computers or other machinery to apply the underlying judicial exception, and adding the words equivalent to “apply it” with the judicial exception. Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP 2106.05. Claim 2 Step 1: a process, as in claim 1 Step 2A Prong 1: This claim recites, inter alia: determining a policy complying with the main constraint; executing the policy by iteratively: executing the actions of the policy, These limitations recite a mentally performable process that encompasses performing a mental process in a computer environment, in which an evaluation and judgement of the respective policy occurs. (See MPEP 2106.04(a)(2)(III)(C)(2)) and adjusting the policy based on outcomes of the actions and received rewards. These limitations recite a mentally performable process that encompasses performing a mental process in a computer environment, in which an evaluation has taken place in deciding the policy. (see MPEP 2106.04(a)(2)(III)(C)(2)) Thus, this claim recites a judicial exception. Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of this claim are as follows: accessing a set of policies, each policy being a mapping from states to actions for the sequences of expert state-action pairs of the expert trajectories, an execution of the policy aiming at maximizing a reward; These additional elements does not integrate the abstract idea into a practical application because it recites a high level of generality and insignificant extra-solution activity of data gathering and selecting a particular data source to be manipulated similar to obtaining information and selecting information for collection, analysis and display. (see MPEP 2106.05(g)) receiving indication of rewards from and states of the environment, These additional elements does not integrate the abstract idea into a practical application because it recites a high level of generality and insignificant extra-solution activity of data gathering and selecting a particular data source to be manipulated similar to obtaining information about transactions (see MPEP 2106.05(g)) Thus, the way in which the additional elements use or interact with the judicial exception when analyzed with this claim as a whole do not integrate the judicial exception into a practical application. Step 2B: The additional elements from Step 2A Prong 2 include insignificant extra-solution activities of data gathering recited by “ accessing a set of policies, each policy being a mapping from states to actions for the sequences of expert state-action pairs of the expert trajectories, an execution of the policy aiming at maximizing a reward;” and “receiving indication of rewards from and states of the environment,” This is well understood, routine, and conventional activity similar storing and retrieving information in memory as described in MPEP 2106.05(d)(II)(iv). Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP 2106.05 Claim 3 Step 1: a process, as in claim 2. Step 2A Prong 1: This claim recites, inter alia: prior to executing the policy: determining a policy-value of the target policy; in response to the policy-value being below a pre-determined value threshold, flagging the set of expert trajectories as insufficient. These limitations recite a mentally performable process that encompasses performing a mental process, in which an evaluation has taken place in determining the flagging. (See MPEP2106.04(a)(2)(III)) Thus, this claim recites a judicial exception. Step 2A Prong 2 & Step 2B: There are no additional elements recited so this claim does not provide a practical application and is not considered to be significantly more. As such, this claim is patent ineligible. Claim 4 Step 1: a process, as in claim 3. Step 2A Prong 1: This claim recites, inter alia: further comprising augmenting the set of expert trajectories with additional expert trajectories until the policy-value exceeds the pre-determined value threshold. These limitations recite a mentally performable process that encompasses performing a mental process that can be done with pen and paper, tracking the adding of a set of data of expert trajectories and using an evaluation for the set’s policy value. (see MPEP2106.04(a)(2)(III)) Thus, this claim recites a judicial exception. Step 2A Prong 2 & Step 2B: There are no additional elements recited so this claim does not provide a practical application and is not considered to be significantly more. As such, this claim is patent ineligible. Claim 5 Step 1: a process, as in claim 3. Step 2A Prong 1: This claim recites, inter alia: wherein determining the policy-value of the policy comprises determining an expected cumulative reward based on rewards associated with the action-state pairs of the policy. These limitations recite a mentally performable process that encompasses performing a mental process in a computer environment, thus an evaluation. (see MPEP 2106.04(a)(2)(III)(C)(2) Thus, this claim recites a judicial exception. Step 2A Prong 2 & Step 2B: There are no additional elements recited so this claim does not provide a practical application and is not considered to be significantly more. As such, this claim is patent ineligible. Claim 6 Step 1: a process, as in claim 1. Step 2A Prong 1: This claim recites, inter alia: wherein generating a main constraint for the set of expert trajectories comprises: determining a constraint distribution based on the set of expert trajectories; selecting a constraint from the constraint distribution based on the pre-determined confidence level as the main constraint. These limitations recite a mentally performable process that encompasses performing a mental process in a computer environment, thus providing a judgement in selection of a constraint after an evaluation. (see MPEP 2106.04(a)(2)(III)(C)(2) Thus, this claim recites a judicial exception. Step 2A Prong 2 & Step 2B: There are no additional elements recited so this claim does not provide a practical application and is not considered to be significantly more. As such, this claim is patent ineligible. Claim 7 Step 1: a process, as in claim 6. Step 2A Prong 1: This claim recites, inter alia: wherein selecting the constraint from the constraint distribution comprises selecting the lower boundary constraint of a quantile of the constraint distribution based on the pre-determined confidence level. These limitations recite a mathematical concept that encompasses a mathematical relationship relating to selecting a variable lower boundary constraint to pre-determined confidence level value of a quantile of the constraint distribution. (see MPEP 2106.04(a)(2)(I)(A) Thus, this claim recites a judicial exception. Step 2A Prong 2 & Step 2B: There are no additional elements recited so this claim does not provide a practical application and is not considered to be significantly more. As such, this claim is patent ineligible. Claim 8 Step 1: a process, as in claim 7. Step 2A Prong 1: This claim recites, inter alia: wherein the main constraint is: q u a n t i l e P ( c ) 1 - λ where P(c) is the constraint distribution and λ is the pre-determined confidence level. These limitations recite a mathematical concept that encompasses a mathematical formulas or equations, e.g. “ q u a n t i l e P ( c ) 1 - λ where P(c) is the constraint distribution and λ is the pre-determined confidence level.”. (see MPEP 2106.04(a)(2)(I)(B) Thus, this claim recites a judicial exception. Step 2A Prong 2 & Step 2B: There are no additional elements recited so this claim does not provide a practical application and is not considered to be significantly more. As such, this claim is patent ineligible. Claim 9 Step 1: a process, as in claim 6 Step 2A Prong 1: This claim recites, inter alia: adjusting a template distribution according to the set of contribution factors to form the constraint distribution. These limitations recite a mathematical concept that encompasses mathematical calculations, similar to using an algorithm for determining the optimal number as described in MPEP 2106.04(a)(2)(I)(C). Thus, this claim recites a judicial exception. Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of this claim are as follows: wherein determining a constraint distribution comprises: employing a neural network encoding the set of expert trajectories to determine, for each of the expert trajectory, a set of contribution factors; and These additional elements does not integrate the abstract idea into a practical application because it merely recites to "apply it", on a computers or other machinery merely as a tool to perform an existing process. (see MPEP 2106.05(f)) Thus, the way in which the additional elements use or interact with the judicial exception when analyzed with this claim as a whole do not integrate the judicial exception into a practical application. Step 2B: The additional elements from Step 2A Prong 2 includes adding the words equivalent to “apply it” with the judicial exception. Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP 2106.05. Claim 10 Step 1: a process, as in claim 9 Step 2A Prong 1: This claim recites the same abstract ideas as in claim 9 as the judicial exception. Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of this claim are as follows: wherein each expert trajectory is encoded with a corresponding encoder having corresponding weights in the neural network. These additional elements does not integrate the abstract idea into a practical application because it merely recites to "apply it", the idea of a solution or outcome. (see MPEP 2106.05(f)) Thus, the way in which the additional elements use or interact with the judicial exception when analyzed with this claim as a whole do not integrate the judicial exception into a practical application. Step 2B: The additional elements from Step 2A Prong 2 includes adding the words equivalent to “apply it” with the judicial exception. Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP 2106.05. Claims 11 – 20 Step 1: These claims are directed to “ A system for… comprising a controller and a memory storing a plurality of executable instructions which, when executed by the controller, cause the system to: “ Therefore, these claims are directed to the statutory category of machines. Step 2A Prong 1: These claims recite the same abstract ideas as in claims 1 – 10 respectively, as the judicial exception. Step 2A Prong 2: The judicial exception recited in this claim is not integrated into a practical application. These claims recite substantially the same additional elements as in claims 1 - 10, respectively. The only difference between claims 11 - 20 and claims 1 – 10, are that claims 11 - 20 are directed to a “ A system for… comprising a controller and a memory storing a plurality of executable instructions which, when executed by the controller, cause the system to:“ However, mere recitation that a judicial exception is to be performed using generic computer equipment in their ordinary capacity, i.e. a computer implemented system, the system comprising a controller and a memory storing a plurality of executable instructions which, when executed by the controller for causing a processing circuitry to execute a process, cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). With that exception, the analysis at this step mirrors that of claims 1 -10 . Step 2B: The additional elements from Step 2A Prong 2 of this claim does not contain significantly more than the judicial exception. The only difference between claims 11 - 20 and claims 1 – 10, are that claims 11 - 20 are directed to a “A system for… comprising a controller and a memory storing a plurality of executable instructions which, when executed by the controller, cause the system to: “However, mere recitation that a judicial exception is to be performed using generic computer equipment in their ordinary capacity, i.e. a computer implemented system, the system comprising a controller and a memory storing a plurality of executable instructions which, when executed by the controller for causing a processing circuitry to execute a process, cannot amount to significantly more than the judicial exception. See MPEP 2106.05(f). With that exception, the analysis at this step mirrors that of claims 1 -10 . Claim Rejections - 35 USC § 103 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claim(s) 1- 6, 9 – 16, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Abdolmaleki et al. (US 20230082326 A1), (hereinafter Abdolmaleki) in view of Papadimitriou et al. (“Bayesian Inverse Constrained Reinforcement Learning”) (hereinafter Papadimitriou). Papadimitriou was cited in IDS filed 11/21/2023. Regarding claim 1, Abdolmaleki teaches the method comprising: accessing a set of expert trajectories ([0006] “obtaining a set of one or more trajectories.” Examiners Remarks: trajectories derives from an agent (the expert)), each expert trajectory comprising a sequence of expert state-action pairs, a given one of the expert trajectories including information about a given state of the environment and a corresponding action that is to be executed in response to the given state, ([0006], “Each trajectory comprises a state of an environment, an action applied by the agent to the environment according to a previous policy in response to the state, and a set of rewards for the action”. Furthermore, [0082],“objective-specific rewards, an action-value function (Q-function) is provided for each objective. The action-value function maps states and actions to a value.” Examiners Remarks: Wherein an action-value function (Q-function) that maps state and action to a value/return describes a sequence of expert state-action pairs , be represented recursively as well [0082]) generating a main constraint for the set of expert trajectories ([0038], “determining an action-value function for each of the plurality of objectives based on the set of one or more trajectories” furthermore [0074] “set a constraint per objective. These constraints can control the influence of each objective on the policy, e.g. by constraining the KL-divergence between each objective-specific distribution and the current policy. The higher the constraint value, the more influence the objective has”), the main constraint being conditioned on a pre-determined confidence level, the pre-determined confidence level being indicative of a probability that the main constraint is at least as constraining as the expert constraint ( [0033]“determining a probability distribution (such as an action distribution or a state-action distribution) for each of the plurality of objectives based on the set of one or more trajectories” Furthermore, [0017], “The difference between the objective-specific policy and the previous policy may be determined based on the Kullback-Leibler divergence or any other appropriate measure of the difference between distributions.” Examiners Remarks: Wherein the KL divergence constraints the new policy from deviating from the old policy.), the main constraint comprising one or more rules limiting the actions that are executable by the AI model ([0014], “The KL constraint in this step has a regularization effect that prevents the policy from overfitting to the sample-based action distributions, and therefore avoids premature convergence and improves stability of learning.”); determining a target policy among a plurality of policies, the target policy complying with the main constraint ([0033], “determining an updated policy based on a combination of the probability distributions for the plurality of objectives” [0081]the neural network system 100 applies multiple objectives.); and executing the target policy by the AI model. ([0033], “ determining an updated policy based on a combination of the probability distributions for the plurality of objectives” Furthermore, [0081], “the neural network system 100 applies multiple objectives”) Abdolmaleki does not teach a computer-implemented method for executing confidence-aware reinforcement learning for an Artificial Intelligence (AI) model for subsequent deployment of that AI model in an environment, and the expert entities complying with an expert constraint that is unknown; However, Papadimitriou teaches A computer-implemented method for executing confidence-aware reinforcement learning for an Artificial Intelligence (AI) model for subsequent deployment of that AI model in an environment, ([Page 3, Section 2.4], “infer constraints is the quantification of the uncertainty in that estimation. Safety critical applications require safety guarantees during the deployment of an agent. We examine the utility of a simple active learning acquisition function which is based on the variance of the constraint estimates.” Furthermore [Page 8, Section 4 ], “Our method manages to infer with high accuracy most of the constraints while providing confidence values on the estimation that can be valuable in safety critical applications”) the expert entities complying with an expert constraint that is unknown; ([Page 7, Section 3.2], “ the Algorithm predicts that part of the goal set is a constraint set as in that part we essentially observe very few demonstrations.” Furthermore, [Page 8, Section 4], “infer the unknown constraints in a discrete and continuous state MDP.”) Because Abdolmaleki and Papadimitriou are analogous art and within the same field of endeavor, specifically computer-implemented systems for trajectory inference from expert demonstrations for the performance of a sequence of actions in any given state. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine Abdolmaleki’s, multi-objective reinforcement learning with Papadimitriou’s, Bayesian inverse constrained reinforcement learning (BICRL) . This modification would have been motivated by the desire infer unknown constraints by observing demonstrations (trajectories) in an environment with a known nominal reward function. (Papadimitriou, [Page 1, Section 1]) Regarding claim 2, the combination Abdolmaleki and Papadimitriou teaches the claimed invention as claimed in claim 1, further comprising: accessing a set of policies, each policy being a mapping from states to actions for the sequences of expert state-action pairs of the expert trajectories (Abdolmaleki ,[0092], “The neural network system evaluates the previous policy—old by learning state-action value (Q) functions.” Furthermore, Abdolmaleki, [0091], “When determining an action-value function for each objective 220, this may be an update to a previous action-value function for the objective. This determination/update is based on the trajectories.”), an execution of the policy aiming at maximizing a reward; (Abdolmaleki ,[0054], “The objectives may include objectives to maximize or minimize one or more of: an estimated likelihood that the user will respond favorably to being recommended the (content) item, a constraint on the suitability of one or more recommended items”) determining a policy complying with the main constraint (Abdolmaleki, [0012], “The relative contribution of each objective to the updated policy can be scaled through use of constraints on the determination of the objective-specific policies.”); executing the policy by iteratively: executing the actions of the policy, (Abdolmaleki ,[0093], “ To achieve this, the following update may be applied at each iteration of training to learn a Q-function Q.sub.k.sup.π.sup.old(s, a) for each objective k, parameterized by ϕ.sub.k: PNG media_image1.png 70 564 media_image1.png Greyscale where {circumflex over (Q)}.sub.k (S, A, R) is a target action-value function (a target Q-function) based on state S, action A and reward R vectors. The target Q-value is an estimate of the sum of discounted rewards (e.g. as determined from one or more trajectories obtained by running the policy).” Examiner Remarks: Wherein iterating through actions in a given state, based on policy ) and adjusting the policy based on outcomes of the actions and received rewards. (Abdolmaleki, [0080], “update parameters of the policy based on the rewards 108 received for each objective”. Examiner’s Remarks: wherein the rewards relates to the (outcome) of the corresponding action) Regarding claim 3, the combination Abdolmaleki and Papadimitriou teaches the claimed invention as claimed in claim 2, further comprising, prior to executing the policy: determining a policy-value of the target policy (Abdolmaleki ,[0039], “each action-value function determining an action value representing an estimated return according to the corresponding objective that would result from the agent performing a given action in response to a given state according to the previous policy”) ; in response to the policy-value being below a pre-determined value threshold, flagging the set of expert trajectories as insufficient. (Papadimitriou , [Page 1, Section 1], “infer a full Bayesian posterior distribution over constraints.” Examiners Remarks: wherein response to the determined “policy-value” an inference is made from viewing that there is insufficient trajectories for the unknown) Because Abdolmaleki and Papadimitriou are analogous art and within the same field of endeavor, specifically computer-implemented systems for trajectory inference from expert demonstrations/trajectories for the performance of a sequence of actions in any given state. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine Abdolmaleki’s, multi-objective reinforcement learning with Papadimitriou’s, Bayesian inverse constrained reinforcement learning (BICRL) . This modification would have been motivated by the desire infer unknown constraints by observing demonstrations in an environment with a known nominal reward function. (Papadimitriou, [Page 1, Section 1]) Regarding claim 4, the combination Abdolmaleki and Papadimitriou teaches the claimed invention as claimed in claim 3, further comprising augmenting the set of expert trajectories with additional expert trajectories (Papadimitriou , [Page 2, Section 1], “the agent can query the expert for specific expert actions or partial demonstrations, without requiring full trajectories.”) until the policy-value exceeds the pre-determined value threshold. (Papadimitriou , [Page 3, Section 2.3], “For each of these samples the log-likelihood of the new configuration is computed and compared with the log-likelihood of the existing configuration.” Examiner Remarks: wherein a query the expert for expert action or partial demonstration is interpretated as augmenting the set of expert trajectories with additional expert trajectories i.e. additional partial demonstration/trajectories) Regarding claim 5, the combination Abdolmaleki and Papadimitriou teaches the claimed invention as claimed in claim 3, wherein determining the policy-value of the policy comprises determining an expected cumulative reward based on rewards associated with the action-state pairs of the policy. (Abdolmaleki ,[0082], “objective-specific rewards, an action-value function (Q-function) is provided for each objective. The action-value function maps states and actions to a value.” Furthermore [0081] “A policy π.sub.θ(a|s) is defined as a state conditional distribution over actions parametrized by θ. Together with the transition probabilities, this gives rise to a state visitation distribution μ(s).”) Regarding claim 6, the combination Abdolmaleki and Papadimitriou teaches the claimed invention as claimed in claim 1, wherein generating a main constraint for the set of expert trajectories comprises: determining a constraint distribution based on the set of expert trajectories; (Abdolmaleki ,[0039], “determining an action-value function for each of the plurality of objectives based on the set of one or more trajectories”) selecting a constraint from the constraint distribution based on the pre-determined confidence level as the main constraint. (Abdolmaleki ,[0074], “set a constraint per objective These constraints can control the influence of each objective on the policy, e.g. by constraining the KL-divergence between each objective-specific distribution and the current policy. The higher the constraint value, the more influence the objective has.”) Regarding claim 9, the combination Abdolmaleki and Papadimitriou teaches the claimed invention as claimed in claim 6, wherein determining a constraint distribution comprises: employing a neural network encoding the set of expert trajectories to determine, for each of the expert trajectory, a set of contribution factors (Abdolmaleki, [0110], “Since the constraints ϵ.sub.k encode the preferences over objectives, solving this optimization problem with good satisfaction of constraints is important for learning a policy that satisfies the desired preferences. For nonparametric action distributions q.sub.k(a|s), these constraints can be satisfied exactly. One could use any policy gradient method to obtain q.sub.k(a|s) in a parametric form instead. However, solving the constrained optimization for parametric q.sub.k(a|s) is not exact, and the constraints may not be well satisfied, which impedes the use of ϵ.sub.k to encode preferences. Moreover, assuming a parametric q.sub.k(a|s) requires maintaining a function approximator (e.g., a neural network) per objective.” Examiners Remarks: Wherein under BRI preference is interpretated as derived from “the set of expert trajectories”, and each of the expert trajectory contain preference.) ; and adjusting a template distribution according to the set of contribution factors to form the constraint distribution. ( Abdolmaleki ,[0106], “ To obtain the per-objective improved action distributions q.sub.k(a|s), the reinforcement learning objective is optimized for each objective Q.sub.k: PNG media_image2.png 126 318 media_image2.png Greyscale ” Examiners Remarks: Wherein this distribution, “template distribution” is subject to change, “adjusting” because of action, state, and etc. “set of contribution factors” ) Regarding claim 10, the combination of Abdolmaleki and Papadimitriou teaches the claimed invention as claimed in claim 9, wherein each expert trajectory is encoded with a corresponding encoder having corresponding weights in the neural network (Abdolmaleki ,[0115], “given trajectory snippets τ={(s.sub.0,a.sub.0,r.sub.0), . . . , (s.sub.T,a.sub.T,r.sub.T)} where r.sub.t denotes a reward vector {r.sub.k(s.sub.t,a.sub.t)}.sub.k=1.sup.N that consists of rewards for all N objectives, value function parameters ϕ.sub.k are found by optimizing the following objective: PNG media_image3.png 62 238 media_image3.png Greyscale ” Furthermore, Abdolmaleki , [0110], “assuming a parametric q.sub.k(a|s) requires maintaining a function approximator (e.g., a neural network) per objective”) Regarding claims 11 – 16, 19, and 20, these are system claims that corresponds to the methods of claims 1 – 6, 9, and 10 respectively. Therefore, claims 11 – 16, 19, and 20 are rejected for the same reason as claims 1 – 6, 9, and 10 above. Claim(s) 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Abdolmaleki et al. (US 20230082326 A1), (hereinafter Abdolmaleki) in view of Papadimitriou et al. (“Bayesian Inverse Constrained Reinforcement Learning”) (hereinafter Papadimitriou) as applied to claims 6, and 16, and in further view of Marecki et al. (US 20190205534 A1) (hereinafter Marecki) Regarding claim 7, the combination Abdolmaleki and Papadimitriou teaches the claimed invention as claimed in claim 6. The combination Abdolmaleki and Papadimitriou does not teach wherein selecting the constraint from the constraint distribution comprises selecting the lower boundary constraint of a quantile of the constraint distribution based on the pre-determined confidence level. However, Marecki teaches wherein selecting the constraint from the constraint distribution comprises selecting the lower boundary constraint of a quantile of the constraint distribution based on the pre-determined confidence level. (Marecki, [0043], y.sub.k.sup.t may represent the extent of underage in the random reward (random due to uncertainty in attacker type k) at time t, that is under and below a lower quantile of the random reward distribution, which is captured by η.sup.t Taken together through the expression in the second term in constraint (12), (i.e. the term that is multiplied by (1-A), the second term may capture the expected undesirable rewards that are below a certain quantile of the reward distribution. ” Examiner Remarks: Wherein (1−λ) is interpreted as the pre-determined confidence level, η.sup.t is interpretated as the lower quantile, and the random reward distribution is interpretated as the constraint distribution.) Because Abdolmaleki, Papadimitriou and Marecki are analogous art and within the same field of endeavor, specifically computer-implemented systems for estimation using probabilities given uncertainty, e.g. unknown factors. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the combination of Abdolmaleki’s and Papadimitriou’s multi-objective - Bayesian inverse constrained reinforcement learning (BICRL) with Marecki’s estimation based on rewards using probability distributions. This modification would have been motivated by the desire to mitigate effects of an attack and reallocation of the resources in the security domain (Marecki, [0002]) Regarding claim 8, the combination Abdolmaleki, Papadimitriou and Marecki teaches the claimed invention as claimed in claim 7, wherein the main constraint is: q u a n t i l e P ( c ) 1 - λ where P(c) is the constraint distribution and λ is the pre-determined confidence level. (Marecki ,[0043] “ y.sub.k.sup.t may represent the extent of underage in the random reward (random due to uncertainty in attacker type k) at time t, that is under and below a lower quantile of the random reward distribution, which is captured by η.sup.t Taken together through the expression in the second term in constraint (12), (i.e. the term that is multiplied by (1-A), the second term may capture the expected undesirable rewards that are below a certain quantile of the reward distribution. ” Examiner Remarks: Wherein (1−λ) is interpreted as the pre-determined confidence level, η.sup.t is interpretated as the lower quantile, and the random reward distribution is interpretated as the constraint distribution, “P(c)”.) Regarding claim 17, this is system claim that corresponds to the methods of claim 7 respectively. Therefore, claim 17 is rejected for the same reason as claim 7. Conclusion The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. GU, (US 20170228662 A1) discloses a system for selecting actions to be performed by an agent interacting with an environment from a continuous action space of actions. Gendron-Bellemare, (US 20210110271 A1) discloses a system implemented as computer programs on one or more computers in one or more locations that selects actions to be performed by an agent interacting with an environment. Clune, (US 20200166896 A1) discloses a deep reinforcement learning based model inputs sensor data to determine one or more actions to be performed by a control system. Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAWN BLAIN whose telephone number is (571)270-1815. The examiner can normally be reached Mon - Fri: 8AM - 5PM. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /S.B./ Examiner, Art Unit 2143 /JENNIFER N WELCH/ Supervisory Patent Examiner, Art Unit 2143
Read full office action

Prosecution Timeline

Nov 21, 2023
Application Filed
Oct 19, 2026
Non-Final Rejection mailed — §101, §103, §112 (current)

Strategy Recommendation AI-generated — please review before filing

Get a prosecution strategy drawn from examiner precedents, rejection analysis, and claim mapping.
Typically takes 5-10 seconds — AI-generated, attorney review required before filing

Prosecution Projections

1-2
Expected OA Rounds
Grant Probability
Low
PTA Risk
Based on 0 resolved cases by this examiner. Grant probability derived from career allowance rate.

Sign in with your work email

Enter your email to receive a magic link. No password needed.

Personal email addresses (Gmail, Yahoo, etc.) are not accepted.

Free tier: 3 strategy analyses per month