DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/19/2022 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
Applicant's arguments filed 02/19/2026 have been fully considered but they are not persuasive.
Rejections Under 35 U.S.C. 101:
Applicant asserts that “The claimed invention provides an improvement to how autonomous agents learn environments and optimize goal achievement. As described in the present application, conventional "reinforcement learning models solving goal-reaching problems may use Q-learning algorithms to determine the optimal action to take in a particular state." Specification at 3. "However, Q-learning has some significant limitations when used in goal-reaching contexts. Q- learning is generally unable to find multiple paths to reach the goal state, and thus is unable to balance the relative risks and rewards of different paths. Additionally, in complex environments, Q-learning requires the agent to interact with the environment a significant number of times before the Q-learning model can give effective results." Id. at 4. The claimed improvements using a "CAE model [that] is trained based on a CAE characteristic" to identify an optimal action using the CAE model that considers the time horizon provides improvements over traditional Q-learning models by enabling more accurate and effective state and action exploration. Specifically, this approach may "reduc[e] the number of times an agent needs to gather information about an environment for the CAE system to be accurate in its action selection." Id. at 8. Further, "by modifying the time horizon, a CAE system allows for improved balancing of the risks of more speedily reaching the goal state with the possibility of not reaching the goal state in time." Id. The provisional application for the present application, incorporated by reference (see paragraph 1 of the specification) provides further discussion of the additional benefits of the claimed approach relative to prior systems. See App. No. 63/155,233. Particularly, prior systems for multi-goal reinforcement learning provide multiple limitations: "an inability to find multiple paths to a goal, high sample complexity, and poor results in complex motion planning tasks." See id. at 1. Further, "[e]xisting algorithms do not allow a dynamic choice of whether to act safely or quickly at test time." In addition, prior solution "still require a very large amount of environmental interactions for effective learning" and prior Q-functions "often causes agents to learn sub-optimal ways of reaching the intended goal." Id. As discussed and with demonstrable experiments, the present invention's use of a CAE characteristic and time horizon provides for learning C* that "is more efficient, and converges in fewer environmental interactions." Id. at 2. By incorporating horizon awareness, "Importantly, an agent need not always act the same way at a particular state in order to reach a particular goal, thanks to horizon-awareness." Id. at 3. This approach also improves on prior "Q-learning" approaches that are empirically shown have "high sample complexity and learn[] sub-optimal paths." Id. at 4. In contrast, and improving on the prior solution of Q-learning, "The C* function does not have this shortcoming." Id. at 4. "This property results in fewer interactions with the environment needed to learn C* and more efficient solutions." Id. The experimental data shown in the provisional application (and incorporated by reference) demonstrate these benefits over prior Q-learning approaches, particularly to show "(i) C-learning enables the speed vs. reliability tradeoff, (ii) the C* function recovered through C-learning meaningfully matches reachability in nonholonomic environments and (iii) C-learning scales to complex and high-dimensional motion planning tasks, resulting in improved sample complexity and goal-reaching." Id. at 7. These experiments did, in fact, demonstrate these benefits. See id. at 7-9. Particularly, these experiments showed that this approach does in fact learn "both risky and safe policies" and experimentally confirmed that "other methods... can only learn one." Id. at 8. For example other specific approaches "recover a single policy, thus not enabling the speed vs reliability trade-off." as recited in Claim 1 (and similarly in Claim 18). (Remarks pg. 6-8)
Examiner’s response:
The Examiner respectfully disagrees. The claim as a whole is still directed to abstract idea mental process. The overall argument is directed towards an improvement in Q-learning and independent claims 1 and 11 do not recite any Q-learning concepts. Examiner highly recommends to amend the claim to reflect the improvement of q-learning as recited in the specification. Although claims are examine in light of specification, the current claim does not mention anything about Q-Learning.
The CAE model as recited is still reciting at a high level. The claim limitation “A system for using a cumulative accessibility estimation (CAE) model comprising: a processor; and one or more non-transitory, computer readable media comprising instructions that, when executed by the processor, cause the processor” is reciting generic computer components see MPEP 2106.05(f). The generic computer components in these steps are recited at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Finally, these technical improvements aspects are reflected in the independent claims. For example the limitations of: access a CAE model, wherein the CAE model estimates a probability that the agent will reach the goal state within a time horizon, and wherein the CAE model is trained based on a CAE characteristic; identify an optimal action of the one or more actions, wherein the optimal action is identified based on the CAE model, and wherein the optimal action is an action for the agent to perform with a highest probability of reaching the goal state within the time horizon; recite the use of these improvements in determining the optimal action sent for the agent to perform. Particularly, the CAE model trained based on a CAE characteristic corresponds to the "C* function" discussed above. See Specification at 20 ("the CAE model 140 models a cumulative accessibility function, or C*"). And, importantly, the limitations for a CAE model that "estimates a probability that the agent will reach the goal state within a time horizon" and determining the optimal action "with a highest probability of reaching the goal state within the time horizon" enables the time-based tradeoff that was not possible with agent-goal planning approaches. These confirm that the disclosed technological improvements are incorporated into the claimed invention. The additional independent claims recite similar limitations. Viewing the claims as a whole under step 2B, this invention provides a practical improvement to the technical field of agent planning for multi-goal problems and that improvement is recited in the claims. As such, the present invention of the independent claims is an improvement to this technical field and is patent eligible under 35 U.S.C 101. The dependent claims are eligible at least due to dependency on these claims. The rejection under § 101 should thus be withdrawn.” (Remarks pg. 9)
Examiner’s response:
The Examiner respectfully disagrees. With regards to the argument “the limitations for a CAE model that "estimates a probability that the agent will reach the goal state within a time horizon" and determining the optimal action "with a highest probability of reaching the goal state within the time horizon" enables the time-based tradeoff that was not possible with agent-goal planning approaches. These confirm that the disclosed technological improvements are incorporated into the claimed invention. The additional independent claims recite similar limitations.” A human could perform identifying an optimal actions and perform what is optimal action and perform highest probability within time horizon. The CAE model as recited is still generic therefore it has been treated under step 2A prong 2. The generic computer components in these steps are recited at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50-57 (January 7, 2019) (“2019 PEG”).
Regarding claim 1
Step 1 – Is the claimed invention directed to a process, machine, manufacture, or composition of matter? – Yes, the claim is directed to a system.
Step 2A Prong 1: The claim recites multiple mental processes, as explained below. The claim recites, inter alia:
“…access a CAE model, wherein the CAE model estimates a probability that the agent will reach the goal state within a time horizon, …identify an optimal action of the one or more actions, wherein the optimal action is identified based on the CAE model, and wherein the optimal action is an action for the agent to perform with a highest probability of reaching the goal state within the time horizon…”
This limitation is directed to the abstract idea of mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (See MPEP 2106.04(a)(2) III. C.).
Step 2A Prong 2: This judicial exception is not integrated into a practical.
In particular, the claim only recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). The additional element of “A system for using a cumulative accessibility estimation (CAE) model comprising: a processor; and one or more non-transitory, computer readable media comprising instructions that… and wherein the CAE model is trained based on a CAE characteristic;” as drafted, is reciting generic computer components. The generic computer components in these steps are recited at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component.
In addition, the claim limitation “…when executed by the processor, cause the processor to: receive agent data from an agent, wherein the agent data describes a present state of the agent and one or more actions that the agent can perform in the present state of the agent; receive a goal state for the agent, wherein the goal state is an intended state for the agent to reach… and transmit, to the agent, the optimal action for the agent to perform.” This limitation as explained by the Supreme Court, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood or conventional. See MPEP 2106.05(g). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea
Step 2B: The claim does not contain significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using generic computer components to perform the abstract idea amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Thus, the claim is not patent eligible.
The claim recites “…when executed by the processor, cause the processor to: receive agent data from an agent, wherein the agent data describes a present state of the agent and one or more actions that the agent can perform in the present state of the agent; receive a goal state for the agent, wherein the goal state is an intended state for the agent to reach… and transmit, to the agent, the optimal action for the agent to perform.” constitute storing and retrieving information in memory, which the courts have found to be well-understood, routine, and conventional. See MPEP 2106.05(d)(II); Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015);
Regarding claim 2
Step 1 – Is the claimed invention directed to a process, machine, manufacture, or composition of matter? – Yes, the claim is directed to a system.
Step 2A Prong 1:
“wherein the agent data contains information about an environment of the agent and wherein the present state of the agent is determined based on the information about the environment of the agent.”
This limitation is directed to the abstract idea of mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (See MPEP 2106.04(a)(2) III. C.).
Step 2A Prong 2: The claim does not appear to recite additional elements that might integrate the judicial exception into a practical application.
Based on the determination in Step 2A of the analysis that the claims are directed to a judicial exception, it must be determined if the claims contain any element or combination of elements sufficient to ensure that the claim amounts to significantly more than the judicial exception. In this case, after considering all claim elements individually and as an ordered combination, it is determined that the claim does not include additional elements that are sufficient to amount of significantly more than the judicial exception.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using generic computer components to perform the abstract idea amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Thus, the claim is not patent eligible.
Regarding claim 3
Step 1 – Is the claimed invention directed to a process, machine, manufacture, or composition of matter? – Yes, the claim is directed to a system.
Step 2A Prong 1:
“wherein the information about the environment of the agent comprises sensor data or meter data captured by the agent.”
This limitation places restriction on the type of item includes and doesn’t change the fact that the underlying data manipulation is mental.
Step 2A Prong 2:
This judicial exception is not integrated into a practical application. The remaining limitations of the claim are as follows:
In particular, the claim only recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). The additional element of “sensor” as drafted, is reciting generic computer components. The generic computer components in these steps are recited at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using generic computer components to perform the abstract idea amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Thus, the claim is not patent eligible.
Regarding claim 4
Step 1 – Is the claimed invention directed to a process, machine, manufacture, or composition of matter? – Yes, the claim is directed to a system.
Step 2A Prong 1:
Step 2A Prong 2:
This judicial exception is not integrated into a practical application. The remaining limitations of the claim are as follows:
In particular, the claim only recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). The additional element of “wherein the CAE model comprises a neural network” as drafted, is reciting generic computer components. The generic computer components in these steps are recited at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using generic computer components to perform the abstract idea amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Thus, the claim is not patent eligible.
Regarding claim 5
Step 1 – Is the claimed invention directed to a process, machine, manufacture, or composition of matter? – Yes, the claim is directed to a system.
Step 2A Prong 1:
“wherein the CAE model estimates a cumulative accessibility function”
This limitation is directed to the abstract idea of mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (See MPEP 2106.04(a)(2) III. C.).
Step 2A Prong 2:
This judicial exception is not integrated into a practical application. The remaining limitations of the claim are as follows:
In particular, the claim only recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). The additional element of “CAE” as drafted, is reciting generic computer components. The generic computer components in these steps are recited at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using generic computer components to perform the abstract idea amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Thus, the claim is not patent eligible.
Regarding claim 6
Step 1 – Is the claimed invention directed to a process, machine, manufacture, or composition of matter? – Yes, the claim is directed to a system.
Step 2A Prong 1:
“wherein the CAE characteristic is a characteristic of the cumulative accessibility function”
This limitation is directed to the abstract idea of mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (See MPEP 2106.04(a)(2) III. C.).
Step 2A Prong 2:
This judicial exception is not integrated into a practical application. The remaining limitations of the claim are as follows:
In particular, the claim only recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). The additional element of “CAE” as drafted, is reciting generic computer components. The generic computer components in these steps are recited at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using generic computer components to perform the abstract idea amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Thus, the claim is not patent eligible.
Regarding claim 7
Step 1 – Is the claimed invention directed to a process, machine, manufacture, or composition of matter? – Yes, the claim is directed to a system.
Step 2A Prong 1:
“wherein the CAE characteristic based on a recursive relationship of the cumulative accessibility function.”
This limitation is directed to the abstract idea of mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (See MPEP 2106.04(a)(2) III. C.).
Step 2A Prong 2: This judicial exception is not integrated into a practical. In particular, the claim only recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). The additional element of “CAE” as drafted, is reciting generic computer components. The generic computer components in these steps are recited at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using generic computer components to perform the abstract idea amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Thus, the claim is not patent eligible.
Regarding claim 8
Step 1 – Is the claimed invention directed to a process, machine, manufacture, or composition of matter? – Yes, the claim is directed to a system.
Step 2A Prong 1:
“wherein the CAE characteristic is based on a modified version of the Bellman equation.”
This limitation is directed to math and a human can perform Bellman equation using pen and paper.
Step 2A Prong 2: The claim does not appear to recite additional elements that might integrate the judicial exception into a practical application.
Based on the determination in Step 2A of the analysis that the claims are directed to a judicial exception, it must be determined if the claims contain any element or combination of elements sufficient to ensure that the claim amounts to significantly more than the judicial exception. In this case, after considering all claim elements individually and as an ordered combination, it is determined that the claim does not include additional elements that are sufficient to amount of significantly more than the judicial exception.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using generic computer components to perform the abstract idea amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Thus, the claim is not patent eligible.
Regarding claim 9
Step 1 – Is the claimed invention directed to a process, machine, manufacture, or composition of matter? – Yes, the claim is directed to a system.
Step 2A Prong 1:
Step 2A Prong 2:
This judicial exception is not integrated into a practical application. The remaining limitations of the claim are as follows:
In particular, the claim only recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). The additional element of “wherein the computer readable media further comprise instructions that cause the processor to update the CAE model based on the agent data and the CAE characteristic” as drafted, is reciting generic computer components. The generic computer components in these steps are recited at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using generic computer components to perform the abstract idea amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Thus, the claim is not patent eligible.
Regarding claim 10
Step 1 – Is the claimed invention directed to a process, machine, manufacture, or composition of matter? – Yes, the claim is directed to a system.
Step 2A Prong 1:
“wherein updating the CAE model based on the agent data comprises determining a difference between a value of the CAE model's estimation of a cumulative accessibility function for the present state of the agent and an action of the one or more actions.”
This limitation is directed to the abstract idea of mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (See MPEP 2106.04(a)(2) III. C.).
Step 2A Prong 2:
This judicial exception is not integrated into a practical application. The remaining limitations of the claim are as follows:
In particular, the claim only recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). The additional element of “CAE model” as drafted, is reciting generic computer components. The generic computer components in these steps are recited at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using generic computer components to perform the abstract idea amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Thus, the claim is not patent eligible.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Naderian et al. (“C-LEARNING: HORIZON-AWARE CUMULATIVE ACCESSIBILITY ESTIMATION”).
Yairi et al. (“Estimating Spatiotemporal Information from Behavioral Sensing Data of Wheelchair Users by Machine Learning Technologies”)
Cheng et al. (“Dynamic Changes in Community Deprivation of Access to Urban Green Spaces by Multiple Transport Modes”)
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAN C MANG whose telephone number is (571)270-7598. The examiner can normally be reached Mon - Fri 8:00-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at 5712707519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VAN C MANG/Primary Examiner, Art Unit 2126
Prosecution Insights