Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The Examiner notes the applicant’s assertion that: “A computer readable storage medium, as that term is used in the present disclosure, is not to be construed as storage in the form of transitory signals per se”, as stated in paragraph 36 of the filed specification.
Response to Amendment
The previous objections on the claim(s) is/are withdrawn based on the amendments submitted.
Response to Arguments
Applicant's arguments have been fully considered but they are not persuasive. In response to applicant’s argument (Remarks pp. 6-9) that the abstract idea is negated by the claims reciting a solution to a technical problem, the Examiner disagrees that the claims recites this solution and simply changes an action based on a threshold (other elements are not recited). In response to the gist of the applicant’s argument (Remarks pp. 10-11) that the prior art does not recite a threshold as claimed, the Examiner disagrees points out that the "utility gap" or difference in the objective to be achieved will be ascertained by the agent interacting with the environment at each time step, and the reinforcement learning approach, as described in the prior art, and as pointed out in the Office Action, adapts to changes at each time step based on these differences and a threshold value (see Ostrovski, for example paragraphs 5, 28-30). Moreover, the use of threshold values/conditions in goal/task accomplishments are well known in RL as pointed out in for example Ghosh, US 2023/0106474 A1, see for example paragraph 35.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1:
All claims are directed towards either a method, a product or a system and thus satisfies Step 1 as falling into one of the statutory categories.
Step 2A, Prong One:
Independent Claim 1 recites (the same analysis applies to similar independent Claims 7 and 13):
computing utilities for respective candidate actions at a current time step, using a return distribution predictor;
this limitation, under its broadest reasonable interpretation, covers concepts that can be performed in the human mind and therefore would fall under the “Mental Processes” groupings of abstract ideas. That is the human mind is capable of computing with pen and paper, objective values or utilities, from a return distribution function or predictor.
computing a utility gap between a utility of a first action at the current time step and a utility of a reference action;
this limitation, under its broadest reasonable interpretation, covers concepts that can be performed in the human mind and therefore would fall under the “Mental Processes” groupings of abstract ideas. That is the human mind is capable of computing a difference or gap between two action functions using evaluation.
computing a threshold at the current time step for the utility gap;
this limitation, under its broadest reasonable interpretation, covers concepts that can be performed in the human mind and therefore would fall under the “Mental Processes” groupings of abstract ideas. That is the human mind is capable of computing a threshold value pertaining to the above difference/gap using evaluation.
determining whether the utility gap is greater than the threshold;
this limitation, under its broadest reasonable interpretation, covers concepts that can be performed in the human mind and therefore would fall under the “Mental Processes” groupings of abstract ideas. That is the human mind is capable of determining whether the difference/gap is greater than the threshold using evaluation.
and executing the first action at the current time step only when the utility gap is above the threshold.
this limitation, under its broadest reasonable interpretation, covers concepts that can be performed in the human mind and therefore would fall under the “Mental Processes” groupings of abstract ideas. That is the human mind is capable of making a determination to change actions based on a difference/gap being above the threshold, using judgement (a person driving a car in traffic, for instance, make these actions/decisions).
Step 2A, Prong Two:
Claim 1 recites the additional elements of (the same analysis applies to similar independent Claims 7 and 13):
adaptively-repeated action selection in reinforcement learning,
this is considered as generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h).
The further additional elements of “computer” and/or “processors” as recited in independent claims 7 and 13 are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are therefore directed to an abstract idea.
Step 2B:
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements are considered as generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h). The further additional elements of “computer” and/or “processors” as recited in these independent claims amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claims are therefore not patent eligible.
Dependent Claims 2-6, and similar Claims 8-12, and 14-18 are considered as generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-18 are rejected under 35 U.S.C. 103 as being unpatentable over Ostrovski, US 2020/0364557 A1, in view of Ghosh, US 2023/0106474 A1.
Regarding Claim 1, Ostrovski teaches:
A computer-implemented method for adaptively-repeated action selection in reinforcement learning, the method comprising (paragraph 5: “there is provided a method of selecting an action to be performed by a reinforcement learning agent interacting with an environment”):
computing utilities for respective candidate actions at a current time step, using a return distribution predictor (paragraph 59: “The system 100 can implement a “risk-sensitive” action selection policy by selecting actions that are based on more than just the measures of central tendency of the return distributions corresponding to each possible action”. Each possible action represented at its corresponding time step);
computing a utility gap between a utility of a first action at the current time step and a utility of a reference action (paragraph 28: “The system 100 selects actions 102 to be performed by an agent 104 interacting with an environment 106 at each of multiple time steps. At each time step, the system 100 receives data characterizing the current state of the environment”; and paragraph 29: “the state of the environment 106 at the time step (as characterized by the observation 108) depends on the state of the environment 106 at the previous time step and the action 102 performed by the agent 104 at the previous time step”. The state of the environment between the two time steps represents the utility gap).
Although Ostrovski teaches the following (see paragraph 5), Ghosh more directly shows:
computing a threshold at the current time step for the utility gap (paragraph 38: “the same threshold condition and number of steps are used when evaluating the action space 104 of a given reinforcement learning model 102. In some embodiments, the same threshold condition and number of steps are used when evaluating the different combinations of actions included in power set 122, but different threshold conditions and/or numbers of steps can be used when evaluating the corresponding reinforcement learning model 102 at different times”);
determining whether the utility gap is greater than the threshold (paragraph 30: “a goal for executing tasks within a computing system, could be reducing task execution time. The reward function 108 could include a component that rewards actions that result in a task execution time below a threshold amount of time, a component that punishes actions that result in a task execution time above a threshold amount of time, and/or a component that provides a reward proportional to the amount of task execution time resulting from the action”);
and executing the first action at the current time step only when the utility gap is above the threshold (paragraph 24: “During execution, a reinforcement learning agent selects an action from action space 104 and performs the selected action within environment 150 or causes the selected action to be performed within environment 150. The current state of reinforcement learning model 102 is based on the current state of environment 150, and the state that reinforcement learning model 102 transitions to after a selected action is performed is based on the state of environment 150 after the selected action is performed. Accordingly, the transition probabilities 110 of reinforcement learning model 102 are defined based on the properties and dynamics of the environment”; And, paragraph 35: “the threshold condition is based on one or more goals associated with the reinforcement learning model 102. For example, the threshold condition could be based on reaching the one or more goals or reaching at least one of the one or more goals. As another example, the threshold condition could be based on satisfying one or more metrics associated with the one or more goals. For example, if a goal is to reduce task execution time, the threshold condition could be reducing task execution time by a first threshold amount, having a task execution time that is below a second threshold amount, and/or the like. The number of steps can be, for example, a pre-configured number, a number specified via user input during execution of action space evaluator 120, a number associated with a given reinforcement learning model 102, a randomly-generated number, and/or the like. Any suitable threshold condition and/or number of steps can be used to evaluate the different combinations of actions”).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the teachings of Ghosh with that of Ostrovski for using a threshold for executing actions in a reinforcement learning setting.
The ordinary artisan would have been motivated to modify Ostrovski in the manner set forth above for the purposes of executing a task below a threshold amount of time [Ghosh: paragraph 30].
Regarding Claim 2, Ostrovski further teaches:
The computer-implemented method of claim 1, wherein the utility of the reference action is a utility at the current time step of the action that has been taken at the previous time step (paragraph 29: “At each time step, the state of the environment 106 at the time step (as characterized by the observation 108) depends on the state of the environment 106 at the previous time step and the action 102 performed by the agent 104 at the previous time step”).
Regarding Claim 3, Ostrovski further teaches:
The computer-implemented method of claim 1, wherein a p-th percentile of utility gaps in last N time steps before the current time step is adopted as the threshold (paragraph 50: “The quantile value for a probability value with respect to a return distribution refers to a threshold return value below which random draws from the return distribution would fall with probability given by the probability value”. The quantile value representative of the p-th percentile).
Regarding Claim 4, Ostrovski further teaches:
The computer-implemented method of claim 3, wherein a value of p and a value of N are predetermined (paragraph 10: “the first estimated quantile value, and (iii) the second estimated quantile value”; and paragraph 28: “The system 100 selects actions 102 to be performed by an agent 104 interacting with an environment 106 at each of multiple time steps”, these multiple time steps can be predetermined).
Regarding Claim 5, Ostrovski further teaches:
The computer-implemented method of claim 1, further comprising: after the adaptively-repeated action selection, updating the return distribution predictor (paragraph 63: “the system 100 may periodically (e.g., after given numbers of training iterations) update the values of the target quantile function network parameters based on the current values of the quantile function network parameters”; and paragraph 65: “FIG. 3 illustrates a quantile function corresponding to the return distribution illustrated by FIG. 2. The quantile function associates a respective quantile value (represented by the vertical axis) with each of multiple possible probability values”).
Regarding Claim 6, Ostrovski further teaches:
The computer-implemented method of claim 1, wherein the first action has a greater impact than the reference action when the utility gap is greater than the threshold, wherein the first action has a similar impact as the reference action when the utility gap is not greater than the threshold (paragraph 5: “selecting an action to be performed by a reinforcement learning agent interacting with an environment, the method comprising: receiving a current observation characterizing a current state of the environment; for each action of a plurality of actions that can be performed by the agent to interact with the environment: randomly sampling one or more probability values; for each probability value: processing the action, the current observation, and the probability value using a quantile function network having a plurality of network parameters, wherein the quantile function network is a neural network that is configured to process the action, the current observation, and the probability value in accordance with current values of the network parameters to generate a network output that indicates an estimated quantile value for the probability value with respect to a probability distribution over possible returns that would result from the agent performing the action in response to the current observation, wherein a quantile value for a probability value with respect to a probability distribution refers to a threshold value below which random draws from the probability distribution would fall with probability given by the probability value; determining a measure of central tendency of the one or more estimated quantile values generated by the quantile function network; and selecting an action from the plurality of possible actions to be performed by the agent in response to the current observation using the measures of central tendency for the actions. A method of this aspect may be implemented by one or more computers”. And paragraph 30: “the system 100 may receive a reward 110 based on the current state of the environment 106 and the action 102 of the agent 104 at the time step. In general, the reward 110 is a numerical value. The reward 110 can be based on any event or aspect of the environment 106. For example, the reward 110 may indicate whether the agent 104 has accomplished a task (e.g., navigating to a target location in the environment 106) or the progress of the agent 104 towards accomplishing a task”).
Claims 7-12 and 13-18 are similar to Claims 1-6 and are rejected under the same rationale as stated above for those claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See PTO-892 for the relevant prior art where for example the NPL of Duan teaches a distribution function/predictor in reinforcement learning.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVE MISIR whose telephone number is (571)272-5243. The examiner can normally be reached M-R 8-5 pm, F some hours.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Al Kawsar can be reached at 5712703169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVE MISIR/Primary Examiner, Art Unit 2127
Prosecution Insights