DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in response to the claimed amendment filed on September 03, 2026, in which claims 1-10 are presented for further examination.
Response to Arguments
Applicants’ arguments with respect to claims 1-10 have been considered but are moot in view the new ground of rejection necessitated by amendment. The 35 UCS 102 rejection set forth in the last office action has been withdrawn.
Claim Objections
Claim1-10 are objected to because of the following informalities: Claim 1 recites “a first risk to be taken”, “a second risk to be taken” and “when an action to be applied”. Should read as “a first risk is taken”, “a second risk is taken” and “when an action is applied”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claims 1, 9 and 10 recite “an actor configured to select the action based on the value function and the first parameter within the partial range, update a state of the analysis target, and calculate a reward increased as the updated state becomes a new state”. It is unclear how the updated state becomes a new state. The claims do not specify how the updated state becomes a new state based on the calculated reward increased.
Second, the claims recite “a learner configured to update the value function based on the reward and the second parameter by applying the asymmetric weighting of the learning rate controlled by the second parameter”. It is unclear how the value function would update based on reward by applying the asymmetric weighting of the learning rate controlled by the second parameter. What would be the difference between the first and second learning rate. Is the reward referred to the calculated reward increased or a different reward. Clarification and amendment are advised.
Claims 2-8 are rejected for incorporating the deficiency of their respective base claims by dependency.
Claims 1, 9 and 10 recite the limitation “a learner configured to update the value function based on the reward and the second parameter by applying the asymmetric weighting of the learning rate controlled by the second parameter”. There is insufficient antecedent basis for this limitation “the reward “ in the claim.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested: “reinforcement learning program for executing reinforcement learning”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-10 are rejected under 35 U.S.C. 103 as being unpatentable over Badia et al., (hereinafter “Badia”) article entitled “ADRIÀ PUIGDOMÈNECH BADIA ET AL: "Agent57: Outperforming the Atari Human Benchmark", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 30 March 2020 (2020-03-30), XP081631544” in view of Shigezumi et al., (hereinafter “Shigezumi”) US 20200234123.
As to claim 1, Badia discloses a reinforcement learning device comprising (Abstract lines 8-11 (emphasis added): "We propose Agent57, the first deep RL agent that outperforms the standard human benchmark on all 57 Atari games", page 20 "Computation used"):
a setting unit configured to set a selection range of a first parameter related to a first risk to be taken when an action to be applied to an analysis target is selected from an action group to a partial range of the selection range, and set a second parameter related to a second risk to be taken in learning of a value function for calculating a value serving as a selection guideline of the action (section 2 lines 1-3: "curiosity-driven exploration" and second and third paragraphs, wherein ß is the first parameter related to a first risk, i.e. to the curiosity-driven exploration, page 4 lines 10-12: "at the beginning of each episode and in each actor, NGU uniformly selects a pair (Bj, yj).", section 3.2 second paragraph, Fig. 5 (see caption): wherein the partial ranges in the selected range [0, ß] are shown, page 3 right-hand column second paragraph lines 1-6: "To learn the state-action value function Q',, NGU trains a recurrent neural network Q(x, a, j; Θ), where / is a one-hot vector indexing one of N implied MDPs (in particular (ß,, yill. X is the current observation, a is an action, and Θ are the parameters of the network (including the recurrent state).", wherein the "second risk" is the empirical risk and the second parameter is a parameter in Θ or the priorities (see last paragraph on page 3), or any hyperparameter in Appendix G.3);
an actor configured to select the action based on the value function and the first parameter within the partial range, update a state of the analysis target, and calculate a reward increased as the updated state becomes a new state (Abstract lines 8-11: "We propose Agent57, the first deep RL agent that outperforms the standard human benchmark on all 57 Atari games", page 2 last paragraph point 1 (emphasis added): "A new parameterization of the state-action value function that decomposes the contributions of the intrinsic and extrinsic rewards. As a result, we significantly increase the training stability over a large range of intrinsic reward scales", section 2 second and third paragraphs, page 3 right-hand column third paragraph lines 1-4 (emphasis added): "Our agent is a deep distributed RL agent, in the lineage of R2D2 and NGU. As such, it decouples the data collection and the learning processes by having many actors feed data to a central prioritized replay buffer.", Fig. 2, section 3.1 first paragraph, and Appendix A, D, and E "Actors");
a learner configured to update the value function based on the reward and the second parameter (see page 3 first paragraph: "which of the policies (parameterized by exploration rate and discount factors) to prioritize throughout the training process. This allows the agent to control the exploration/ exploitation trade-off by dedicating more resources to one or the other.", page 3 right-hand column second paragraph lines 1-6: "To learn the state-action value function Qₙ, NGU trains a recurrent neural network Q(x, a, j; θ), where / is a one-hot vector indexing one of N implied MDPs (in particular (B,, v)), X is the current observation, a is an action, and Θ are the parameters of the network (including the recurrent state).", page 3 right-hand column third paragraph lines 1-7 (emphasis added): "Our agent is a deep distributed RL agent, in the lineage of R2D2 and NGU. As such, it decouples the data collection and the learning processes by having many actors feed data to a central prioritized replay buffer. A learner can then sample training data from this buffer, as shown in Fig. 2 (for implementation details and hyperparameters refer to App. E).", Fig. 2, section 3.1 first paragraph, page 20 §"Learner"); and
a determination unit configured to determine, based on a history of the reward calculated by the actor when each of a plurality of the first parameters within the partial range is used, the first parameter to be output to the actor as a specific first parameter used when the actor selects a specific action of updating the analysis target to the new state, and output the specific first parameter to the actor (see section 2 third paragraph, see in particular: "/ is a one-hot vector indexing one of N implied MDPs (in particular (B), Vp))", page 5 first paragraph).
However, Badia fails to disclose the claimed “wherein the second parameter is a value that controls an asymmetric weighting of a learning rate applied when updating the value function, such that a first learning rate is applied when a supervised signal exceeds a predicted value of the value function and a second learning rate, different from the first learning rate, is applied when the supervised signal does not exceed the predicted value of the value function”, “update the value function based on the reward and the second parameter by applying the asymmetric weighting of the learning rate controlled by the second parameter; and wherein the determination unit is configured to extract, for each of the plurality of first parameters, a reward group from the history of the reward based on a percentile range defined by the partial range of the first parameter, and to calculate an expected value of the reward for each of the plurality of first parameters using the extracted reward group”.
Meanwhile, Shigezumi discloses the claimed ““wherein the second parameter is a value that controls an asymmetric weighting of a learning rate applied when updating the value function, such that a first learning rate is applied when a supervised signal exceeds a predicted value of the value function and a second learning rate, different from the first learning rate, is applied when the supervised signal does not exceed the predicted value of the value function”, “update the value function based on the reward and the second parameter by applying the asymmetric weighting of the learning rate controlled by the second parameter; and wherein the determination unit is configured to extract, for each of the plurality of first parameters, a reward group from the history of the reward based on a percentile range defined by the partial range of the first parameter, and to calculate an expected value of the reward for each of the plurality of first parameters using the extracted reward group” (see par.[0005], [0033[-[0042], [0046]-[0052], [0071]-[0087], [0096]-[0132] and [0201]-[0220], reinforcement learning apparatus may update the value function so that the value of the current state approaches the value of the other state having the experience level greater than the current state, and uses the value of the other state having the experience level greater than the current state and thus may reduce the error of the value function and improve the accuracy of the value function. The reinforcement learning apparatus can suppress a correction width at the time of updating of the value function to be equal to or less than the difference between the value of the current state and the value of the other state and may reduce the possibility of adversely affecting the accuracy of the value function.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Badia to have a value that controls an asymmetric weighting of a learning rate applied when updating the value function, such that a first learning rate is applied when a supervised signal exceeds a predicted value of the value function and a second learning rate, different from the first learning rate, is applied when the supervised signal does not exceed the predicted value of the value function and update the value function based on the reward and the second parameter by applying the asymmetric weighting of the learning rate controlled by the second parameter; and wherein the determination unit is configured to extract, for each of the plurality of first parameters, a reward group from the history of the reward based on a percentile range defined by the partial range of the first parameter, and to calculate an expected value of the reward for each of the plurality of first parameters using the extracted reward group, for the purpose of suppressing a correction width at the time of updating of the value function to be equal to or less than the difference between the value of the current state and the value of the other state, thereby reducing the error of the value function and improving the accuracy of the value function.
As to claim 9, claim 9 is a reinforcement learning method for performing the device of claim 1 above. It is rejected under the same rationale.
As to claim 10, claim 10 is reinforcement learning program having instruction for executing the device of claim 1 above. It is rejected under the same rationale.
As to claim 2, the combination of Badia and Shigezumi discloses the invention as claimed. In addition, Badia discloses the reinforcement learning device according to claim 1, wherein the determination unit is configured to calculate an expected value of the reward for the history of the reward of each of the plurality of first parameters within the partial range, and determine, based on the expected value of the reward of the first parameter, the specific first parameter used for next action selection (page 5 first paragraph and Appendix D).
As to claim 3, the combination of Badia and Shigezumi discloses the invention as claimed. In addition, Badia discloses the reinforcement learning device according to claim 2, wherein the determination unit is configured to determine, as the specific first parameter, the first parameter within the partial range in which the expected value of the reward is maximum (see Appendix D and page 20 line 4).
As to claim 4, the combination of Badia and Shigezumi discloses the invention as claimed. In addition, Badia discloses the reinforcement learning device according to claim 1, wherein a lower limit value of the partial range is a lower limit value of the selection range, and an upper limit value of the partial range is smaller than an upper limit value of the selection range (section 4.2 lines 9-16, Fig. 5 (see caption), in particular ß₀, and Appendix G.1).
As to claim 5, the combination of Badia and Shigezumi discloses the invention as claimed. In addition, Badia discloses the reinforcement learning device according to claim 1, wherein a lower limit value of the partial range is larger than a lower limit value of the selection range, and an upper limit value of the partial range is an upper limit value of the selection range (section 4.2 lines 9-16, Fig. 5 (see caption), in particular ß₃₁, and Appendix G.1).
As to claim 6, the combination of Badia and Shigezumi discloses the invention as claimed. In addition, Badia discloses the reinforcement learning device according to claim 1, wherein the learner is configured to update a learning parameter of the value function based on the second parameter and a gradient of the value function (see last paragraph on page 3, first paragraph on page 4, Fig.2, section 4.3, and Appendix C.2 on page 17, page 20 ("Learner").
As to claim 7, the combination of Badia and Shigezumi discloses the invention as claimed. In addition, Badia discloses the reinforcement learning device according to claim 1, further comprising: a plurality of execution entities each including the setting unit, the actor, the learner, and the determination unit, wherein the actors of the plurality of execution entities share the updated state (see page 3 right-hand column third paragraph lines 1-4 (emphasis added): "Our agent is a deep distributed RL agent, in the lineage of R2D2 and NGU. As such, it decouples the data collection and the learning processes by having many actors feed data to a central prioritized replay buffer.", Fig. 2, Appendix E $"Actors", and Figs. 9-10).
As to claim 8, the combination of Badia and Shigezumi discloses the invention as claimed. In addition, Badia discloses the reinforcement learning device according to claim 1, further comprising: a first execution entity including the setting unit, the actor, the learner, and the determination unit; and a second execution entity including the setting unit, the actor, the learner, and the determination unit, and in which the action group includes an action against the action group in the first execution entity, wherein the actors of the first execution entity and the second execution entity are configured to share the updated state, and the actor of the second execution entity is configured to select the action based on the value function, update the state of the analysis target, and calculate the reward such that the reward decreases as the updated state becomes the new state (page 3 right-hand column third paragraph lines 1-4 (emphasis added): "Our agent is a deep distributed RL agent, in the lineage of R2D2 and NGU. As such, it decouples the data collection and the learning processes by having many actors feed data to a central prioritized replay buffer.", Fig. 2, Appendix E "Actors", and Figs. 9-10).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
WO2024/209795 (involved in acquiring an appropriate value of a parameter that is to be used in reinforcement learning, wherein an information processing device sets each of the plurality of candidate values as an upper limit of the difference between action probability distributions that are obtained from the model before and after updating a policy which is applied to the model corresponding to a problem that is a subject of the reinforcement learning, the information processing device ends the reinforcement learning for each of the plurality of candidate values without waiting for reward convergence, on the basis of a prescribed index value and selects at least one candidate value from the plurality of candidate values, on the basis of the prescribed index value).
US20220343180 (involved in estimating a reward function by multiple importance sampling using samples of a decision-making history of a subject and of a decision-making history generated based on a sampling policy. A policy estimation unit estimates a policy by reinforcement learning using the estimated reward function. The reward function estimation unit sets the policy estimated by the policy estimation unit as a new sampling policy, and estimates the reward function by the multiple importance sampling using the samples of the decision-making history of the subject and of the decision-making history generated based on the sampling policy).
US 20200234123 (involved in calculating, in reinforcement learning of repeatedly executing a learning step for a value function that has monotonicity as a characteristic of a value according to a state or an action of a control target, a contribution level of the state or the action of the control target used in the learning step, the contribution level of the state or the action to the reinforcement learning being calculated for each learning step and calculated using a basis function used for representing the value function; determining whether to update the value function, based on the value function after each learning step and the calculated contribution level calculated in each learning step; and updating the value function when the determining determines to update the value function).
US 20220405643 (involved in training an automated agent are disclosed. An example system includes: a communication interface; at least one processor; memory in communication with said at least one processor; software code stored in said memory, which when executed causes said system to: instantiate an automated agent that maintains a reinforcement learning neural network and generates, according to outputs of said reinforcement learning neural network, signals for communicating task requests; receive a plurality of states and a plurality of actions for the automated agent; initialize a learning table Q for the automated agent based on the plurality of states and the plurality of actions; compute a plurality of updated learning tables based on the initialized learning table Q using a utility function, the utility function comprising a monotonically increasing concave function; and generate an averaged learning table Q′ based on the plurality of updated learning tables).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEAN M CORRIELUS whose telephone number is (571)272-4032. The examiner can normally be reached Monday-Friday 6:30a-10p(Midflex).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ann J Lo can be reached at (571)272-9767. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEAN M CORRIELUS/Primary Examiner, Art Unit 2159 October 8, 2026
