DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in response to claimed invention filed on April 25, 2024, in which claims 1-10 are presented for examination.
Information Disclosure Statement
The information disclosure statement filed on April 25, 2024 complies with the provisions of 37 CFR 1.97, 1.98 and MPEP § 609. It has been placed in the application file. The information referred to therein has been considered as to the merits.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-10 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Badia et al., (hereinafter “Badia”) article entitled “ADRIÀ PUIGDOMÈNECH BADIA ET AL: "Agent57: Outperforming the Atari Human Benchmark", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 30 March 2020 (2020-03-30), XP081631544”
As to claim 1, Badia discloses a reinforcement learning device comprising (Abstract lines 8-11 (emphasis added): "We propose Agent57, the first deep RL agent that outperforms the standard human benchmark on all 57 Atari games", page 20 "Computation used"):
a setting unit configured to set a selection range of a first parameter related to a first risk to be taken when an action to be applied to an analysis target is selected from an action group to a partial range of the selection range, and set a second parameter related to a second risk to be taken in learning of a value function for calculating a value serving as a selection guideline of the action (section 2 lines 1-3: "curiosity-driven exploration" and second and third paragraphs, wherein ß is the first parameter related to a first risk, i.e. to the curiosity-driven exploration, page 4 lines 10-12: "at the beginning of each episode and in each actor, NGU uniformly selects a pair (Bj, yj).", section 3.2 second paragraph, Fig. 5 (see caption): wherein the partial ranges in the selected range [0, ß] are shown, page 3 right-hand column second paragraph lines 1-6: "To learn the state-action value function Q',, NGU trains a recurrent neural network Q(x, a, j; Θ), where / is a one-hot vector indexing one of N implied MDPs (in particular (ß,, yill. X is the current observation, a is an action, and Θ are the parameters of the network (including the recurrent state).", wherein the "second risk" is the empirical risk and the second parameter is a parameter in Θ or the priorities (see last paragraph on page 3), or any hyperparameter in Appendix G.3);
an actor configured to select the action based on the value function and the first parameter within the partial range, update a state of the analysis target, and calculate a reward increased as the updated state becomes a new state (Abstract lines 8-11: "We propose Agent57, the first deep RL agent that outperforms the standard human benchmark on all 57 Atari games", page 2 last paragraph point 1 (emphasis added): "A new parameterization of the state-action value function that decomposes the contributions of the intrinsic and extrinsic rewards. As a result, we significantly increase the training stability over a large range of intrinsic reward scales", section 2 second and third paragraphs, page 3 right-hand column third paragraph lines 1-4 (emphasis added): "Our agent is a deep distributed RL agent, in the lineage of R2D2 and NGU. As such, it decouples the data collection and the learning processes by having many actors feed data to a central prioritized replay buffer.", Fig. 2, section 3.1 first paragraph, and Appendix A, D, and E "Actors");
a learner configured to update the value function based on the reward and the second parameter (see page 3 first paragraph: "which of the policies (parameterized by exploration rate and discount factors) to prioritize throughout the training process. This allows the agent to control the exploration/ exploitation trade-off by dedicating more resources to one or the other.", page 3 right-hand column second paragraph lines 1-6: "To learn the state-action value function Qₙ, NGU trains a recurrent neural network Q(x, a, j; θ), where / is a one-hot vector indexing one of N implied MDPs (in particular (B,, v)), X is the current observation, a is an action, and Θ are the parameters of the network (including the recurrent state).", page 3 right-hand column third paragraph lines 1-7 (emphasis added): "Our agent is a deep distributed RL agent, in the lineage of R2D2 and NGU. As such, it decouples the data collection and the learning processes by having many actors feed data to a central prioritized replay buffer. A learner can then sample training data from this buffer, as shown in Fig. 2 (for implementation details and hyperparameters refer to App. E).", Fig. 2, section 3.1 first paragraph, page 20 §"Learner"); and
a determination unit configured to determine, based on a history of the reward calculated by the actor when each of a plurality of the first parameters within the partial range is used, the first parameter to be output to the actor as a specific first parameter used when the actor selects a specific action of updating the analysis target to the new state, and output the specific first parameter to the actor (see section 2 third paragraph, see in particular: "/ is a one-hot vector indexing one of N implied MDPs (in particular (B), Vp))", page 5 first paragraph).
As to claim 9, claim 9 is a reinforcement learning method for performing the device of claim 1 above. It is rejected under the same rationale.
As to claim 10, claim 10 is reinforcement learning program having instruction foe executing the device of claim 1 above. It is rejected under the same rationale.
As to claim 2, Badia discloses the reinforcement learning device according to claim 1, wherein the determination unit is configured to calculate an expected value of the reward for the history of the reward of each of the plurality of first parameters within the partial range, and determine, based on the expected value of the reward of the first parameter, the specific first parameter used for next action selection (page 5 first paragraph and Appendix D).
As to claim 3, Badia discloses the reinforcement learning device according to claim 2, wherein the determination unit is configured to determine, as the specific first parameter, the first parameter within the partial range in which the expected value of the reward is maximum (see Appendix D and page 20 line 4).
As to claim 4, Badia discloses the reinforcement learning device according to claim 1, wherein a lower limit value of the partial range is a lower limit value of the selection range, and an upper limit value of the partial range is smaller than an upper limit value of the selection range (section 4.2 lines 9-16, Fig. 5 (see caption), in particular ß₀, and Appendix G.1).
As to claim 5, Badia discloses the reinforcement learning device according to claim 1, wherein a lower limit value of the partial range is larger than a lower limit value of the selection range, and an upper limit value of the partial range is an upper limit value of the selection range (section 4.2 lines 9-16, Fig. 5 (see caption), in particular ß₃₁, and Appendix G.1).
As to claim 6, Badia discloses the reinforcement learning device according to claim 1, wherein the learner is configured to update a learning parameter of the value function based on the second parameter and a gradient of the value function (see last paragraph on page 3, first paragraph on page 4, Fig.2, section 4.3, and Appendix C.2 on page 17, page 20 ("Learner").
As to claim 7, Badia discloses the reinforcement learning device according to claim 1, further comprising: a plurality of execution entities each including the setting unit, the actor, the learner, and the determination unit, wherein the actors of the plurality of execution entities share the updated state (see page 3 right-hand column third paragraph lines 1-4 (emphasis added): "Our agent is a deep distributed RL agent, in the lineage of R2D2 and NGU. As such, it decouples the data collection and the learning processes by having many actors feed data to a central prioritized replay buffer.", Fig. 2, Appendix E $"Actors", and Figs. 9-10).
As to claim 8, Badia discloses the reinforcement learning device according to claim 1, further comprising: a first execution entity including the setting unit, the actor, the learner, and the determination unit; and a second execution entity including the setting unit, the actor, the learner, and the determination unit, and in which the action group includes an action against the action group in the first execution entity, wherein the actors of the first execution entity and the second execution entity are configured to share the updated state, and the actor of the second execution entity is configured to select the action based on the value function, update the state of the analysis target, and calculate the reward such that the reward decreases as the updated state becomes the new state (page 3 right-hand column third paragraph lines 1-4 (emphasis added): "Our agent is a deep distributed RL agent, in the lineage of R2D2 and NGU. As such, it decouples the data collection and the learning processes by having many actors feed data to a central prioritized replay buffer.", Fig. 2, Appendix E "Actors", and Figs. 9-10).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20200258420 A1 (involved in a personalized and adaptive automated math learning system and method based on personal attributes, structured prediction, and reinforcement learning is disclosed. The personalization is achieved by data mining the personal attributes and creating competency clusters. The lesson plan and course is designed based on learners' competency levels to teach the subject matter in the shortest possible time. The adaptive automated machine learning method can change teaching methods and formats to become more interactive. After completion of the course, learners are expected to achieve expert competency).
US 20200074302 and US 11544556 B2 (involved in a plurality of individual learners. Each of the individual learners includes a planner configured to generate information for defining an operation of the operation subject corresponding to itself, and a reward deriver configured to derive a reward obtained by evaluating information to be evaluated including feedback information obtained from a simulator by inputting information based on the information for defining the operation of the operation subject to the simulator. The planner performs reinforcement learning based on the reward derived by the reward deriver, and at least two of the plurality of individual learners are different in the operations of the operation subject in which the reward derived by the reward deriver is maximized.)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEAN M CORRIELUS whose telephone number is (571)272-4032. The examiner can normally be reached Monday-Friday 6:30a-10p(Midflex).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ann J Lo can be reached at (571)272-9767. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users.
To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEAN M CORRIELUS/Primary Examiner, Art Unit 2159 July 14, 2026
Prosecution Insights