Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This communication is in reply to 10/29/2024. Claimed priority is granted from foreign Application NumberJP2022-076195 filed on 05/02/2022.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 06/30/2025 and 11/20/2024 was filed in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-5, and 7-8 are rejected under 35 U.S.C. 103 as being unpatentable over TAYLOR et al., US 20190236455 A1, in view of Zaremoodi et al., US 20210304074 A1 (hereinafter Zare).
Regarding claim 1, The combination of TAYLOR and Zare teaches the invention as claimed. It discloses: A learning device comprising:
a processing unit that performs learning of a learning model of an agent (TAYLOR, 0068 supplies processor/memory); 0108-0109 supplies the action model and explicit hyperparameters, including grid search),
wherein the learning model includes a hyperparameter (TAYLOR, 0107-0110, teaches demonstration-action classification. Modification can be directed to applying Zare’s optimization loop, 0106, 0109, 0122-0123, to select settings based on imitation performance),
the learning includes imitation learning and reinforcement learning (TAYLOR, 0058-0061 supplies reward-directed learning. Modification can be directed to applying Are’s optimization to the reinforcement-learning settings using reward performance as the objective),
the imitation learning causing the processing unit to perform learning of the hyperparameter of the learning model (TAYLOR, 0108-0109 first, then Zare, 0109 and 0122 expressly teach the specified order)
such that the agent executes a predetermined action in a predetermined state under a predetermined environment (TAYLOR, 0130 checks classification-loss convergence before transfer and Zare, 0116, 0123 outputs the evaluated, optimized model and settings).
the reinforcement learning causing the processing unit to perform learning of the
hyperparameter of the learning model such that a reward assigned to the agent under the
predetermined environment is maximized (TAYLOR, 0110-0112, and 0130 transfers the pretrained model into reinforcement learning, including an all-layer transfer alternative), and
the processing unit executes:
a step of setting a parameter value of the hyperparameter of the learning model (TAYLOR, 0109 and Zare, 0109);
a step of executing the imitation learning by using the learning model after the
setting (TAYLOR, 0107-0109);
a step of evaluating the learning model after the imitation learning and extracting
the evaluated learning mode (TAYLOR , 0130; Zare, 0116, and 0122-0123); and
a step of executing the reinforcement learning by using the extracted learning
model (TAYLOR, 0110-0112 and 0130).
Accordingly, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify TAYLOR’s demonstration-based pretraining process using Zare’s hyperparameter tuning and evaluation procedure to select an evaluated imitation-trained model before reinforcement learning. TAYLOR permits hyperparameter grid search (0109) and uses pretrained model to initiate reinforcement learning (0130), while Zare evaluates candidate configurations and outputs an optimized model (0122-0123). The modification would predictably work because TAYLOR because TAYLOR already transfers pretrained parameters into its reinforcement-learning network (0110-0112).
Likewise, regarding claims 3-5, and 7-8
Claim 3. (Currently Amended) The learning device according to Claim 1,
wherein, in the step of extracting the learning model, in a case where an evaluation result
of the learning model satisfies an evaluation index, one learning model that satisfies the
evaluation index is extracted (Zare, 0106, 0116, and 0123 teaches satisfying constraints and outputting an optimized model. Select one passing candidate for TAYLEO’s subsequent training).
Claim 4. (Currently Amended) The learning device according to Claim 1,
wherein, in the step of setting the parameter value, in a case where an evaluation result in
the step of evaluating the learning model does not satisfy an evaluation index, a new parameter
value is set (Zare, 0122 expressly changes hyperparameter values and continues training when a constraint fails or performance is inadequate).
Claim 5. (Currently Amended) The learning device according to Claim 1,
wherein, in the step of extracting the learning model, evaluation based on a matching
degree between a predetermined state and a predetermined action that are to be trained in the
imitation learning under a predetermined environment and a predetermined state and a
predetermined action in the learning model under the predetermined environment is executed (TAYLOR, 0107-0110, and 0130 evaluates imitation through action-classification loss. Use that loss on demonstration states as Zare’s selection metric: compare demonstrated actions against model outputs for those states).
Claim 7. (Original) A learning method of performing learning of a learning model of an agent
by using a learning device,
wherein the learning model includes a hyperparameter,
the learning includes imitation learning and reinforcement learning, the imitation learning
causing the learning device to perform learning of the hyperparameter of the learning model such
that the agent executes a predetermined action in a predetermined state under a predetermined
environment, the reinforcement learning causing the learning device to perform learning of the
hyperparameter of the learning model such that a reward assigned to the agent under the predetermined environment is maximized, and the learning method causes the learning device to execute:
a step of setting a parameter value of the hyperparameter of the learning model;
a step of executing the imitation learning by using the learning model after the
setting;
a step of evaluating the learning model after the imitation learning and extracting
the evaluated learning model; and
a step of executing the reinforcement learning by using the extracted learning
model.
This claim provides the same operations and sequence mapped above for claim 1, performed as a computer-implemented method. The same references, motivation and reason to combine used for the rejection of claim 1 is also valid for this claim.
Claim 8. A learning program for performing learning of a learning model of an agent
by using a learning device,
wherein the learning model includes a hyperparameter,
the learning includes imitation learning and reinforcement learning, the imitation learning
causing the learning device to perform learning of the hyperparameter of the learning model such
that the agent executes a predetermined action in a predetermined state under a predetermined
environment, the reinforcement learning causing the learning device to perform learning of the
hyperparameter of the learning model such that a reward assigned to the agent under the
predetermined environment is maximized, and the learning program causes the learning device to execute:
a step of setting a parameter value of the hyperparameter of the learning model;
a step of executing the imitation learning by using the learning model after the
setting;
a step of evaluating the learning model after the imitation learning and extracting
the evaluated learning model; and
a step of executing the reinforcement learning by using the extracted learning
model.
This claim provides the same operations and sequence mapped above for claim 1, implemented by program instructions expressed in par. 0172-0173. The same references, motivation and reason to combine used for the rejection of claim 1 is also valid for this claim.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over TAYLOR and ZARE as applied to claims 1, 3-5 and 7-8 above, and further in view of Pietquin et al, US 2020/0151562 A1.
Regarding claim 2, the combination of TAYLOR-Zare teaches the learning device of claim 1, but fails to specifically disclose the detail of a learning device according to Claim 1, wherein the environment is an environment in which the reward is sparse. This teaching is well-known in the art as evidenced Pietquin. In the same field of endeavor, Pietquin expressly discloses sparse feedback, including reward only upon task completion (Pietquin, 0011, 0019, 0044 and 0054).
It would have been obvious to apply TAYLOR’s pretraining process, as modified by Zare, to Pietquin’s sparse-reward environment, where reward may remain zero until task completion (Pietquin, 0044, 0054)/ Pietquin explains that dense reward sharing can produce undesirable behavior 0044, providing a reason to use task-completion rewards. TAYLOR’s demonstration-based initialization would provide a suitable starting model for subsequent reward-based learning 0130, predictably supporting training where useful reward feedback is infrequent. By this rationale, claim 2 is rejected
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over TAYLOR and ZARE as applied to claims 1, 3-5 and 7-8 above, and further in view of Meng et al., US 2020/0334538 A1.
Regarding claim 6, the combination of TAYLOR-Zare teaches the learning device of claim 1, but fails to specifically disclose the detail of learning device according to Claim 5, wherein, as the matching degree, a KL divergence is used. This teaching is well-known in the art as evidenced Meng. In the same field of endeavor, Meng expressly discloses KL output-distribution comparison and convergence testing (0021, 0026, 0054-0055, 0059-0061).
It would have been obvious to modify TAYLOR’s imitation-model evaluation, as combined with Zare, to use Meng’s KL-divergence comparison between reference and learned output distributions (Meng, 0026 and 0054). A skilled artisan would use that measure to quantify how closely the model’s action probabilities match the demonstration behavior before selecting it for reinforcement learning. The substitution would predictably work because TAYLOR already uses probabilistic action outputs and cross-entropy loss (0108-0110), and meng explains that minimizing KL divergence with a fixed reference distribution is equivalent to minimizing the corresponding cross-entropy objective (0055). By this rationale, claim 6 is rejected.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 8 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because its opening “A learning program” appears to claim software itself without a storage medium. If construed as a program per se, it fails the statutory-category requirement. Recasting it as a non-transitory computer-readable storage medium storing the instructions would address that issue.
CONCLUSION
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jude Jean-Gilles whose telephone number is 571-272-3914. The examiner can normally be reached on Mon-Fri, from 9:00AM-5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tonia Dollinger can be reached on 571-272-4170. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JUDE JEAN GILLES/Primary Examiner, Art Unit 2459
November 10, 2026