Prosecution Insights
Last updated: November 04, 2026
Application No. 18/653,330

METHOD, APPARATUS AND ELECTRONIC DEVICE FOR TRAINING A REINFORCEMENT LEARNING MODEL

Non-Final OA §101§102§112
Filed
May 02, 2024
Examiner
SCHNEE, HAL W
Art Unit
2129
Tech Center
2100 — Computer Architecture & Software
Assignee
Horizon Robotics Inc.
OA Round
1 (Non-Final)
84%
Grant Probability
Favorable
1-2
OA Rounds
3m
Est. Remaining
99%
With Interview

Examiner Intelligence

Grants 84% — above average
84%
Career Allowance Rate
517 granted / 613 resolved
+29.3% vs TC avg
Strong +22% interview lift
Without
With
+22.5%
Interview Lift
resolved cases with interview
Typical timeline
2y 9m
Avg Prosecution
20 currently pending
Career history
632
Total Applications
across all art units

Statute-Specific Performance

§101
10.2%
-29.8% vs TC avg
§103
39.5%
-0.5% vs TC avg
§102
15.5%
-24.5% vs TC avg
§112
30.3%
-9.7% vs TC avg
Black line = Tech Center average estimate • Based on career data from 613 resolved cases

Office Action

§101 §102 §112
Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claim Rejections - 35 USC § 112 The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Regarding Claim 1, it recites “adjusting policy parameters for a first reinforcement learning model based on the adjusted the reward model and the plurality of data information sets” (emphasis added). The phrase “the adjusted the reward model” is awkward, and its antecedent basis is absent or unclear. The claim previously recites “adjusting weight parameters for a reward model.” It does not explicitly recite an adjusted reward model. It is also awkward to repeat “the” in the present phrase (the adjusted the reward model), and it is unclear what such repetition means or represents. Since the claim previously recites adjusting policy parameters, the examiner will interpret “the adjusted the reward model” to mean “the adjusted policy parameters.” Regarding Claim 2, it recites “determining a corresponding image vector and a corresponding text vector” (line 4). It is unclear what the image vector and the text vector correspond to. Do they correspond to each other? Do they both correspond to a single, separate entity (such as a state image)? Do they each correspond to different entities? There is no way to determine which of these interpretations is intended. The claim further recites “the first state image” (lines 5-6). However, Claim 1 recites a plurality of first state images, so it cannot be determined which one of the state images is being referred to as “the first state image.” The claim further recites “adjusting weight parameters for the optimization submodel based on the first reward values corresponding to the plurality of the first state images” (last 2 lines). The term “the first reward values” in the plural is indefinite because the claim previously recites “a first reward value” in the singular; it is unclear what the additional reward values are/refer to. The phrase “corresponding to the plurality of the first state images” is unclear for two reasons: first, the correspondence is unclear, particularly because the number of reward values is indefinite. Second, the claim previously recited a (singular) first state image, so it cannot be determined what “the plurality of first state images” refers to. For all of these reasons, the examiner is unable to make a meaningful interpretation of the present claim for examination under prior art. Regarding Claim 3, it recites “the first state image” (line 2). As detailed above, claim 1 recites “a plurality of first state images,” so it cannot be determined which one is being referred to as “the first state image.” The claim also recites “determining the first reward value corresponding to the first state image . . .” (line 8). This limitation is indefinite because “the first state image” is indefinite. For these reasons, and the dependance of the present claim on claim 2, the examiner is unable to make a meaningful interpretation of the present claim for examination under prior art. Regarding Claims 4 and 5, they depend on claim 2 and therefore recite the indefinite limitations of claim 2, so the examiner is unable to make a meaningful interpretation of the present claims for examination under prior art. Regarding Claim 6, it recites “the adjusted the reward model” (line 2 and lines 4-5) and “the first state image” (line 4). These terms are indefinite as detailed for claim 1. The present claim also recites “ . . . based on the second reward values corresponding to the respective first state images, and the first state images, the actions, third reward values and second state images in the respective data information sets” (last 3 lines). This phrase is a run-on with several indefinite term. The term “the respective first state images” is indefinite because it is unclear what the first state images are respective to. It is unclear what “and the first state images” means in the context of the limitation, and how these might differ from “the respective first state images.” The term “the actions” lacks antecedent basis; there has been no previous mention of any actions. And it is unclear what “third reward values and second state images in the respective data information sets” refers to. Here too, the term “respective” is indefinite because the claim does not recite a clear correspondence among elements; it cannot be determined how the data information sets are “respective” to one another, or how they relate to third reward values and second state images. Which data information sets contain third reward values and which contain second state images? How do these figure into the second loss function? Since so many of the terms and limitations of the present claim are indefinite and unclear, the examiner is unable to make a meaningful interpretation of the present claim for examination under prior art. Regarding Claim 7, it recites several instances of “the first state image” and “the adjusted reward model,” which are indefinite as described for claim 1. The present claim also recites “determining a corresponding image vector and a corresponding text vector.” In a similar manner to the claims above, the correspondence is unclear—it cannot be determined what the image vector corresponds to and what the text vector corresponds to. Since so many of the terms and limitations of the present claim are indefinite and unclear, in addition to its dependence on claim 6, the examiner is unable to make a meaningful interpretation of the present claim for examination under prior art. Regarding Claim 8, it recites “the adjusted reward model,” which is indefinite as detailed above. Due to the present claim’s dependence on claim 6, the examiner is unable to make a meaningful interpretation of the present claim for examination under prior art. Regarding Claims 10 and 16, they recite elements similar to those of claim 1, so they are indefinite for the same reasons. Regarding Claims 11 and 17, they recite elements similar to those of claim 2, so they are indefinite for the same reasons. As with claim 2, the examiner is unable to make a meaningful interpretation of claims 11 and 17 for examination under prior art. Regarding Claims 12 and 18, they recite elements similar to those of claim 3, so they are indefinite for the same reasons. As with claim 3, the examiner is unable to make a meaningful interpretation of claims 12 and 18 for examination under prior art. Regarding Claim 13, it recites elements similar to those of claim 4, so it is indefinite for the same reasons. As with claim 4, the examiner is unable to make a meaningful interpretation of claim 13 for examination under prior art. Regarding Claim 14, it recites elements similar to those of claim 6, so it is indefinite for the same reasons. As with claim 6, the examiner is unable to make a meaningful interpretation of claim 14 for examination under prior art. Claim Rejections - 35 USC § 101 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 10-15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. the disclosed invention is inoperative and therefore lacks utility. The present claims recite “a non-transitory computer-readable storage medium, on which a computer program is stored, wherein the computer program is configured to implement a method for . . .” The claimed invention is merely a medium storing instructions, but it does not state that the instructions are executed by a processor or other computing component to perform the method. The instructions are only “configured” to perform a method. Claim Rejections - 35 USC § 102 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. Claims 1, 9-10, 15-16, and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wang, Yufei, et al. (“Rl-vlm-f: Reinforcement learning from vision language foundation model feedback,” arXiv preprint arXiv:2402.03681 (2024); hereinafter “Wang”). Regarding Claim 1, Wang teaches a method for training a reinforcement learning model (section 1—RL-VLM-F is a method for training a reinforcement learning model to automatically generate reward functions for agents), including: determining a task instruction for instructing an agent to perform a target task (section 1—a text description of a task goal is determined. The method in section 5 also describes a task goal description); determining a plurality of data information sets and a plurality of first state images generated during the performing of the target task by the agent (sections 5 and 5.1—the VLM responses and labels constitute data information sets, and the image observations are first state images); adjusting weight parameters for a reward model based on the task instruction and the plurality of first state images (section 5.1—training the reward model comprises adjusting weight parameters, as is well-known in the art. Appendix B.1 further describes training the reward model using a loss function, which is known to adjust weight parameters); adjusting policy parameters for a first reinforcement learning model based on the adjusted the reward model and the plurality of data information sets (section 5—the policy is updated based on the reward function, which is based on an adjusted reward model). Regarding Claim 10, Wang teaches a non-transitory computer-readable storage medium, on which a computer program is stored, wherein the computer program is configured to implement a method for training a reinforcement learning model (section 1—RL-VLM-F is a method for training a reinforcement learning model to automatically generate reward functions for agents. The experimental setup in section 6.1, including AI tools, clearly indicates use of a non-transitory computer-readable storage medium, on which a computer program is stored), including: determining a task instruction for instructing an agent to perform a target task (section 1—a text description of a task goal is determined. The method in section 5 also describes a task goal description); determining a plurality of data information sets and a plurality of first state images generated during the performing of the target task by the agent (sections 5 and 5.1—the VLM responses and labels constitute data information sets, and the image observations are first state images); adjusting weight parameters for a reward model based on the task instruction and the plurality of first state images (section 5.1—training the reward model comprises adjusting weight parameters, as is well-known in the art. Appendix B.1 further describes training the reward model using a loss function, which is known to adjust weight parameters); adjusting policy parameters for a first reinforcement learning model based on the adjusted the reward model and the plurality of data information sets (section 5—the policy is updated based on the reward function, which is based on an adjusted reward model). Regarding Claim 16, Wang teaches an electronic device, including: a processor; a memory configured for storing processor-executable instructions; wherein the processor is configured for reading the executable instruction from the memory, and executing the instruction to implement a method for training a reinforcement learning model (section 1—RL-VLM-F is a method for training a reinforcement learning model to automatically generate reward functions for agents. The experimental setup in section 6.1, including AI tools, clearly indicates use of a an electronic device including a processor and memory), including: determining a task instruction for instructing an agent to perform a target task (section 1—a text description of a task goal is determined. The method in section 5 also describes a task goal description); determining a plurality of data information sets and a plurality of first state images generated during the performing of the target task by the agent (sections 5 and 5.1—the VLM responses and labels constitute data information sets, and the image observations are first state images); adjusting weight parameters for a reward model based on the task instruction and the plurality of first state images (section 5.1—training the reward model comprises adjusting weight parameters, as is well-known in the art. Appendix B.1 further describes training the reward model using a loss function, which is known to adjust weight parameters); adjusting policy parameters for a first reinforcement learning model based on the adjusted the reward model and the plurality of data information sets (section 5—the policy is updated based on the reward function, which is based on an adjusted reward model). Regarding Claims 9, 15, and 20, Wang teaches the determining a plurality of data information sets and a plurality of first state images generated during the performing of the target task by the agent includes: performing the target task alternately by the first reinforcement learning model and a second reinforcement learning model based on the task instruction (sections 5.2 and Appendix A.1—SAC is a second reinforcement learning model that is used alternately with the VLM model); and determining the plurality of data information sets and the plurality of first state images during the performing of the target task by the agent (sections 5 and 5.1—image observations are determined and stored in a buffer during the performing of the target task by the agent). Conclusion The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. This art includes: Kimura (U.S. 2024/0086714) teaches adjusting weights of reinforcement learning rewards by training using state images Georgiev et al. (U.S. 2026/0187471) teaches an agent trained with reinforcement learning that uses state images and a text-based task request, using a reward neural network and a policy neural network to determine a reward for the agent Zolna et al. (U.S. 2023/0256593) teaches a robot/agent controller with a reward model that is trained/updated iteratively using state images Baumli, Kate, et al. (“Vision-language models as a source of rewards,” arXiv preprint arXiv:2312.09187 (2023)) teaches a visual language model that is used to determine a reward for reinforcement learning, using encoded vectors of text and images Any inquiry concerning this communication or earlier communications from the examiner should be directed to HAL W SCHNEE whose telephone number is (571) 270-1918. The examiner can normally be reached M-F 7:30 a.m. - 6:00 p.m. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at 303-297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /HAL SCHNEE/Primary Examiner, Art Unit 2129
Read full office action

Prosecution Timeline

May 02, 2024
Application Filed
Oct 19, 2026
Non-Final Rejection mailed — §101, §102, §112 (current)

Precedent Cases

Applications granted by this same examiner with similar technology

Patent 12783958
AUTOMATED DATA GENERATION FOR INTENT CLASSIFICATION MODELS
2y 9m to grant Granted Nov 03, 2026
Patent 12776975
MACHINE LEARNING ANOMALY DETECTION ON QUALITY OF SERVICE NETWORKING METRICS
3y 4m to grant Granted Oct 27, 2026
Patent 12770013
REINFORCEMENT LEARNING SYSTEM FOR MAINTENANCE DECISION MAKING
3y 8m to grant Granted Oct 20, 2026
Patent 12762478
METHOD OF GENERALIZED BALANCED FEW-SHOT LEARNING USING ONLY NOVEL DATA WITHOUT OLD DATA
3y 2m to grant Granted Oct 13, 2026
Patent 12755206
METHOD OF ANALYZING SOCIAL INFLUENCE BETWEEN INTERNET FORUMS AND APPARATUS FOR THE SAME
2y 9m to grant Granted Oct 06, 2026
Study what changed to get past this examiner. Based on 5 most recent grants.

Strategy Recommendation AI-generated — please review before filing

Get a prosecution strategy drawn from examiner precedents, rejection analysis, and claim mapping.
Typically takes 5-10 seconds — AI-generated, attorney review required before filing

Prosecution Projections

1-2
Expected OA Rounds
84%
Grant Probability
99%
With Interview (+22.5%)
2y 9m (~3m remaining)
Median Time to Grant
Low
PTA Risk
Based on 613 resolved cases by this examiner. Grant probability derived from career allowance rate.

Sign in with your work email

Enter your email to receive a magic link. No password needed.

Personal email addresses (Gmail, Yahoo, etc.) are not accepted.

Free tier: 3 strategy analyses per month