Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to the Amendment filed on 11/25/2025. Claims 1-7, 11, and 12 are pending in the case. Claims 8-10 have been cancelled.
Response to Arguments
Applicant's amendments and arguments regarding the 35 U.S.C. § 101 rejections are persuasive. Accordingly, these rejections are hereby withdrawn.
Applicant's prior art arguments have been fully considered but are moot in view of the new grounds of rejection presented below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 11, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Such, F.P. et al. (2020, Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data, as filed in the IDS on 12/14/21), hereinafter Such, in view of Salimans et al. (2017, Evolution Strategies as a Scalable Alternative to Reinforcement Learning), hereinafter Salimans; Schmidhuber et al. (1999, Direct Policy Search and Uncertain Policy Evaluation), hereinafter Schmidhuber; and Goldman et al. (U.S. Pat. App. Pub. No. 2020/0150653), hereinafter Goldman.
Regarding claim 1:
Such directly teaches:
(b) training the strategy on the synthetic environment constructed depending on the disturbed synthetic environment parameters;
Page 3, Paragraph 1, Line 3+ teaches: “The learner is then trained on this synthetic data for a fixed number of inner-loop training steps with any optimizer,” The learner in Such is synonymous with the strategy of the claim. The synthetic data in question is disturbed though not specifically with random noise.
(c) determining rewards achieved by the trained strategy, which is applied on the real environment;
Page 17, Paragraph 2, Line 4 teaches: “Thus, it would create the initial state and then iteratively receive actions and generate the next state and optionally a reward” This recitation directly teaches the claim language.
Where Such fails to teach, Salimans teaches:
Providing synthetic environment parameters, a real environment, and a population of initialized strategies;
See Algorithm 2 (Parallelized Evolution Strategies) Page 3, section 2.1: The algorithm teaches in line 2 “Initialize: n workers with known random seeds, and initial parameters”. Workers being synonymous with strategies.
Page 6, Paragraph 6 teaches: “By perturbing in parameter space instead of action space, black box optimizers are naturally invariant to the frequency at which our agent acts in the environment. For MDP-based reinforcement learning algorithms, on the other hand, it is well known that frame skip is a crucial parameter to get right for the optimization to succeed [Braylan et al., 2005].” Here, the parameter space is synonymous with synthetic space and the action space is that of the real environment.
It would have been Prima Facie Obvious for one of ordinary skill in the art to combine the Salimans reference with the Such reference. Salimans describes the benefit of evolution strategies to be “highly parallelizable” thus increasing the speed of the overall process.
Repeating subsequent steps for a predetermined number of repetitions as a first loop: (1) carrying out for each strategy of the population of strategies subsequent steps as a second loop:(a) disturbing the synthetic environment parameters with random noise(b) training the strategy on the synthetic environment constructed depending on the disturbed synthetic environment parameters: andenvironment;
Page 3, Paragraph 2 teaches: “The resulting algorithm (1) repeatedly executes two phases: 1) Stochastically perturbing the parameters of the policy and evaluating the resulting parameters by running an episode in the environment, and 2) Combining the results of these episodes, calculating a stochastic gradient estimate, and updating the parameters.”
PNG
media_image1.png
136
539
media_image1.png
Greyscale
Salimans teaches of two phases or loops. Phase 1 teaches steps (a) and (b) via the stochastic perturbing of the parameters (a) and running an episode in the environment (b). Phase 2 is more relevant for later claims.
It would have been Prima Facie Obvious for one of ordinary skill in the art to integrate the use of loop iteration from Salimans with Such so that the highest scoring parameters are keep from one iteration to the next, ensuring a fully optimized algorithm.
wherein an actuator of the agent is controlled depending on determined actions by the outputted strategy,
Page 2, final paragraph, line 1 teaches: "will be the stochastic return provided by an environment, and Q will be the parameters of a deterministic or stochastic policy pQ describing an agent acting in that environment, controlled by either discrete or continuous actions".
It would have been Prima Facie Obvious for one of ordinary skill in the art to incorporate the use of Salimans 'agent' to indirect act within/control the environment.
Where Such fails to teach, Schmidhuber teaches:
(2) updating the synthetic environment parameters depending on the rewards of the trained strategies of the second; loop
Page 120, Column 2, Paragraph 4 teaches: "typically, in large (partially observable) environments, maximizing cumulative expected reinforcement within a limited life-time would be too ambitious a goal for any method. Instead designers of direct policy search methods are content with methods that can be expected to find better and better policies. But what does "better" mean in our context? Our agent’s obvious goal at checkpoint t is to generate POL-modifications accelerating reward intake: it wants to let R(T)-R(t) exceed the current average speed of re- T--t ward intake." Schmidhuber recites “generating modifications accelerating reward intake” Keeping in mind the broadest reasonable interpretation this recitation is synonymous with updating synthetic parameters depending on rewards. The goal of both the claim and the prior art is to gradually get better. By generating modifications that accelerate reward intake the prior art is accomplishing this. Updating the synthetic parameters based on the rewards is also a form of accomplishing this.
It would have been Prima Facie Obvious for one of ordinary skill in the art to utilize the method described in Schmidhuber; because, as described, the constant updating of parameters based on rewards leads to a better performing algorithm.
outputting the strategy of the trained strategies, which achieved a highest reward on the real environment or which achieved a highest reward during training on the synthetic environment.
Page 121, column 1, Paragraph 1 teaches: "Success Story Criterion (SSC) demands that each checkpoint in V marks the beginning of a long-term reward acceleration measured up to the current time t. SSC is achieved by the success story algorithm (SSA) which is invoked at every checkpoint: 1. WHILE SSC is not satisfied: Undo all POL modifications made since the most recent checkpoint in V, and remove that checkpoint from V. 2. Add the current checkpoint to V. ‘Undoing’ a modification means restoring the preceding POL -- this requires storing past values of POL components on a stack prior to modification. Thus each POL modification that survived SSA is part of a bias shift generated after a checkpoint marking a lifelong reward speed-up: the remaining checkpoints in V and the remaining policy modifications represent a 'success story.'" Schmidhuber recites a SSC wherein the “success story” is the output. Thus, if a higher rewarded criterion or SSC is output that is the one that is kept over the older one. Within this loop if the SSC shows a lower averaging reward, then the process is stopped and sometimes undone.
It would have been Prima Facie Obvious for one of ordinary skill in the art to consider outputting the highest performing strategy over any other to better accelerate learner training as SSC in Schmidhuber achieves.
Where Such fails to teach, Goldman teaches:
(2) wherein the agent is a manufacturing machine, an at least partially autonomous vehicle, and/or an access control system.
Paragraph 13, "autonomous vehicle"
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Such to include the autonomous vehicle techniques of Goldman to improve the safety and efficiency (see Goldman at paragraph 22).
Regarding Claim 2:
Schmidhuber teaches:
The method according to claim 1, wherein the updating of the synthetic environment parameters is carried out by stochastic gradient estimate based on a weighted sum of the determined rewards of the trained strategies in the second loop.
Page 1, column 2, paragraph 3, line 1 teaches “Uncertainty. Policy evaluation by direct (elitist) methods is straight-forward in simulated environments that allow for separating the search phase into repeatable, deterministic trials such that each trial with a given policy yields the same reward. In more realistic situations, however, sources of uncertainty arise: (1) The policy may be stochastic, i.e., the learner’s actions are selected nondeterministically according to probability distributions conditioned on the policy. Stochastic policies are widely used to prevent learners from getting stuck. Results of policy evaluations, however, will then vary from trial to trial. (2) Environment and reward may be stochastic. And even if the environment is deterministic it may appear stochastic from an individual learner’s perspective, due to partial observability.” Here Schmidhuber refers to a “simulated environment” this is synonymous with a synthetic environment. As the “learner” recited in the prior art is synonymous with the strategy of the claim language, it can be read that the learner parameters are updated via stochastic gradient estimate.
It would have been Prima Facie Obvious for one of ordinary skill in the art to combine Schmidhuber with Such in that Schmidhuber specifies that stochastic policies are widely used to keep learners from becoming stuck.
Regarding Claim 3:
Salimans teaches:
The method according to claim 1, wherein the training of the strategies of the population of strategies are carried out in parallel.
Page 3, paragraph 3, line 1 teaches: “ES is well suited to be scaled up to many parallel workers”. Evolution strategies (ES) being suited to and processed in parallel is directly related to the claim language of training the strategies of the population in parallel.
Page 3, paragraph 4, line 1 teaches: “A simple parallel version of ES is given in Algorithm 2.” Here is the above-mentioned algorithm highlighting the parallel training of ES.
PNG
media_image2.png
225
538
media_image2.png
Greyscale
It would have been Prima Facie Obvious for one of ordinary skill in the art to combine Salimans with Such to utilize parallel processing, as discussed earlier this has a direct positive impact on the processing speed.
Regarding Claim 4:
Such directly teaches:
The method according to claim 1, wherein each strategy is randomly initialized before training the strategy on the synthetic environment.
Page 3, paragraph 4, line 3 teaches:” we choose a new learner architecture according to a predefined set and randomly initialize” Such uses the term “learner” synonymously with the claim language of “strategy. Thus, this is a direct teaching of the claim language.
Regarding Claim 5:
Schmidhuber teaches:
The method according to claim 1, wherein the step of training the strategy is terminated if a change of a moving average of cumulative rewards over a given number previous episodes of the training is smaller than a given threshold
Page 2, column 1, paragraph 2, line 7 teaches: “A change is considered "good" as long as the average reward per time since its creation exceeds the corresponding ratios for previous "good" changes. Changes that eventually turn out to be "bad" get undone by+J24 an efficient backtracking scheme called the success-story algorithm (SSA)”. Schmidhuber uses the term “good” to mean better in that the average reward exceeds the previous “good” changes. “Bad” is described as the average reward that fails to exceed or meet the previous “good” result. “Bad” changes are thus stopped and/or undone by another algorithm.
It would have been Prima Facie Obvious for one of ordinary skill in the art to combine Schmidhuber with Such so that the evaluation is stopped and/or reversed to prevent time loss with ‘bad’ changes.
Regarding Claim 6:
Such directly teaches:
The method according to claim 1, wherein a Hyperparameter Optimization is carried out to optimize hyperparameters of a training method for the training of the strategies and/or of an optimization method for updating the synthetic environment parameters
Page 3, paragraph 3, line 5 teaches: “The gradients of the generator parameters are computed w.r.t. to the meta-loss to update the generator (5). Both a learned curriculum and weight normalization substantially improve GTN performance. (b) Weight normalization improves meta-gradient training of GTNs, and makes the method much more robust to different hyperparameter settings. Each boxplot reports the final loss of 20 mns obtained during hyperparameter optimization with Bayesian Optimization (lower is better). (c) shows a comparison between GTNs with different types of curricula. The GTN method with the most control over how samples are presented performs the best.” Such teaches the direct language of the claim within this excerpt.
Regarding Claim 7:
The method according to claim 1, wherein the synthetic environment is represented by a neural network, wherein the synthetic environment parameters are weights of the neural network.
It is taught at least in the abstract: “This paper investigates the intriguing question of whether we can create learning
algorithms that automatically generate training data. learning environments, and curricula in order to help AI agents rapidly learn. We show that such algorithms are possible via Generative Teaching Networks (GTNs), a general approach that is, in theory, applicable to supervised, unsupervised, and reinforcement learning, although our experiments only focus on the supervised case. GTNs are deep neural networks that generate data and or training environments that a learner (e.g. a freshly initialized neural network),” Such teaches the direct language of the claim within the abstract in that the learner is also a strategy and is called a neural network. It is common knowledge within the art that parameters and weights are synonymous.
Regarding claims 11 and 12:
But for the additional elements claims 10 through 12 directly read upon claim 1 and can be evaluated and rejected to under the same references.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Casey R. Garner whose telephone number is 571-272-2467. The examiner can normally be reached Monday to Friday, 8am to 5pm, Eastern Time.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on 571-270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center and Private PAIR to authorized users only. Should you have questions about access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/Casey R. Garner/Primary Examiner, Art Unit 2123
Prosecution Insights