Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
2. This Office Action is sent in response to Applicant’s Communication received on 08/17/2026 for application number 18/535,928.
Response to Amendments
3. The Amendment filed 08/17/2026 has been entered. Claims 1-20 remain pending in the application.
Response to Arguments
Applicant argues that Raghavan fails to teach or suggest “determining from the received task features if the task related to the received task features has changed.” However, Examiner respectfully disagrees, Raghavan teaches a new task not previously learned by machine learning system 102 and that generative memory 104 determines a similarity score between the task represented by the input data and previously learned archetype tasks. Raghavan further teaches observing an input state in a current (possibly novel) task, calculating task similarity with each archetype task, and using such similarity in processing the current task. Thus, Raghavan determines from features of the received task whether the current task corresponds to, differs from, or is novel relative to previously learned tasks. It is also noted that the claim does not require the specific implementation described in Applicant’s Specification (para. [0021]), such limitations appearing in the Specification but not recited in the claim are not read into the claim.
Applicant argues that Raghavan fails to teach or suggest "automatically annotating or automatically storing feature characteristics of received task features, including differences between the features of the original task and the features of the changed task, to enable the learning system to more efficiently learn and/or perform at least the changed task." However, Examiner respectfully disagrees, Raghavan is not relied upon merely for determining a similarity score. Raghavan teaches automatically assigning training inputs to archetype tasks and clustering tasks based on similar inputs, similar output labels, similar data types, or other task characteristics. Raghavan further teaches encoding task inputs into latent space data, representing tasks having different inputs and outputs based on feature or embedding similarity across latent space, and using an embedding that provides concept separation by quantifying whether memories are similar or different so that such memories may subsequently be recalled. Raghavan further teaches that such task characteristic information is used to improve learning of new tasks. It is also noted that Applicant relies on Specification (para. [0036, 0037]) as describing an annotator module that may employ metadata and may annotate differences between a previous task and a changed task. However, claim 1 does not require that specific implementation.
Applicant argues that the Examiner does not cite Grove for teaching or suggesting "determining from the received task features if the task related to the received task features has changed" and "automatically annotating or automatically storing feature characteristics of received task features, including differences between the features of the original task and the features of the changed task, to enable the learning system to more efficiently learn and/or perform at least the changed task" and, as such, concedes that Grove fails to teach or suggest at least "determining from the received task features if the task related to the received task features has changed" and "automatically annotating or automatically storing feature characteristics of received task features, including differences between the features of the original task and the features of the changed task, to enable the learning system to more efficiently learn and/or perform at least the changed task" as claimed by at least the Applicant's independent claim 1. However, Examiner respectfully disagrees, the rejection is based on the combined teachings of Raghavan and Grove. A finding that one reference is relied upon for a particular limitation does not constitute a concession that another reference fails to disclose that limitation, nor is each reference individually required to disclose every limitation of the claim. Rather, the references are considered for what their collective teachings would have suggested to one of ordinary skill in the art.
As set forth in the rejection, Raghavan teaches receiving task defining input data and evaluating received task relative to previously learned tasks. Moreover, Grove provides additional explicit teaching of detecting a change condition from newly received data, Grove defines features as attributes, often represented as a vector. Grove further teaches evaluating a model using incoming/new training data and detecting model drift or change resulting from that new data. Grove further teaches that retraining is performed only when necessary and that new training data may otherwise be accumulated without retraining. Thus, Grove distinguishes between a detected change condition, in which adaptation/retraining is performed, and a no change/no degradation condition in which the existing model continues to be used. Thus, the combination teaches the claimed changed task and unchanged task branches.
Claim Rejections – 35 USC § 103
4. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5. Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Raghavan et al. (U.S. Patent Application Pub. No. US 20200302339 A1) in view of Grove et al. (U.S. Patent Application Pub. No. US 20160371601 A1).
Claim 1: Raghavan teaches a method for lifelong reinforcement learning (i.e. the techniques disclosed herein may enable more efficient training of machine learning systems, such as reinforcement learning systems. Furthermore, the techniques disclosed herein may enable a machine learning system to perform scalable, lifelong learning of solutions for new tasks the machine learning system has not previously been trained to solve, while reducing the occurrence of catastrophic forgetting (e.g., forgetting solutions to old tasks as a result of learning solutions to new tasks); para. [0009]), comprising:
receiving task features of a task to be performed (i.e. receiving, by an input device, training data defining one or more tasks, wherein the training data comprises a plurality of pairs of training inputs and training labels; para. [0011]);
communicating the task features to a learning system (i.e. Machine learning system 102 represents one or more computing devices to perform operations described herein to process input data 120 to determine one or more output labels 122 for system 100 to take to solve tasks for input data 120; para. [0027, 0034, 0037]), wherein the learning system learns and/or performs a task related to the received task features based on learning and/or performing similar previous tasks (i.e. generative memory 104 assigns, to each archetype task 106, a skill 116. In some examples, machine learning model 112 applies skill 116 to the input data 120 to generate labels 122. In some examples, machine learning model 112 applies skill 116 to the input data 120 to obtain an action sequence for solving the cluster of related tasks 108 represented by the corresponding archetype task 106 … generative memory 104 selects one or more archetype tasks 106 most similar to the task for input data 120 and obtains one or more skills 116 for the one or more archetype tasks 106 most similar to the task for input data 120; para. [0033, 0035, 0039]);
determining from the received task features if the task related to the received task features has changed (i.e. generative memory 104 may use the VAE to determine a similarity score between a task for input data 120 and the archetype task 106; para. [0045, 0046, 0096]);
the task has changed, communicating the task features of the changed task to the learning system (i.e. During wake phase 302, input device 202 receives input data 120 for a new task. The input data comprises a plurality of task inputs. Generative memory 104 selects, based on the plurality of task inputs, an archetype task 106 of the plurality of archetype tasks 106. Generative memory 104 obtains a skill 116 assigned to the archetype task 106 of the plurality of archetype tasks … The input data may define a new task not previously learned by machine learning system 102; para. [0054, 0076]), wherein the learning system learns and/or performs the changed task related to the received task features based on learning and/or performing similar previous tasks (i.e. Machine learning system 102 may train machine learning model 112 with both the plurality of inputs of input data and the plurality of auxiliary inputs of the auxiliary data in proportion to the similarity score between the task for the input data and each archetype task 106 to solve the task for the input data; para. [0047, 0055]);
at least one of automatically annotating or automatically storing feature characteristics of received task features (i.e. Generative memory 104 assigns one or more of the training inputs to each archetype task of a plurality of archetype tasks 106A-106N (hereinafter, “archetype tasks 106”); para. [0033, 0038, 0056]), including differences between the features of the original task and the features of the changed task (i.e. generative memory 104 may use the VAE to determine a similarity score between a task for input data 120 and the archetype task 106; para. [0046, 0047]), to enable the learning system to more efficiently learn and/or perform at least the changed task (i.e. the use of the auxiliary data in training machine learning model 112 may enable decreasing the time required for machine learning model 112 to learn new tasks; para. [0037, 0040]); and
the task has not changed, processing the task features of a current task by the learning system to learn and/or perform the current task (i.e. machine learning model 112 applies skill 116 to the input data 120 to obtain an action sequence for solving the cluster of related tasks 108 represented by the corresponding archetype task 106; para. [0035]).
Raghavan does not explicitly teach if the task has changed, communicating the task features of the changed task; if the task has not changed, processing the task features.
However, Grove teaches determining from if the task has changed (i.e. By treating new training data as test data, the system and/or method of the present disclosure in one embodiment can quickly evaluate a model's quality during updates, and detect when it degrades; para. [0024, 0045]); if the task has changed, communicating the task (i.e. it may be determined or judged as to whether the quality of the model is below a quality threshold set for retraining criteria. The processing at 308, 310 and 312 are performed in one embodiment responsive to determining that the model should be retrained, for example, the quality of the model is below the quality threshold, and thus retraining is needed; para. [0051]); if the task has not changed, processing the task features and/or perform the current task (i.e. If the quality of an old model on new examples did not degrade, then a middleware system or the like may continue to use the old model; para. [0043]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Raghavan to include the feature of Grove. One would have been motivated to make this modification because it improves the learning system by determining when incoming task data represents a meaningful change requiring adaptation and by continuing normal processing without retraining when no such change is detected.
Claim 2: Raghavan and Grove teach the method of claim 1. Raghavan further teaches wherein the learning system implements a wake or sleep learning process to learn and/or perform a task (i.e. system 100 may operate in a “wake” phase 302 and a “sleep” phase 304. During “wake” phase 302, system 100 may optimize machine learning model 112 to generate solutions to a novel input task, while during “sleep” phase 304, system 100 may optimize machine learning model 112 to consolidate and generate solutions for old tasks (e.g., compute solutions to archetype tasks) while system 100 is offline; para. [0053, 0054]).
Claim 3: Raghavan and Grove teach the method of claim 1. Raghavan further teaches comprising compressing stored information (i.e. Each VAE 802 comprises a set of encoders 806 that reduces a size of input task 120 to a smaller latent space; para. [0095]) to enable more information to be stored (i.e. the techniques disclosed herein may enable more efficient use of the memory of the machine learning system, thereby allowing for a reduction in the size of the memory of the machine learning system; para. [0040]).
Claim 4: Raghavan and Grove teach the method of claim 1. Raghavan further teaches wherein the learning system implements a generative model trained (i.e. Generative memory 104 trains a generative model to separate the training data into clusters of similar states and generates, for each cluster, an archetype task as an example of the cluster (1106); para. [0036, 0107]) to at least approximate a distribution of the learning (i.e. Generative memory 104 enables sampling from a joint distribution over all tasks and instances, which enables scalable training of machine learning model 112 while reducing catastrophic forgetting; para. [0096]) and/or performance of tasks (i.e. Machine learning system 102 may train machine learning model 112 with both the input data 120 (e.g., comprising a plurality of pairs of training inputs and training labels) and the auxiliary data (e.g., comprising a plurality of pairs of the auxiliary inputs and auxiliary labels) to solve the task for the input data; para. [0082]).
Claim 5: Raghavan and Grove teach the method of claim 1. Raghavan further teaches wherein a sleep phase of a learning process of the learning system is triggered based on the received task features (i.e. system 100 may operate in a “wake” phase and a “sleep” phase. During the “wake” phase, system 100 may optimize machine learning model 112 to generate solutions to a novel input task, while during the “sleep” phase, system 100 may optimize machine learning model 112 to consolidate and generate solutions for old tasks (e.g., compute solutions to archetype tasks) while system 100 is offline; para. [0091]).
Claim 6: Raghavan and Grove teach the method of claim 1. Raghavan further teaches wherein the received task features are pre-processed (i.e. the one or more encoders encode the plurality of inputs into latent space data; para. [0046]) before communicating the task features to the learning system (i.e. Observation module 118 consolidates input data 120 (e.g., a plurality of inputs paired with a plurality of labels 122 in the form of actions taken by machine learning model 112 in response to input data 120) and a resulting reward into consolidated state data 126 for processing by generative memory 104; para. [0051]) to configure the task features for use by the learning system (i.e. Generative memory 104 provides the consolidated state data to machine learning system 102 as advice 124 for training machine learning model 112 to obtain an action sequence 122 for solving input task 120; para. [0111]).
Claim 7: Raghavan and Grove teach the method of claim 6. Raghavan further teaches wherein the pre-processing comprises at least one of weighting task features (i.e. Machine learning system 102 may train machine learning model 112 with both the plurality of inputs of input data and the plurality of auxiliary inputs of the auxiliary data in proportion to the similarity score between the task for the input data and each archetype task 106 to solve the task for the input data; para. [0047, 0094]) and training machine learning models to identify task changes based on the received task features (i.e. Generative memory 104 may apply each VAE to a corresponding archetype task 106 to create a model of the cluster of tasks 108 represented by the archetype task 106. For example, at least one encoder of each VAE encodes inputs for a task (e.g., task inputs) into latent space data, and the at least one decoder decodes the latent space data into, e.g., a similarity score between the task for the inputs and the archetype task. Thus, generative memory 104 may use the VAE to determine a similarity between a task for input data 120 and the cluster of tasks 108 represented by the archetype task 106; para. [0077]).
Claim 8: Raghavan and Grove teach the method of claim 1. Raghavan further teaches wherein the annotated or stored differences (i.e. generative memory 104 may determine, for each archetype task 106 and using the VAE corresponding to the archetype task 106, a confidence in the similarity score between the task for the input data and archetype task 106; para. [0033, 0039, 0046]) are communicated to the learning system (i.e. Machine learning system 102 may train machine learning model 112 with both the plurality of inputs of input data and the plurality of auxiliary inputs of the auxiliary data in proportion to the similarity score between the task for the input data and each archetype task 106 to solve the task for the input data; para. [0047]) and used by the learning system to learn and/or perform subsequent tasks (i.e. Machine learning model 112 applies the one or more skills 116 to the plurality of inputs to obtain one or more output labels 122 for the plurality of inputs; para. [0032, 0038, 0039]).
Claim 9 is similar in scope to Claims 1 and is rejected under a similar rationale.
Raghavan teaches an apparatus for lifelong reinforcement learning (i.e. the techniques disclosed herein may enable more efficient training of machine learning systems, such as reinforcement learning systems. Furthermore, the techniques disclosed herein may enable a machine learning system to perform scalable, lifelong learning of solutions for new tasks the machine learning system has not previously been trained to solve, while reducing the occurrence of catastrophic forgetting (e.g., forgetting solutions to old tasks as a result of learning solutions to new tasks); para. [0009]), comprising: a processor; and a memory accessible to the processor, the memory having stored therein at least one of programs or instructions executable by the processor to configure the apparatus (i.e. Computation engine 230 includes machine learning system 102, observational module 118 and generative memory 104. Each of machine learning system 102, observational module 118, and generative memory 104 may represent software executable by processing circuitry 206 and stored on storage device 208, or a combination of hardware and software. Such processing circuitry 206 may include any one or more of a microprocessor; para. [0043]).
Claims 10-14 are similar in scope to Claims 2, 4-6, 8 and are rejected under a similar rationale.
Claim 15 is similar in scope to Claims 1 and is rejected under a similar rationale.
Raghavan teaches a system for lifelong reinforcement learning (i.e. the techniques disclosed herein may enable more efficient training of machine learning systems, such as reinforcement learning systems. Furthermore, the techniques disclosed herein may enable a machine learning system to perform scalable, lifelong learning of solutions for new tasks the machine learning system has not previously been trained to solve, while reducing the occurrence of catastrophic forgetting (e.g., forgetting solutions to old tasks as a result of learning solutions to new tasks); para. [0009]), comprising: a pre-processor module; an annotator module; a learning system; and an apparatus comprising a processor and a memory accessible to the processor, the memory having stored therein at least one of programs or instructions executable by the processor to configure the apparatus to (i.e. Computation engine 230 includes machine learning system 102, observational module 118 and generative memory 104. Each of machine learning system 102, observational module 118, and generative memory 104 may represent software executable by processing circuitry 206 and stored on storage device 208, or a combination of hardware and software. Such processing circuitry 206 may include any one or more of a microprocessor; para. [0027, 0043, 0044]).
Claim 16: Raghavan and Grove teach the system of claim 15. Raghavan further teaches wherein the learning system comprises at least one of a wake policy module, a sleep policy module, a memory module or a skill selector module and implements a wake-sleep learning process to learn and/or perform a task (i.e. system 100 may operate in a “wake” phase 302 and a “sleep” phase 304. During “wake” phase 302, system 100 may optimize machine learning model 112 to generate solutions to a novel input task, while during “sleep” phase 304, system 100 may optimize machine learning model 112 to consolidate and generate solutions for old tasks (e.g., compute solutions to archetype tasks) while system 100 is offline; para. [0032, 0035, 0053, 0054]).
Claims 17-18 are similar in scope to Claims 4, 5 and are rejected under a similar rationale.
Claim 19: Raghavan and Grove teach the system of claim 15. Raghavan further teaches comprising a replay buffer (i.e. one or more results of actions performed by machine learning model 112 may be stored in buffer 320 for consolidation into generative memory 104 during the sleep phase; para. [0056, 0074]) in which stored information to be used by the learning system to learn and/or perform tasks is stored (i.e. generative memory 104 enables the iterative consolidation of auxiliary data for learned tasks with the input data for unlearned tasks so as to increase the speed at which machine learning model 112 may learn solutions for new tasks while reducing the occurrence of catastrophic forgetting of solutions for previously-learned tasks when learning the solutions for new tasks; para. [0032, 0111]) in a compressed form (i.e. Each VAE 802 comprises a set of encoders 806 that reduces a size of input task 120 to a smaller latent space; para. [0095]).
Claim 20: Raghavan and Grove teach the system of claim 15. Raghavan further teaches wherein the pre-processor module, the annotator module and the learning system comprise lifelong learning systems (i.e. FIG. 1 is a block diagram illustrating an example system 100 for training machine learning model 112 using generative memory 104 in accordance with the techniques of the disclosure. Machine learning system 102 represents one or more computing devices to perform operations described herein to process input data 120 to determine one or more output labels 122 for system 100 to take to solve tasks for input data 120. Machine learning system 102 may include processing circuitry and memory as described in further detail with respect to FIG. 2. As described herein, generative memory 104 is capable of generating the auxiliary data from old tasks for use in training machine learning model 112 to obtain labels for new tasks for which machine learning model 116 has not previously been trained to enable machine learning system 102 to perform lifelong, scalable learning; para. [0027, 0046, 0051]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Zhang et al. (Pub. No. US 20200175362 A1), Methods, devices, and computer-readable media for multi-task based lifelong learning. A method for lifelong learning includes identifying a new task for a machine learning model to perform. The machine learning model trained to perform an existing task. The method includes adaptively training a network architecture of the machine learning model to generate an adapted machine learning model based on incorporating inherent correlations between the new task and the existing task. The method further includes using the adapted machine learning model to perform both the existing task and the new task.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAN TRAN whose telephone number is (303)297-4266. The examiner can normally be reached on Monday - Thursday - 8:00 am - 5:00 pm MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached on 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TAN H TRAN/Primary Examiner, Art Unit 2141
