DETAILED ACTION
This is a non-final, first office action on the merits. Claims 1-13 are pending. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant is claiming Foreign Priority to Foreign Applications KR10-2024-0048666 filed on 04/11/2024.
Claim Objections
Claims 1-2, 4-5, 7-9, and 11-12 are objected to because of the following informalities:
Claims 1-2, 4-5, 7-9, and 11-12 “previoulsy” being misspelled and it should recite previously. Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations (See claim 1) that does not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “a data management unit, an offline model training unit, and an online model training unit” in claim 1.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
The Specification describes:
Para 0028 Fig. 1, the apparatus according to the present embodiment may include a data management unit 100, an offline model training unit 102, and an online model training unit 104.
According to the Specification, the generic place holders could be by hardware, software, and/or any combination of hardware, software. The Specification does not specifically point out the structure. The generic place holders are not modified by significant structure, material, or acts of performing the claimed function. Therefore the limitations of claim 1 listed above have invoked 112(f).
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-6 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim 1 has a number of limitations that invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. Claim 1 does NOT state that a data management unit, an offline model training unit, and an online model training unit stored in memory and executed by the processor. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. Para 0028 Fig. 1, the apparatus according to the present embodiment may include a data management unit 100, an offline model training unit 102, and an online model training unit 104. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claims 2-6 are rejected for having the same deficiencies as those set forth with respect to the claims that they depend from, independent claim 1.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-13 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. Specifically, claims 1-13 are directed to an abstract idea without additional elements amounting to significantly more than the abstract idea.
With respect to Step 2A Prong One of the framework, claims 1 and 7-8 recite an abstract idea. Claims 1 and 7-8 include “collecting and processing data in advance; training an offline policy and an offline state-action value function using a previoulsy collected dataset that includes a state, an action, a next state, a reward, and an accumulated reward collected; and performing to update parameters of the offline policy using an online dataset that includes action information determined based on state information acquired through interaction with the offline policy and an environment, state information at a next time point according to the action information, and the reward, and the previously collected dataset”.
The limitations above recite an abstract idea under Step 2A Prong One. More particularly, the elements above recite mental processes-concepts performed in the human mind (including an observation, evaluation, judgment, opinion) and mathematical calculations because the elements describe a process for collecting, observing, and processing data. As a result, claims 1 and 7-8 recite an abstract idea under Step 2A Prong One.
Claims 2-6 and 9-13 further describe the process for collecting, observing, and processing data. As a result, claims 2-6 and 9-13 recite an abstract idea under Step 2A Prong One for the same reasons as stated above with respect to claims 1 and 7-8.
With respect to Step 2A Prong Two of the framework, claims 1 and 7-8 do not include additional elements that integrate the abstract idea into a practical application. Claims 1 and 7-8 include additional elements that do not recite an abstract idea under Step 2A Prong One. The additional elements of claims 1 and 7-8 include a processor, a memory, a data management unit, an offline model training unit, an online model training unit, an offline policy network, and reinforcement learning. When considered in view of the claim as a whole, the additional elements do not integrate the abstract idea into a practical application because the additional computing elements are generic computing elements that are merely used as a tool to perform the recited abstract idea. As a result, claims 1 and 7-8 do not include additional elements that integrate the abstract idea into a practical application under Step 2A Prong Two.
Claim 11 does not include any additional elements beyond those recited with respect to claims 1 and 7-8. As a result, claim 11 does not include additional elements that integrate the abstract idea into a practical application under Step 2A Prong Two for the same reasons as stated above with respect to claims 1 and 7-8.
Claims 2-6, 9-10, and 12-13 include additional elements that do not recite an abstract idea under Step 2A Prong One. The additional elements of claims 2-6, 9-10, and 12-13 include an offline policy network, data management unit, offline model training unit, and online model training unit. When considered in view of the claims as a whole, the additional elements do not integrate the abstract idea into a practical application because the additional computing elements do no more than generally link the use of the recited abstract idea to a particular technological environment. As a result, claims 2-6, 9-10, and 12-13 do not include additional elements that integrate the abstract idea into a practical application under Step 2A Prong Two.
With respect to Step 2B of the framework, claims 1 and 7-8 do not include additional elements amounting to significantly more than the abstract idea. As noted above, claims 1 and 7-8 include additional elements that do not recite an abstract idea under Step 2A Prong One. The additional elements of claims 1 and 7-8 include a processor, a memory, a data management unit, an offline model training unit, an online model training unit, an offline policy network, and reinforcement learning. The additional elements do not amount to significantly more than the abstract idea because the additional computing elements are generic computing elements that are merely used as a tool to perform the recited abstract idea. Further, looking at the additional elements as an ordered combination adds nothing that is not already present when considering the additional elements individually. As a result, independent claims 1 and 7-8 do not include additional elements that amount to significantly more than the abstract idea under Step 2B.
Claim 11 does not include any additional elements beyond those recited with respect to claims 1 and 7-8. As a result, claim 11 does not include additional elements that amount to significantly more than the abstract idea under Step 2B for the same reasons as stated above with respect to claims 1 and 7-8.
Claims 2-6, 9-10, and 12-13 include additional elements that do not recite an abstract idea under Step 2A Prong One. The additional elements of claims 2-6, 9-10, and 12-13 include an offline policy network, data management unit, offline model training unit, and online model training unit. The additional elements do not amount to significantly more than the abstract idea because the additional computing elements do no more than generally link the use of the recited abstract idea to a particular technological environment. Further, looking at the additional elements as an ordered combination adds nothing that is not already present when considering the additional elements individually. As a result, claims 2-6, 9-10, and 12-13 do not include additional elements that amount to significantly more than the abstract idea under Step 2B.
Therefore, the claims are directed to an abstract idea without additional elements amounting to significantly more than the abstract idea. Accordingly, claims 1-13 are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-13 are rejected under 35 U.S.C. 103 as being unpatentable over Osa et al. (US Pub No. 2024/0005206) (hereinafter Osa et al.) in view of Zhang et al. (US Pub No. 2023/0289623) (hereinafter Zhang et al.).
Regarding claims 1 and 7-8, Osa discloses an offline pre-training and online fine-tuning apparatus based on reinforcement learning (see Osa, paras [0078]- [0079]), comprising:
a data management unit collecting and processing data for offline reinforcement learning in advance (see Osa, para [0004], wherein the goal of offline reinforcement learning is to learn an optimal policy from a dataset collected; and para [0007], wherein a policy obtained by known offline reinforcement learning algorithms highly depends on the quality of a given dataset);
an offline model training unit training an offline policy network and an offline state-action value function network using a previoulsy collected dataset that includes a state, an action, a next state, a reward, and an accumulated reward collected by the data management unit (see Osa, paras [0087]-[0090], wherein Q function QTC(s, a) is the expected value of the reward when starting from a state s, taking an action a, and following the policy re under the given Markov decision process. Offline reinforcement learning assumes a dataset of the following expression (5) consisting of states, actions, and rewards collected by an unknown policy.…..offline reinforcement learning is formulated as follows. Given a dataset D (of expression (5)), the goal of a learning process is to obtain, without interacting with an environment, a policy re that maximizes a reward obtained by interacting with the environment…In offline reinforcement learning, the expected value of the reward is evaluated with respect to states stored in a given dataset; and para [0058], wherein Fig. 4, the dataset consists of tuples of four elements, for example, state s, action a, next state s', and reward r); and
update parameters of the offline policy network using an online dataset that includes action information determined based on state information acquired through interaction with the offline policy network and an environment, state information at a next time point according to the action information, and the reward, and the previously collected dataset (see Osa, para [0017], wherein the latent variable action update step may include updating a first model by updating an actor and a posterior distribution to maximize; paras [0003]-[0006], wherein a reinforcement learning agent interacts with an environment during a learning process. Reinforcement learning generates a prediction model, for example, using a plurality of input parameters; paras [0087]-[0090], wherein Q function QTC(s, a) is the expected value of the reward when starting from a state s, taking an action a, and following the policy re under the given Markov decision process. Offline reinforcement learning assumes a dataset of the following expression (5) consisting of states, actions, and rewards collected by an unknown policy.…..offline reinforcement learning is formulated as follows. Given a dataset D (of expression (5)), the goal of a learning process is to obtain, without interacting with an environment, a policy re that maximizes a reward obtained by interacting with the environment…In offline reinforcement learning, the expected value of the reward is evaluated with respect to states stored in a given dataset; and para [0058], wherein Fig. 4, the dataset consists of tuples of four elements, for example, state s, action a, next state s', and reward r).
Osa et al. fails to explicitly disclose an online model training unit performing fine-tuning to update parameters of the offline policy network using an online dataset that includes action information determined based on state information acquired through interaction with the offline policy network.
Analogous art Zhang discloses an online model training unit performing fine-tuning to update parameters of the offline policy network using an online dataset that includes action information determined based on state information acquired through interaction with the offline policy network and an environment, state information at a next time point according to the action information (see Zhang, para [0046], wherein Several techniques for autonomous learning 200 can be used, including continuous learning, transfer learning, reinforcement learning, fine-tuning techniques, boosting, and so on, in accordance with the desired implementation. Depending on the desired implementation, continuous learning could be done in real-time or based on a predefined schedule. The re-trained models are evaluated against reserved test data; if the models have better performance, they are deployed to the online system at 211, otherwise, the current versions of the models remain on the online system; para [0058], wherein a policy defines how the learning agent behaves at a given time in the current state of the environment. For this example, it defines a mapping from perceived current states of the plant to actions to be taken to optimize the yields; para [0085], wherein the approach and associated parameters to deploy the offline models into the online system as shown in 141; automating the application of prescriptive actions to the online system as shown in 142; and para [0031], wherein the autonomous component focuses on continuous learning based on new data and feedback (i.e., state information at a next time makes up the core feedback loop of how an AI agent learns to interact with its environment), and can include reinforcement learning, transfer learning, automatons learning (e.g., knowledge store, knowledge graph, user behavior, etc.), cohort learning through several sub-systems (or domains), and automating the continuous learning based on the new data/evidence and/or error/feedback. The automation and autonomous components are integrated together to support automating the application of prescriptive actions, retrieve new data, error and feedback, re-train the predictive and prescriptive models, and make intelligent decisions on the prescriptive actions on controlling the systems to work more efficiently and effectively, by leveraging the autonomous learning capabilities).
Osa directed to a system for optimizing policy from a dataset collected. Zhang directed to online fine-tuning using reinforcement learning. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Osa, regarding the System for Learning Device, to have included an online model training unit performing fine-tuning to update parameters of the offline policy network using an online dataset that includes action information determined based on state information acquired through interaction with the offline policy network and an environment, state information at a next time point according to the action information because both inventions teach steadily improving during the training process. Further, the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claims 2 and 9, Osa discloses the offline pre-training and online fine-tuning apparatus of claim 1, wherein the offline model training unit performs adaptation for constructing a new state-action value function network matching the offline policy network (see Osa, para [0058], wherein Fig. 4, the dataset consists of tuples of four elements, for example, state s, action a, next state s (i.e., current condition and new inputs), and reward r; and para [0153], wherein V2AE which is the method of the present embodiment have achieved performance comparable to TD3+BC and EDAC which are the latest offline reinforcement learning methods in the Mujoco tasks.), and
Osa et al. fails to explicitly disclose the online model training unit updates parameters of the new state-action value function network according to the offline policy network and the adaptation using the online dataset and the previoulsy collected dataset.
Analogous art Zhang discloses the online model training unit updates parameters of the new state-action value function network according to the offline policy network and the adaptation using the online dataset and the previoulsy collected dataset (see Zhang, para [0046], wherein Several techniques for autonomous learning 200 can be used, including continuous learning, transfer learning, reinforcement learning, fine-tuning techniques, boosting, and so on, in accordance with the desired implementation. Depending on the desired implementation, continuous learning could be done in real-time or based on a predefined schedule. The re-trained models are evaluated against reserved test data; if the models have better performance, they are deployed to the online system at 211, otherwise, the current versions of the models remain on the online system; para [0058], wherein a policy defines how the learning agent behaves at a given time in the current state of the environment. For this example, it defines a mapping from perceived current states of the plant to actions to be taken to optimize the yields; para [0085], wherein the approach and associated parameters to deploy the offline models into the online system as shown in 141; automating the application of prescriptive actions to the online system as shown in 142; para [0093], wherein after the offline models are deployed into the online system, the offline models can be improved based on historical data in parallel and then seamlessly replace the online models if performance improvement from offline model (compared to current version of online models) meets the predefined criteria, without interrupting the normal operation of the online systems; and para [0031], wherein the autonomous component focuses on continuous learning based on new data and feedback (i.e., state information at a next time makes up the core feedback loop of how an AI agent learns to interact with its environment), and can include reinforcement learning, transfer learning, automatons learning (e.g., knowledge store, knowledge graph, user behavior, etc.), cohort learning through several sub-systems (or domains), and automating the continuous learning based on the new data/evidence and/or error/feedback. The automation and autonomous components are integrated together to support automating the application of prescriptive actions, retrieve new data, error and feedback, re-train the predictive and prescriptive models, and make intelligent decisions on the prescriptive actions on controlling the systems to work more efficiently and effectively, by leveraging the autonomous learning capabilities).
One of ordinary skill in the art would have recognized that applying the known technique of Zhang would have yielded predictable results and resulted in an improved system for the same reasons as stated above with respect to claim 1.
Regarding claims 3 and 10, Osa discloses the offline pre-training and online fine-tuning apparatus of claim 2, wherein the online model training unit initializes all or at least a part of the parameters of the new state- action value function according to the offline policy network and the adaptation (see Osa, paras [0003]-[0006], wherein a reinforcement learning agent interacts with an environment during a learning process. Reinforcement learning generates a prediction model, for example, using a plurality of input parameters; para [0014], wherein z is the discrete latent variable, z' is a next discrete latent variable, s is a state, s' is a next state, Qw is an estimate of a Q value parameterized by a vector w, y is a target value, r is a reward in learning, y is a discount factor, 0 is a vector representing parameters of a policy, <p is a vector representing parameters of a model of a posterior distribution, (z~)' is the next discrete latent variable that has been estimated, f" is a function that quantifies performance of a policy n is a variational lower bound, and a is an action, the estimation step; and para [0087], wherein a Q function QTC(s, a) is the expected value of the reward when starting from a state s, taking an action a, and following the policy re under the given Markov decision process. Offline reinforcement learning assumes a dataset of the following expression (5) consisting of states, actions, and rewards collected by an unknown policy).
Regarding claims 4 and 11, Osa discloses the offline pre-training and online fine-tuning apparatus of claim 1, wherein the data management unit collects observation information and action information through observation equipment (see Osa, para [0079], wherein the learning device 1 determines a latent variable that maximizes the action value for an observed state using the trained first model 131), and
stores the observation information, the action information, the observation information at the next time point, and the reward as the previoulsy collected dataset (see Osa, paras [0182]-[0183], wherein a storage device such as a hard disk provided in a computer system. The "computer readable recording medium" includes one that holds the program for a certain period of time, like a volatile memory; [0079], wherein the learning device 1 determines a latent variable that maximizes the action value for an observed state using the trained first model 131; and paras [0068]-[0070], wherein (Step S11) The acquisition unit 11 acquires a dataset in advance and stores the acquired dataset in the storage unit 12 (acquisition step). [0070] (Step S13) The processing unit 17 extracts some tuples of state s, action a, next state s', and reward r from the dataset).
Osa et al. fails to explicitly disclose matches the observation information and the action information with observation information at a next time point; calculates a reward using the matched observation information, action information, and observation information at the next time point.
Analogous art Zhang discloses matches the observation information and the action information with observation information at a next time point (see Zhang, para [0038], wherein determine errors and/or feedback based on the comparison of predictions along with associated prescriptive actions to the observed incidents and the action items. With the real-time data and current version of the model, the information is utilized to re-train and improve the predictive and prescriptive models. The re-trained models are evaluated against reserved test data. If the re-trained models have better performance, they will be deployed to the online system, otherwise, the current versions of the models will remain);
Analogous art Zhang discloses calculates a reward using the matched observation information, action information, and observation information at the next time point (see Zhang, para [0060], wherein the value of a state is the total aggregated amount of rewards that the learning agent can expect to obtain since that state. For this example, it is the aggregated plant yields over a time period, say one month, from a state. A value function defines the rewards in the long run; and para [0038], wherein determine errors and/or feedback based on the comparison of predictions along with associated prescriptive actions to the observed incidents and the action items. With the real-time data and current version of the model, the information is utilized to re-train and improve the predictive and prescriptive models. The re-trained models are evaluated against reserved test data. If the re-trained models have better performance, they will be deployed to the online system, otherwise, the current versions of the models will remain).
One of ordinary skill in the art would have recognized that applying the known technique of Zhang would have yielded predictable results and resulted in an improved system for the same reasons as stated above with respect to claim 1.
Regarding claims 5 and 12, Osa discloses the offline pre-training and online fine-tuning apparatus of claim 1, wherein the offline model training unit samples at least some data from the previoulsy collected dataset (see Osa, para [0169], wherein a dataset contains samples of various qualities, some latent variables are associated with samples of actions with poor performance and corresponding lower policies also perform poorly and therefore such lower policies are not activated during execution. Conversely, the information of high-performance action samples in the dataset is actively utilized according to the present embodiment),
calculates and updates an objective function of a state-action value function network that evaluates a value of a specific state-action pair using the sampled data (see Osa, paras [0090]-[0095] & [0158], wherein in offline reinforcement learning, the expected value of the reward is evaluated with respect to states stored in a given dataset. Thus, the objective function is given by the following expression (6)…..The state-action pairs were reduced in dimensionality using t-SNE. The distribution of latent variable values shows how the state-action space is divided. The KL information quantity DKL(q(zls,a)llp(zls)) was minimized as a part of the objective function. Therefore, samples generated from q(zls,a) and p(zls) are to be similar), and
calculates and updates the objective function of the offline policy network (see Osa, paras [0015], [0090]-[0095], & [0158], wherein the value function update step may include calculating the target value….in offline reinforcement learning, the expected value of the reward is evaluated with respect to states stored in a given dataset. Thus, the objective function is given by the following expression (6)…..The state-action pairs were reduced in dimensionality using t-SNE. The distribution of latent variable values shows how the state-action space is divided. The KL information quantity DKL(q(zls,a)llp(zls)) was minimized as a part of the objective function. Therefore, samples generated from q(zls,a) and p(zls) are to be similar).
Regarding claims 6 and 13, Osa discloses the offline pre-training and online fine-tuning apparatus of claim 1, wherein the online model training, as set forth with claim 1,
collects initial observation information (see Osa, para [0060], wherein (Step S1) the acquisition unit 11 acquires a dataset including state information and action information on which a policy is to be learned in advance and stores the acquired dataset in the storage unit 12 (acquisition step); and para [0132], wherein a result of preliminary experiments, it was found that, the scaling factor has a nontrivial effect on performance and the optimal value of differs for each task…..avoid changing the scaling parameter for each task, normalization of the advantage function),
determines an action using the offline policy network and the initial observation information (see Osa, paras [0078]-[0079], wherein Fig. 7 is a flowchart of an exemplary procedure for a process for estimating an action using a trained model according to the present embodiment. (Step S21) The learning device 1 determines a latent variable that maximizes the action value for an observed state using the trained first model 131),
collects observation information at a next time point according to the determined action (see Osa, paras [0068]-[0071], wherein (Step S11) The acquisition unit 11 acquires a dataset in advance and stores the acquired dataset in the storage unit 12 (acquisition step). (Step S12) The processing unit 17 initializes the first model 131, the second model 141, and the third model 151. (Step S13) The processing unit 17 extracts some tuples of state s, action a, next state s', and reward r from the dataset. The processing unit 17 selects a plurality of tuples, the number of which ranges, for example, from 256 to 1024. (Step S14) The identification unit 16 identifies a latent variable z that maximizes the action value for the state s),
acquires a reward using the initial observation information, the action, and the observation information at a next time point (see Osa, para [0050], wherein acquires a dataset including rewards, state information, and action information on which a policy is to be learned. If the dataset does not contain rewards, it is necessary to recalculate only rewards according to a task to be learned; and para [0060], wherein (Step S1) the acquisition unit 11 acquires a dataset including state information and action information on which a policy is to be learned in advance and stores the acquired dataset in the storage unit 12 (acquisition step), and
updates the parameters of the offline policy network using the initial observation information, the action, and the reward (see Osa, para [0075], wherein (Step S18) the optimal action learning unit 14 trains and updates the second model 141 (the lower policy) using the states and the latent variable estimated in step S16; and paras [0014] & [0097]-[0098], wherein a lower policy that determines an action for givens and z. It is assumed that the lower policy re(als,z) is a deterministic policy……the lower policy deterministically determines an action for given s and z parameterized by a vector).
Osa et al. fails to explicitly disclose wherein the online model training unit loads the offline policy network pre-trained by the offline model training unit; and using the initial observation information, the action, the observation information at the next time point, and the reward.
Analogous art Zhang discloses wherein the online model training unit loads the offline policy network pre-trained by the offline model training unit (see Zhang, para [0097], wherein uploads the data to the service provider, and uses the web-based user interface…; and para [0093], wherein offline training system for generating models can be integrated with the online system deploying the models, thereby allowing for seamless updating of models for controlling the online systems without requiring the online systems to cease operations. After the offline models are deployed into the online system, the offline models can be improved based on historical data in parallel and then seamlessly replace the online models if performance improvement from offline model (compared to current version of online models) meets the predefined criteria, without interrupting the normal operation of the online systems)); and
Analogous art Zhang discloses updates the parameters of the offline policy network using the initial observation information, the action, the observation information at the next time point, and the reward (see Zhang, para [0091], wherein autonomously updates the predictive model and prescriptive model based on real time data, errors and feedback from the second system as illustrated in FIG. 2 and FIG. 5(b); para [0060], wherein the value of a state is the total aggregated amount of rewards that the learning agent can expect to obtain since that state. For this example, it is the aggregated plant yields over a time period, say one month, from a state. A value function defines the rewards in the long run; and para [0038], wherein determine errors and/or feedback based on the comparison of predictions along with associated prescriptive actions to the observed incidents and the action items. With the real-time data and current version of the model, the information is utilized to re-train and improve the predictive and prescriptive models. The re-trained models are evaluated against reserved test data. If the re-trained models have better performance, they will be deployed to the online system, otherwise, the current versions of the models will remain).
One of ordinary skill in the art would have recognized that applying the known technique of Zhang would have yielded predictable results and resulted in an improved system for the same reasons as stated above with respect to claim 1.
Regarding claim 7 is rejected based upon the same rationale as the rejection of claim 1, respectively, since it is apparatus claim corresponding to the apparatus claim. Claim 7 recites additional features a processor, and a memory connected to the processor (see Osa, para [0182]).
Regarding claims 8-13 are rejected based upon the same rationale as the rejection of claims 1-6, respectively, since they are method claims corresponding to the apparatus claims. Claim 8 recites additional features a processor and a memory (see Osa, para [0182]).
Conclusion
The prior arts made of record and not relied upon is considered pertinent to applicant's disclosure. (US Pub No. 2022/0237061; US Pat No. 12,619,911; US Pub No. 2019/0102676; US Pub No. 2025/0124334; US Pat No. 12,384,029; US Pub No. 2023/0185881; US Pub No. 2024/0403651; US Pub No. 2024/0051575; US Pub No. 2025/0156680; US Pat No. 11,313,292; US Pub No. 2023/0409957; S Wang, Q Yang, J Gao, M Lin et al. (Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning)… - Advances in …, 2023 - proceedings.neurips.cc; and K Zhao, Y Ma, J Liu, J Hao, Y Zheng (Improving offline-to-online reinforcement learning with q-ensembles)… - … new frontiers in learning …, 2023 - openreview.net.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HAFIZ A KASSIM whose telephone number is (571)272-8534. The examiner can normally be reached 9:00 - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Rutao Wu can be reached at 571-272-6045. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAFIZ A KASSIM/Primary Examiner, Art Unit 3623 09/28/2026
