Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 07/26/2026 has been entered.
Status of Claims
This action is a responsive to the application filed on 07/26/2026.
Claims 1-4, 6-14, and 16-20 are pending.
Claims 1-2, 9, 11-12, and 19 have been amended.
Claims 5 and 15 have been canceled.
Response to Arguments
Applicant’s arguments, with respect to the rejection(s) of claim(s) 1 and 11 under 35 U.S.C. 103, have been considered but they are not persuasive. Applicant argues that no reference teaches the amended limitation now stating “sampling an execution time from the probability distribution defined by the tensors of the warm up table associated with the node, and determining a reward by using the sampled execution time instead of executing the selected action to completion such that the reinforcement learning model is trained without waiting for the workload to complete execution”, since “Mondal requires that workloads must be executed in order for rewards to be obtained. That is directly contradicted by the claimed invention, which explicitly avoids runtime execution during training”, and “Tong does not disclose precomputing execution-time distributions offline and then using those distributions as a substitute for executing the workload during reinforcement learning training”. The examiner respectfully disagrees.
Due to the broadness of the claim language the combination has been found to teach the claimed limitations. Tong, sections 4.2-5.1 teach a probability distribution of state transitions based on “execution time of individual tasks” for the reward; wherein “It works by learning some value functions that give the expected reward of taking each action in any state. It does not need any training samples except the historical allocations. There is no need to model task arrival and task execution”.
See 35 U.S.C 103 section for full mapping of claim limitations necessitated by applicant amendments.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 6-14, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Mondal et al (“Scheduling of Time-Varying Workloads Using Reinforcement Learning”, 2021) hereinafter Mondal, in view of Tong et al (“Proactive scheduling in distributed computing—A reinforcement learning approach”, 2014) hereinafter Tong, in view of Dirac et al (US Patent 10963810) hereinafter Dirac.
Regarding claims 1 and 11, Mondal teaches a method; a non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising (section “Design of TVW-RL” teaches machines for executing embodiments of the disclosure including one or more CPUs and one or more memories):
generating, prior to training, warm up tables for respective nodes in a computing environment, each warm up table including a plurality of tensors representing probability distributions of execution times for workloads executed at the respective nodes (sections “Background”-“Rewards Design for RL” and Figs. 1-2 teach a time-varying workloads (TVW) vectors (tensors) implemented as learned policies (probability distributions) that include a state of the system and corresponding action at a timestep most likely to move the state forward. For each machine (nodes) scheduled to run a TVW, “Length of the history, i.e. h helps the agent to learn the temporal characteristics of each TVW” so executing certain workloads times have minimum overlap (execution times for workloads executed at the respective nodes), then training a network on the data), wherein the probability distributions are defined by at least a first tensor storing mean execution times and a second tensor storing standard deviations for respective combinations, and wherein the tensors store execution times for different combinations of one or more workloads executed at the respective nodes (Mondal, sections “Background”-“Rewards Design for RL”, “Equivalence-class Analysis”, and Figs. 1-2 teach learning TVW policies (warmup tables/probability distributions) based on computed TVW vectors (first/second tensors) including their execution workload times, “avg., std.” of times (mean/std deviation), to have minimum overlap with other TVWs for different “job types” (different)); and
training a reinforcement learning model using the warm up tables, by during training, selecting an action corresponding to an allocation of a workload to a node, retrieving, from the warm up table associated with the node, the probability distribution corresponding to the selected action, sampling an execution time from the probability distribution defined by the tensors of the warm up table associated with the node, and determining a reward by using the sampled execution time such that the reinforcement learning model is trained without waiting for the workload to complete execution (sections “Background”-“Rewards Design for RL” and Figs. 1-2 teach a time-varying workload reinforcement learning (TVW-RL) model implemented as an “agent” tasked with learning (training a RL mode…during training) policies (warmup tables) of state, action, rewards (retrieving…the probability distribution corresponding to the selected action). “We use negative rewards” based on computed TVW vectors including their execution workload times to have minimum overlap with other TVWs when scheduling jobs on machines (allocation of a workload to a node)).
However, Mondal does not explicitly teach determining a reward by using the sampled execution time instead of executing the selected action to completion.
Tong teaches determining a reward by using the sampled execution time instead of executing the selected action to completion (sections 4.2-5.1 teach a probability distribution of state transitions based on “execution time of individual tasks” for the reward; wherein “It works by learning some value functions that give the expected reward of taking each action in any state. It does not need any training samples except the historical allocations. There is no need to model task arrival and task execution”).
Further, Mondal at least implies probability distributions for time required to execute workload (see mappings above); however, Tong teaches probability distributions for time required to execute workload (sections 4.2-5.1 teach a probability distribution of state transitions based on “execution time of individual tasks” for the reward).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to implement Tong’s teachings of determining a probability distribution of processing units executing tasks times for a reward in a reinforcement learning structure into Mondal‘s teaching of time-varying workload scheduling through reinforcement learning in order to increase efficiency of task scheduling achieve “much better load balance” (Tong, sections 5.2 and 7).
Further still, Mondal at least implies wherein the probability distributions are defined by at least a first tensor storing mean execution times and a second tensor storing standard deviations for respective combinations (see mappings above); however, Dirac teaches wherein the probability distributions are defined by at least a first tensor storing mean execution times and a second tensor storing standard deviations for respective combinations (Col. 16, line 57-Col. 17, line 16, and Col. 22, lines 27-33 teach extracting statistical numeric values for each data attribute including the “standard deviation” of executing different jobs types on certain resources; and Col. 83, lines 22-42, Col. 85, lines 1-20, and Col. 142, lines 6-10 teach numeric values as vectors for parameters and being comprised in a “probability distribution”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Mondal‘s teaching of time-varying workload scheduling through reinforcement learning as modified by Tong’s teachings of determining a probability distribution of processing units executing tasks times for a reward in a reinforcement learning structure, to include as taught by Dirac in order to achieve better job implementation flexibility based on resource parameter data knowledge (Dirac, Col. 16, line 57-Col. 17, line 43).
Regarding claims 2 and 12, the combination of Mondal, Tong, and Dirac teach all the claim limitations of claims 1 and 11 above; and further teach further comprising generating execution times for workloads prior to training the reinforcement learning model (Mondal, sections “Background”-“Rewards Design for RL” and Figs. 1-2 teach a time-varying workloads (TVW) implemented as policies that include a state of the system and corresponding action at a timestep most likely to move the state forward. For each machine scheduled to run a TVW, “Length of the history, i.e. h helps the agent to learn the temporal characteristics of each TVW” so executing certain workloads times have minimum overlap (time required to execute workload). Then the RL model is “learned” on the policies (prior to training the reinforcement learning model).).
Regarding claims 3 and 13, the combination of Mondal, Tong, and Dirac teach all the claim limitations of claims 2 and 12 above; and further teach further comprising generating execution times for different types of workloads (Mondal, sections “Background”-“Rewards Design for RL” and Figs. 1-2 teach workloads times for different “job types”).
Regarding claims 4 and 14, the combination of Mondal, Tong, and Dirac teach all the claim limitations of claims 1 and 11 above; and further teach further comprising generating execution times for one or more workloads of one or more workload types (Mondal, sections “Background”-“Rewards Design for RL” and Figs. 1-2 teach workloads times for different “job types”).
Regarding claims 6 and 16, the combination of Mondal, Tong, and Dirac teach all the claim limitations of claims 5 and 15 above; and further teach wherein the first tensor stores execution times for different combinations of one or more workloads of one or more types (Mondal, sections “Background”-“Rewards Design for RL” and Figs. 1-2 teach learning TVW policies (warmup tables) based on computed TVW vectors (first/second tensors) including their execution workload times to have minimum overlap with other TVWs for different “job types”).
Regarding claims 7 and 17, the combination of Mondal, Tong, and Dirac teach all the claim limitations of claims 6 and 16 above; and further teaches wherein the second tensor stores standard deviations for the one or more workloads of one or more types (Dirac, Col. 16, line 57-Col. 17, line 16 teach extracting statistical values for each data attribute including the “standard deviation” of executing different jobs types on certain resources).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Mondal‘s teaching of time-varying workload scheduling through reinforcement learning as modified by Tong’s teachings of determining a probability distribution of processing units executing tasks times for a reward in a reinforcement learning structure, to include as taught by Dirac in order to achieve better job implementation flexibility based on resource parameter data knowledge (Dirac, Col. 16, line 57-Col. 17, line 43).
Regarding claims 8 and 18, the combination of Mondal, Tong, and Dirac teach all the claim limitations of claims 1 and 11 above; and further teach further comprising, during training, selecting an action and executing the action in a state (Mondal, sections “Background”-“Rewards Design for RL” and Figs. 1-2 teach a time-varying workload reinforcement learning (TVW-RL) model implemented as an “agent” tasked with learning (training a RL model) policies (warmup tables) that include a state of the system and corresponding action at a timestep most likely to move the state forward).
Regarding claims 9 and 19, the combination of Mondal, Tong, and Dirac teach all the claim limitations of claims 8 and 18 above; and further teach further comprising generating the rewards prior to termination of the workloads (Mondal, sections “Background”-“Rewards Design for RL” and Figs. 1-2 teach a time-varying workload reinforcement learning (TVW-RL) model implemented as an “agent” tasked with learning (training a RL model) policies (warmup tables). “We use negative rewards” when computing TVW vectors including their execution workload times to have minimum overlap with other TVWs. Further, “we add a penalty proportional to the sum of unused resources in the used machines. A used machine is one with at least one workload running.”).
Regarding claims 10 and 20, the combination of Mondal, Tong, and Dirac teach all the claim limitations of claims 9 and 19 above; and further teach further comprising observing the new state, computing a loss and updating the states (Mondal, sections “State Space Representation for RL”-“Rewards Design for RL” teach algorithm 1 analyzing an input state, calculating a penalty from error, and updating the policy network).
Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Harsha et al (US Pub 20230041035) teaches reinforcement learning and utilizing probability distribution for determining action and reward pairings.
Conclusion
13. Any inquiry concerning this communication or earlier communications from the examiner should be directed to CLINT MULLINAX whose telephone number is 571-272-3241. The examiner can normally be reached on Mon - Fri 8:00-4:30 PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on 571-270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.M./Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights