DETAILED ACTION
This office action is in response to application filed on 3/21/2025.
Claims 1 – 5 are pending.
Priority is claimed as 371 application for PCT/CN2022/126468 (filed on 10/20/022), which claims priority from Chinese application CN202210309973.8 (filed on 3/28/2022).
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 – 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al, “Learning-Based Resource Allocation in Cloud Data Center Using Advantage Actor-Critic”, 2019, IEEE, pages 1 – 6 (hereinafter Chen), in view of Hammarlund et al (US 20050141554, hereinafter Hammarlund)
As per claim 1, Chen discloses: A deep reinforcement learning (DRL)-based cloud data center adaptive efficient resource allocation method,
wherein a unified resource allocation model is designed, the resource allocation model takes a job delay, and energy efficiency as optimization goals; (Chen page 2, left column, last paragraph: “A deep Q-Iearning based system for resource provisioning and task scheduling was designed in [10] to minimize the energy cost of cloud service providers.”; page 1, right column, last paragraph: “As the optimization goal (job latency) and the state (resource usage) have continuous space as well as for fast decision making in cloud computing”.)
based on the resource allocation model, a state space, an action space, and a reward function of cloud resource allocation are defined as a Markov decision process, and the Markov decision process is used in a DRL-based cloud resource allocation method; (Chen page 3, left column, last paragraph and equation 5 – page 3, right column, first paragraph: “During the learning process of optimizing policy for resource allocation, the RL agent (scheduler) chooses an action at (resource allocation and job scheduling) under current state St (resource usage and requests) of environment (cloud data center), and then it receives reward rt (penalty of job latency) and goes to the next state StH. This process can be formulated as a Markov Decision Process (MDP)… Due to the uncertainty of states in cloud data center, we formulate the problem of resource allocation and job scheduling with model-free RL. Meanwhile, as the above problem is a discrete-time based MDP with continuous spaces, we propose an actor-critic based RL method for better adaptiveness and achieving the accurate optimal policy more efficiently.”)
an actor-critic DRL-based resource allocation method is provided, to resolve an optimal policy problem of job scheduling in cloud data center; and in addition, based on the actor-critic DRL-based resource allocation method, policy parameters of a plurality of DRL agents are asynchronously updated. (Chen page 3, right column, first and second paragraphs under section IV: “Actor-critic is a hybrid of RL algorithms, which incorporates the value-based (e.g., Q-learning) and policy-based (e.g., policy gradient) RL algorithms. On the one hand, the value-based algorithms use temporal-difference (TD) learning to evaluate TD error generated during the learning process. They first use function approximators to determine the value function and then use c-greedy method to balance exploration and exploitation, which allows the RL agent both exploring new actions and utilizing existing experiences to choose the optimal action. On the other hand, the policy-based methods parameterize the policy and directly output actions during the learning process without storing the value function, so it can choose actions under the continuous state and action spaces. As RL methods, the optimization goal of actor-critic is getting as much reward as possible. Thus, an objective function is to measure the learning quality. For continuous state and action spaces, we use the following formula to accumulate the instant reward obtained from actions adopted at each state based on a probability distribution.”)
Chen did not explicitly disclose:
Wherein the resource allocation model takes a job dismissing rate as optimization goals;
However, Hammarlund teaches:
Wherein the resource allocation model takes a job dismissing rate as optimization goals; (Hammarlund [0041]: “monitoring the retirement rate of the threads with increased resources”.)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Hammarlund into that of Chen in order to have the resource allocation model takes a job dismissing rate as optimization goals. Hammarlund has shown the claimed limitation is a commonly known criteria for resource allocation optimization, thus applicants have merely claimed the combination of known parts in the field to achieve predictable results and is therefore rejected under 35 USC 103.
As per claim 2, the combination of Chen and Hammarlund further teach:
The deep reinforcement learning-based cloud data center adaptive efficient resource allocation method according to claim 1, wherein the DRL-based cloud resource allocation method specifically comprises: step S1: generating, by a resource allocation system RAS, a job scheduling policy according to resource requests of jobs of different users and current state information of the cloud data center, wherein the resource allocation system RAS comprises a DRL-based resource controller, a job scheduler, an information collector, and an energy agent; step S2: allocating, by the job scheduler, a job in a job sequence to a server of the cloud data center according to a policy delivered by the DRL-based resource controller; and step S3: recording, by the information collector during resource allocation, use conditions of different resources and current energy consumption measured by the energy agent in the cloud data center, and generating, by the DRL-based resource controller, the corresponding job scheduling policy. (Chen page 2, right column – page 3, right column: section III: system model and problem formulation.)
As per claim 3, the combination of Chen and Hammarlund further teach:
The deep reinforcement learning-based cloud data center adaptive efficient resource allocation method according to claim 2, wherein a state space, an action space, and a reward function in the DRL are defined as follows: the state space: in a state space S, a state s.sub.t∈S is represented by a time step t and formed by resource usage of all servers and resource requests of all arrived jobs; on one hand, U.sub.t.sup.res=[[u.sub.1,1,u.sub.1,2, . . . ,u.sub.1,n], [u.sub.2,1,u.sub.2,2, . . . ,u.sub.2,n], . . . ,[u.sub.m,1,u.sub.m,2, . . . ,u.sub.m,n]], where u.sub.m,n is a use condition of an n.sup.th resource type on a server virtual machine; on the other hand, O.sub.t.sup.res=[[o.sub.1,1,o.sub.1,2, . . . ,o.sub.1,n], [o.sub.2,1,o.sub.2,2, . . . ,o.sub.2,n], . . . ,[o.sub.m,1,o.sub.m,2, . . . ,o.sub.m,n]], which represents occupancy requests of all arrived jobs for different resource types, where o.sub.j,n is an occupancy request of a recently arrived job j for the n.sup.th resource type, D.sub.t.sup.job=[d.sub.1,d.sub.2, . . . ,d.sub.j] represents durations of all arrived jobs at the time step t, and d.sub.j represents a duration of the job j, so that the state of the cloud data center at the time step t is defined as: st=[stV,stJ]=[Utres,[Otres,Dtjob]]Formula(1) where s.sub.t.sup.V=U.sub.t.sup.res and s.sub.t.sup.J=[O.sub.t.sup.res,D.sub.t.sup.job] are used for representing states of all the servers and arrived jobs, V={v.sub.1, v.sub.2, . . . , v.sub.m}, J represents a job sequence; and when a job arrives or is completed, the state space is changed, and a dimension of the state space depends on conditions of the server and the arrived job, which is calculated by (mn+z(n+1)), where m, n, and z respectively represent a quantity of servers, resource types, and arrived jobs; the action space: at the time step t, an action performed by the job scheduler is to select and perform a job from the job sequence according to the job scheduling policy delivered by the DRL-based resource controller; the policy is generated according to a current state of the resource allocation system, and the job scheduler allocates a job to a corresponding server; once a job is scheduled to a corresponding server, the server automatically allocates a corresponding resource according to a resource request of the job; and therefore, an operation space A indicates only whether a job is processed by the server and is defined as: A={at|at∈{0,1,2,.Math.,m}}Formula(2) where a.sub.t ∈A; and when a.sub.t=0, the job scheduler does not allocate the job at the time step t, and the job needs to wait in the job sequence; otherwise, the job is processed by the corresponding server; a state transition probability matrix: the matrix represents probabilities of transition between two states, where there is no to-be-processed job at a time step t.sub.0, and an initial state s.sub.0=[0,[[0],[0]]], where three “0” items respectively represent the CPU usage of a server, an occupancy request of a job, and a duration of the job; at t.sub.1, a job j.sub.1 is immediately scheduled because available resources are sufficient; after the operation is performed, the state develops into s.sub.1=[50,[[50],[d.sub.1]]],d.sub.1, where the first “50” item represents utilization of the CPU of the server, the second “50” item represents an occupancy request of j.sub.1 for CPU resource, and d.sub.1 represents a duration of j.sub.1; and similarly, after j.sub.2 is scheduled at t.sub.2, the state develops into s.sub.2=[80,[[50,30],[d.sub.1,d.sub.2]]], where the state transition probability matrix is denoted as IP(s.sub.t+1|s.sub.t,a.sub.t), which represents a probability of a transition to a next state s.sub.t+1 when one action a.sub.t is performed in a current state s.sub.t; and a value of the transition probability is obtained by running a DRL algorithm, and probabilities that different actions are adopted in a state are outputted by using the algorithm; and the reward function: a DRL agent is guided to learn a better job scheduling policy with higher discounted long-term reward through the reward function, to improve system performance of cloud resource allocation; and therefore, at the time step t, a total reward R.sub.t is formed by two parts of a QoS reward that is denoted as R.sub.t.sup.QoS and energy efficiency that is denoted as R.sub.t.sup.energy and is defined as: Rt=RtQoS+RtenergyFormula(3) specifically, R.sub.t.sup.QoS reflects penalties of delays of different types at the time step t, which includes T.sub.t.sup.j,wait, T.sub.t.sup.j,work, and T.sub.t.sup.j,miss and is defined as: RtQes=-.Math.j∈Jseq(w1.Math.Ttj,wait+Ttj,workdj+w2.Math.Ttj,miss)Formula(4) where w.sub.1 and w.sub.2 are used for weighting the penalty; because R.sub.t.sup.QoS is a negative value, a job with a relatively long duration tends to wait for a relatively short time; and in addition, R.sub.t.sup.energy reflects a penalty for energy consumption at the time step t and is defined as: Rtenergy=-w3.Math..Math.j∈JsegEtj,execFormula(5) where Ej,exect is a time step t consumed to perform a job, and w.sub.3 is a weight of the penalty. (Chen page 2, right column – page 3, right column: section III: system model and problem formulation.)
As per claim 4, the combination of Chen and Hammarlund further teach:
The deep reinforcement learning-based cloud data center adaptive efficient resource allocation method according to claim 1, wherein the actor-critic DRL-based resource allocation method uses an actor-critic-based DRL framework and asynchronous advantage actor-critic A3C to speed up a training process; specifically, the actor-critic DRL-based resource allocation method combines a value-based DRL algorithm and a policy-based DRL algorithm: on one hand, the value-based DRL determines a value function by using a function approximator, and balances exploration and development by using ∈-greedy; and on the other hand, the policy-based DRL parameterizes the job scheduling policy and outputs actions directly in probability distribution during learning without storing Q-values thereof. (Chen page 3, right column – page 5, left column: section IV: “ACTOR-CRITIC BASED RL ALGORITHM WITH ADVANTAGE FUNCTION”.)
As per claim 6, the combination of Chen and Hammarlund further teach:
The deep reinforcement learning-based cloud data center adaptive efficient resource allocation method according to claim 3, wherein in each DRL agent, a critic network Q.sup.π.sup.θ estimates a state-action value function QW(st, at)≈Qπθt(st, at) and updates a parameter w; in addition, an actor network V.sup.π.sup.θ guides the update of a job scheduling policy parameter according to an estimation value of the critic network; and a corresponding policy gradient is defined as: ∇θtJ(θt)=Eπθt[∇θtlogπθt(st,at)Qw(st,at)]Formula(6). (Chen page 3, right column – page 5, left column: section IV: “ACTOR-CRITIC BASED RL ALGORITHM WITH ADVANTAGE FUNCTION”.)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Fu et al, “An actor-critic reinforcement learning-based resource management in mobile edge computing systems”, International Journal of Machine Learning and Cybernetics (2020) 11: pages 1875–1889, teaches “Reinforcement learning (RL) as an effective tool has attracted great attention in wireless communication field nowadays. In this paper, we investigate the offloading decision and resource allocation problem in mobile edge computing (MEC) systems based on RL methods. Different from existing literature, our research focuses on improving mobile operators’ revenue by maximizing the amount of the offloaded tasks while decreasing the energy expenditure and time-delays. Considering the dynamic characteristics of wireless environment, the above problem is modeled as a Markov decision process (MDP). Since the action space of the MDP is multidimensional continuous variables mixed with discrete variables, traditional RL algorithms are powerless. Therefore, an actor-critic (AC) with eligibility traces algorithm is proposed to resolve the problem. The actor part introduces the parameterized normal distribution to generate the probabilities of continuous stochastic actions, and the critic part employs a linear approximator to estimate the value of states, based on which the actor part updates policy parameters in the direction of performance improvement. Furthermore, an advantage function is designed to reduce the variance of the learning process. Simulation results indicate that the proposed algorithm can find the best strategy to maximize the amount of the tasks executed by the MEC server while decreasing the energy consumption and time-delays.”
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHARLES M SWIFT whose telephone number is (571)270-7756. The examiner can normally be reached Monday - Friday: 9:30 AM - 7PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, April Blair can be reached at 5712701014. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHARLES M SWIFT/Primary Examiner, Art Unit 2196