DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 6-9, 10,12-14 and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wilde et al. (US 20220291734) in view of Carter et al. (US 20110178652), Jackson (US 20140075222), and Bodas et al. (US 20160054780) (Bodas2).
Regarding claim 1, Wilde teaches
A system comprising:
one or more processors; and memory having programmed thereon instructions for executing a workload scheduling program, the workload scheduling program causing the one or more processors to: ([0046, “cause the processor 712 to communicate with the follower instances to acquire node performance data about the job while the job is being executed by the plurality of nodes. The instructions 708, when executed by the processor 712, further cause the processor to determine, based on the performance data, a performance footprint that is associated with the job. The instructions 708, when executed by the processor 712, further cause the processor to determine, based on the performance footprint, whether to request an increase in a node power consumption budget”)
receive a workload for execution by one or more of a plurality of machines; ([0021], “In general, a “job” refers to an organization of work that is associated with a particular application and is assigned to a group of nodes 110 by a job scheduler 140. A message passing interface (MPI) job is an example of a “job.” A given job 121 may be divided into a set of ranks (e.g., MPI ranks), or processes. Each rank of a given job 121 may be processed in parallel with the other ranks of the job 121. The job scheduler 140 schedules, or assigns, the ranks of each job 121 to a particular set of nodes 110.”)
assign the workload to one or more designated machines of the plurality of machines; ([0021], “The job scheduler 140 schedules, or assigns, the ranks of each job 121 to a particular set of nodes 110.”)
determine a respective power quota for each of the one or more designated machines based on the predicted respective amounts of power consumed by each of the one or more designated machines; (Figs. 4-6, [0038], “the optimal node processing frequency may correspond to the same power limit for each node 110 executing the job; and the agent leader instance 310 may correspondingly communicate a power request 360 to the power dispatcher 150 to either request an increase or a decrease to the job power cap 220. Moreover, in accordance with some implementations, the agent 116 may determine not to modify the job power cap 220. In accordance with further example implementations, the optimal node processing frequency may correspond to power consumption limits that vary among the nodes 110 due to differences (e.g., different processing cores or performance benchmarks) of the nodes 110.”, [0043], “It is noted that this is the overall cap for the jobs 110 that execute the job, as the agent leader instance 310 may distribute the node power caps differently, depending on specific knowledge of the relationship between node power consumption and frequency of the nodes 110.”)
update, after assigning the workload to the one or more designated machines, a record indicating (i) available power of a domain including the plurality of machines and/or (ii) available machines within the domain. ([0019], “each power domain 104 receives its power from an associated “power pool 120,” which refers to an available amount of power. … the global power dispatcher 150 may allocate power to the domain 104 from its associated power pool 120, with the power budgets for the power pools 120 being equal to the system power cap, less a power reserve.” And claim 2, “the global power dispatcher granting the power consumption request, wherein the global power dispatcher increases or decreases a power budget associated with the job in response to granting the power consumption request and reduces or increases the available power for job related power domains managed by the global power dispatcher in response to granting the power consumption request.”)
Wilde teaches power capping of a node but does not specifically teach a power capping control loop to control each of its nodes to its respective power quota.
Carter teaches
instruct a programmable power capping control loop to control operation of each of the one or more designated machines according to its respective power quota; and (Fig. 2, [0014], “use voltage and/or frequency settings in combination with an on-chip energy governor that dynamically power caps the amount of energy a unit, such as a processor, a processor core, a data processing component associated with a processor or processor core, a memory, or the like, may consume over a particular time interval, even if the workload changes dramatically.” [0029], “a governor of the power capping mechanism may place the unit or group of units into a safer operating range by shifting a voltage and/or a frequency using Dynamic Voltage and Frequency Scaling (DVFS) supplied to the unit to a known safe execution state, suspend the unit or group of units if the counter value reaches zero, or perform some other action to enforce the energy budget of the unit or group of units.” And [0030], “within data processing system 200 comprises governors 204a-204n and counters 206a-206n which are respectively associated with units 208a-208n. Units 208a-208n may be processors in a multiprocessor system”)
Wilde and Carter are analogous art. Carter is cited to teach a similar concept of managing power for nodes and a system related to a workload/job. Based on Carter, it would have been obvious before the effective filing date of the invention to a person having ordinary skill in the art to which said subject matter pertains to have modified Wilde to include individual power capping governors/loops for controlling power in the system. Furthermore, being able to use a power cap loop improves on Wilde by being able to dynamically adjust the operating state and the power cap. To one of ordinary skill in the art before the effective filing data of the invention it would have been advantageous to make this modification to “provide a mechanism for aggressively exploiting a given power cap, such as through aggressively "over-clocking" or "under-voltaging," to safely reduce safety (voltage) margins. The illustrative embodiments use voltage and/or frequency settings in combination with an on-chip energy governor that dynamically power caps ”, [0014]
Wilde and Carter teach controlling power by the use of power capping but do not teach that the power cap/quota for a given machine is the sum of power caps of multiple workloads running on the machine.
Jackson teaches
wherein for a given machine of the one or more designated machines, adjustment of the control operation of the given machine is based on summing the power quota of the assigned workload with power quotas of other workloads assigned to the given machine; ([0108], “ if an analysis of the current workload identifies that each individual job has their own power cap, such as a first job has a power cap of 15 kilowatts and a second job has a power cap of 5 kilowatts, the system can basically sum up these and other power caps and durations and identify what the total power consumption is going to be. … the system will then make decisions if that number exceeded the system wide power cap then the system would perform power reduction operations on individual jobs.”)
Jackson, Wilde, and Carter are analogous art. Carter is cited to teach a similar concept of managing power for nodes and a system related to a workload/job. Jackson teaches the power cap/quota for a given machine is the sum of power caps of multiple workloads running on the machine and adjusting operation based on the power cap/quota. Based on Jackson, it would have been obvious before the effective filing date of the invention to a person having ordinary skill in the art to which said subject matter pertains to have modified Wilde and Carter to include the power cap/quota for a given machine is the sum of power caps of multiple workloads running on the machine and adjusting operation based on the power cap/quota. Furthermore, being able to power cap based on the sum of power cap jobs/workloads for a particular machine improves on Wilde and Carter by being able to better adjust power consumption based on the information. To one of ordinary skill in the art before the effective filing data of the invention it would have been advantageous to make this modification to “a server running a workload management software that communicates with resource managers and other facilities to enable improved power consumption”, [0021], and “can use the information to optimally pack workload or improve the packing of workload within the specified power budget.”, [0089]
Jackson, Wilde and Carter teach controlling power by the use of power capping but do not teach predicting amount of power consumed by a workload on the devices or determining the power capping based on the predicted power consumption.
Bodas2 teaches
before initiation of the workload at the one or more designated machines: ([0102], “Before a scheduler can start a job, it is estimated how much power is needed.”) Bodas 2
predict respective amounts of power consumed by each of the one or more designated machines based on one or more power related properties of the workload; ([0057], “A power-aware scheduler is used to estimate the power required to run a job. Power-performance calibration of nodes is used to develop such an estimate. In one embodiment, the power estimate is determined based upon power-performance data collected on sample workloads or past runs of the job.”)
determine a respective power quota for each of the one or more designated machines based on the predicted respective amounts of power consumed by each of the one or more designated machines; ([0057], “A power-aware scheduler is used to estimate the power required to run a job. Power-performance calibration of nodes is used to develop such an estimate. In one embodiment, the power estimate is determined based upon power-performance data collected on sample workloads or past runs of the job.”, [0074], “job manager 312 is used to control power performance of all components (e.g., nodes, or other components) involved in execution of a job” and [0037], “o support operation under a power limit (cap) a HPC job launch-time scheduler and run-time manager, as described herein is power aware to deliver best performance within a fixed power budget.”)
Bodas2, Jackson, Wilde, and Carter are analogous art. Bodas2 is cited to teach a similar concept of managing power for nodes and a system related to a workload/job. Bodas2 teaches predicting power consumption of processors/devices based on the job to be executed and determining/adjusting power caps/budgets/quota for the devices based on the predicted power consumption. Based on Bodas2, it would have been obvious before the effective filing date of the invention to a person having ordinary skill in the art to which said subject matter pertains to have modified Wilde, Carter, and Jackson to adjust the power allocation/quota of devices/processors based on predictions/estimations of power consumption of the job which is about to be executed. Furthermore, being able to adjust the power allocation/quota of devices/processors based on predictions/estimations of power consumptions of the workload which is about to be executed improves on Jackson, Wilde and Carter by improve power efficiency and performance in the system. To one of ordinary skill in the art before the effective filing data of the invention it would have been advantageous to make this modification because “An advantage of embodiments described herein is that the power consumption is managed by allocating power to the jobs. As such, the power consumption is allocated in a way to cause significant reduction in the performance variations of the nodes and subsequently improvement in job completion time. In other words, the power allocated to a particular job is distributed among the nodes dedicated to run the job in such a way to achieve the increased performance.”, [0028]
Regarding claim 2, Carter teaches further comprising the programmable power capping control loop. (Fig. 2, [0029], “a governor of the power capping mechanism may place the unit or group of units into a safer operating range by shifting a voltage and/or a frequency using Dynamic Voltage and Frequency Scaling (DVFS) supplied to the unit to a known safe execution state, suspend the unit or group of units if the counter value reaches zero, or perform some other action to enforce the energy budget of the unit or group of units.” And [0030], “within data processing system 200 comprises governors 204a-204n and counters 206a-206n which are respectively associated with units 208a-208n. Units 208a-208n may be processors in a multiprocessor system”)
Regarding claim 3, Carter teaches wherein the programmable power capping control loop is configured to monitor power inputs at the one or more designated machines and adjust power consumption at each of the one or more machines based on the monitored power inputs. ([0034-35], “Upon completion of the predetermined interval, power control loop 210 makes new measurements of the power at the current voltage and frequency settings of each of units 208a-208n. Power control loop 210 determines a new energy budget. However in determining the new energy budget for each of units 208a-208n, power control loop 210 not only uses the power cap of a computing system and the new measurements, but also uses the counter values of the associated counters 206a-206n. That is, if power control loop 210: determines that the counter value of a counter associated with a unit is much greater than zero (counter value>a predetermined value>0), indicating that the unit did not execute the workload initially estimated, power control loop 210 may allocate a reduced new energy budget;”)
Regarding claim 6, Wilde teaches wherein, in response to initiation of the workload, the instructions cause the one or more processors to increase the respective power quota of each of the machines and update the record accordingly, and in response to completion of the workload, the instructions cause the one or more processors to decrease the respective power quota of each of the machines and update the record accordingly.([0032], “With each job 121 scheduled, the global power dispatcher 105 reduces the available power budget until sufficient power is unavailable to schedule any more jobs 121 in the available power budget. The global power dispatcher 150 refills, or increases, the available power budget in response to an agent 116 returning excess power during runtime of a job 121 and when the job 121 finishes.”)
Regarding claim 7, Bodas2 teaches wherein the instructions further cause the one or more processors to:
in response to receiving the workload, adjust the overall power quota of a previously received workload; and instruct the programmable power capping control loop to control operation of each of one or more previously designated machines to which the previous workload is assigned to adjust their respective power quotas to meet the adjusted overall power quota of the previously received workload. ([0143], “ At operation 972 it is determined if a power allocation for a current job can be reduced. … If the power allocation for the current job can be reduced, at operation 973 it is determined if the job runs at a frequency greater than a minimum frequency. … If the job runs at a frequency greater than the minimum frequency, at operation 974 a power estimate for a next job in a queue is determined. At operation 975 it is determined if a power headroom is available for a next job. If the power headroom is available for the next job, at operation 976 the next job is accommodated.” [0137], “The operating frequency Fo is decreased when the power allocation for the job reduces.”, [0112], “Re-evaluations of power budgets and uniform frequency are performed periodically during runtime. The available power may drop so much that all jobs cannot continue running. In that case the power aware job scheduler picks a job at the lowest priority from a list of jobs that can be suspended.” And [0115], “the power is allocated for a job based on a job priority.”)
Regarding claim 8, Wilde, Carter, Jackson, and Bodas2 do not teach but Bodas2 teaches wherein adjustment of the overall power quota of the previously received workload is based on respective priority levels of the workload and the previously received workload. ([0047], “A per job power cap is set dynamically based on at least one of a facility power capability and a suspended job priority.”, [0112], “The scheduler starts the job if the available power is equal to or greater than a startup power. When dynamic monitoring is available, in certain power policies, e.g., an auto mode the uniform frequency used by all nodes of the system may be changed periodically based on a power headroom. In one embodiment, the job that started earlier in time has higher priority in using the additional power headroom. Re-evaluations of power budgets and uniform frequency are performed periodically during runtime. The available power may drop so much that all jobs cannot continue running. In that case the power aware job scheduler picks a job at the lowest priority from a list of jobs that can be suspended.” And [0115], “the power is allocated for a job based on a job priority.”)
Regarding claim 9, Wilde, Carter, Jackson, and Bodas2 do not teach but Bodas2 teaches wherein the instructions further cause the one or more processors to transmit an indication of the adjustment of the overall power quota to the previously received workload. ([0137], “The operating frequency Fo is decreased when the power allocation for the job reduces.”)
Regarding claim 11, Wilde teaches wherein each machine of the plurality of machines is a tray including a plurality of processor chips. “a given rack may contain chassis units, other than chassis units that contain the nodes 110 of the cluster computer system 100. As a more specific example, in accordance with some implementations, a chassis unit of a rack may be a server blade enclosure that has slots (e.g., eight slots) to receive a number (e.g., eight) of corresponding server blades, and each server blade may contain a number (e.g., four) of the nodes 110.”
As to claims 10 and 12, Jackson, Wilde, Bodas2, and Carter teach these claims according to the reasoning provided in claims 1 and 2.
As to claim 13, Jackson, Wilde, Bodas2, and Carter teach this claim according to the reasoning provided in claims 1 and 2.
As to claim 14, Jackson, Wilde, Bodas2, and Carter teach this claim according to the reasoning provided in claim 3.
As to claim 17, Jackson, Wilde, Bodas2, and Carter teach this claim according to the reasoning provided in claim 6.
As to claims 18-20, Wilde, Carter, Jackson, and Bodas2 teach these claims according to the reasoning provided in claims 7-9, respectively.
Claim(s) 4-5 and 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wilde, Carter, Jackson, and Bodas2 as applied to claims 1 and 12 above, and further in view of Bodas et al. (US 20160054779).
Regarding claim 4, Wilde, Carter, Jackson, and Bodas2 do not teach but Bodas teaches wherein the instructions further cause the one or more processors to:
in response to receiving the workload, access a power characterization data source indicating predicted power related properties of the workload; ([0053], “ Power Aware Job scheduler 411 receives power-performance characteristics of the job at different operating points from Estimator 413 and Calibrator 414. Resource Manager 410 forecasts how much power a particular job needs and take corrective actions when actual power differs from the estimation.”)
determine an overall power quota for the workload based on the predicted power-related properties of the workload; and ([0053], “Power Aware Job scheduler 411 considers the policies and priorities of Facility Administrator 102, Utility Provider 103, User 201, and HPC System Administrator 202 and determines accordingly what hardware resources of HPC System 400 is needed to run a particular job. Additionally, Power Aware Job scheduler 411 receives power-performance characteristics of the job at different operating points from Estimator 413 and Calibrator 414. Resource Manager 410 forecasts how much power a particular job needs and take corrective actions when actual power differs from the estimation.”)
determine the respective power quota for each of the one or more designated machines based on the overall power quota. ([0043], “Job Manager 420 adjusts the soft limit associated with Nodes 500 to maintain both soft limit and power consumption of Nodes 500 at or below the hard limit.”)
Bodas, Wilde, Carter, Jackson, and Bodas2 are analogous art. Bodas is cited to teach a similar concept of managing power for nodes and a system related to a workload/job. Bodas teaches using predicated power related properties such as average power to determine an overall power quota/cap for the workload/job. Wilde teaches dynamically changing the power cap based on determining an optimal frequency for the job’s operation using prediction (i.e. machine learning). Based on Bodas, it would have been obvious before the effective filing date of the invention to a person having ordinary skill in the art to which said subject matter pertains to have modified Wilde, Carter, Jackson, and Bodas2 to determine the overall quota and node quotas based on predicted power properties such as average power. Furthermore, being able to determine the quotas based on predicted power properties improves on Wilde, Carter, Jackson, and Bodas2 by being able to increase performance of the job. To one of ordinary skill in the art before the effective filing data of the invention it would have been advantageous to make this modification to “An advantage of embodiments described herein is that the power consumption is managed by allocating power to the jobs. As such, the power consumption is allocated in a way to cause significant reduction in the performance variations of the nodes and subsequently improvement in job completion time. In other words, the power allocated to a particular job is distributed among the nodes dedicated to run the job in such a way to achieve the increased performance.”, [0028]
Regarding claim 5, Wilde, Carter, Jackson, and Bodas2 do not teach but Bodas teaches wherein the power related properties include one or more of an average power consumption, a maximum current change slew rate, an average utilization ratio, or a maximum utilization ratio. ([0054], “Estimator 413 provides a power consumption estimate based on, for example, maximum and average power values”)
Bodas, Wilde, Carter, Jackson, and Bodas2 are analogous art. Bodas is cited to teach a similar concept of managing power for nodes and a system related to a workload/job. Bodas teaches using predicated power related properties such as average power to determine an overall power quota/cap for the workload/job. Wilde teaches dynamically changing the power cap based on determining an optimal frequency for the job’s operation using prediction (i.e. machine learning). Based on Bodas, it would have been obvious before the effective filing date of the invention to a person having ordinary skill in the art to which said subject matter pertains to have modified Wilde, Carter, Jackson, and Bodas2 to determine the overall quota and node quotas based on predicted power properties such as average power. Furthermore, being able to determine the quotas based on predicted power properties improves on Wilde, Carter, Jackson, and Bodas2 by being able to increase performance of the job. To one of ordinary skill in the art before the effective filing data of the invention it would have been advantageous to make this modification to “An advantage of embodiments described herein is that the power consumption is managed by allocating power to the jobs. As such, the power consumption is allocated in a way to cause significant reduction in the performance variations of the nodes and subsequently improvement in job completion time. In other words, the power allocated to a particular job is distributed among the nodes dedicated to run the job in such a way to achieve the increased performance.”, [0028]
As to claims 15-16, Wilde, Carter, Jackson, Bodas2 and Bodas teach these claims according to the reasoning provided in claims 4-5, respectively.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 10, and 12 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHERI L. HARRINGTON whose telephone number is (571)270-0468. The examiner can normally be reached Generally, M-F, 7:30a-4p.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jaweed Abbaszadeh can be reached at 571-270-1640. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHERI L HARRINGTON/Examiner, Art Unit 2176 March 15, 2026
/JAWEED A ABBASZADEH/Supervisory Patent Examiner, Art Unit 2176