Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12 November 2025 has been entered
Response to Arguments
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the arguments do not apply to any of the references being used in the current rejection.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(B) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of pre-AIA 35 U.S.C. 112, second paragraph::
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-8, 11-14, and 16-19 are rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
The independent claims 1, 11, and 16 recite “a first hardware device” and “a second hardware device.” However, dependent claims 2-6, 12-14, and 17-19 recite “the original hardware device” and “the new hardware device” without providing any antecedent basis for those terms. Because “original hardware device” and “new hardware device” are not disclosed in the independent claim, the scope of the dependent claims is indefinite.
Claims 7-8 are additionally rejected because they depend from claims with defective antecedent basis and therefore inherit the indefiniteness of their parent claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made
Claims 1, 2, 4-11, 13-16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Bernat et al. (US 2020/0409748, hereinafter Bernat) in view of Zhao et al. (US 2022/0214902, hereinafter Zhao) and Krishnamurthy et al. (US 2012/0054771, hereinafter Krishnamurthy).
Regarding claim 1, Bernat discloses
An apparatus comprising:
a hardware processing device communicably coupled to a plurality of heterogeneous hardware accelerator devices, the hardware processing device processor to execute a data parallel programming runtime that is to manage programming of the plurality of heterogeneous hardware accelerator devices and is further to (paragraph [0119]: the orchestrator server 1616, in operation, may perform a method 2600 for scaling-out accelerator kernel resources to heterogeneous accelerator devices paragraph [0122]: a kernel may be created using the OpenCL framework, which allows the kernel to be compiled into a bit stream compatible with a desired device):
identify a change in device status that triggers a device transfer process from a first hardware device of the plurality of heterogeneous hardware accelerator devices, wherein the first hardware device (paragraph [0091]: the affected accelerator device may be cloned as a device instance and migrated to newly available accelerator resources to another accelerator sled; paragraph [0120]: As shown, the method 2600 begins in block 2602, in which the orchestrator server 1616 detects a trigger to initiate a scale-out operation of one or more kernels associated with a workload. An example trigger includes a determination that resource usage in a given accelerator device or sled is relatively high) is associated with a queue of an application program of the data parallel programming runtime (paragraph [0110]: Doing so results in requests targeted to the accelerator kernel to remain in egress queues of requesting devices; paragraph [0122]: a kernel may be created using the OpenCL framework, which allows the kernel to be compiled into a bit stream compatible with a desired device);
migrate at least one of a state or data of the first hardware device to the second hardware device (paragraph [0122]: In block 2608, the orchestrator server 1616 migrates one or more of the accelerator kernels associated with the workload to the accelerator devices identified in the configuration);
logically map, without user intervention, the queue to the second hardware device in the data parallel programming runtime (paragraph [0110]: in block 2212, the orchestrator server 1616 may cause the source accelerator to suspend operation of the accelerator kernel. Doing so results in requests targeted to the accelerator kernel to remain in egress queues of requesting device; paragraph [0122]: In block 2608, the orchestrator server 1616 migrates one or more of the accelerator kernels associated with the workload to the accelerator devices identified in the configuration); and
initiate execution of the application program on the second hardware device using the queue (Fig. 15 Application A 1532; paragraph [0031]: Referring now to FIG. 1, a data center 100 in which disaggregated resources may cooperatively execute one or more workloads (e.g., applications on behalf of customers) includes multiple pods 110, 120, 130, 140; paragraph [0122]: In block 2608, the orchestrator server 1616 migrates one or more of the accelerator kernels associated with the workload to the accelerator devices identified in the configuration … In block 2614, the orchestrator server 1616 initializes inter-kernel communication channels between the kernels associated with the workload. Doing so allows the kernels associated with the workload to send data to one another).
Bernat does not teach analyze resource intersections between the first hardware device and a second hardware device of the plurality of heterogeneous hardware accelerator devices, wherein the resource intersections comprise at least resource intersections between device capabilities of the first hardware device and the second hardware device, and wherein the device capabilities comprise at least one of features or attributes of the first hardware device or second hardware device; determine that the second hardware device is compatible with the first hardware device based on the resource intersections analyzed between the second hardware device and the first hardware device. Zhao teaches analyze resource intersections between the first hardware device and a second hardware device of the plurality of heterogeneous hardware accelerator devices, wherein the resource intersections comprise at least resource intersections between device capabilities of the first hardware device and the second hardware device, and wherein the device capabilities comprise at least one of features or attributes of the first hardware device or second hardware device; determine that the second hardware device is compatible with the first hardware device based on the resource intersections analyzed between the second hardware device and the first hardware device (paragraph [0078]: A target host, e.g. HOST-T 451, can be of a same or similar hardware and software configuration as HOST-S 401. Accelerators 410 and accelerators 460 should be of a same or similar type, such as having a compatible instruction set for their respective processors. The HOST-T 451 should have, quantitatively, sufficient resources available as may be required by VM-S so that VM1-S may be migrated to VM1-T. Qualitatively, HOST-S 401 and HOST T-451 should have compatible operating hardware and software. For example, HOST-S 401 accelerators 410 may be of a same manufacturer, and a compatible model, as the accelerators ACC2 460 on HOST-T 451, else the migration may not be successful; Note: Zhao teaches that successful migration depends on selecting destination devices with compatible capabilities and sufficient resources). It would have been obvious to one of ordinary skill in the art at the time the claimed invention was effectively filed to incorporate Zhao’s compatibility analysis between accelerators for migration into Bernat’s orchestrator workflow to select destination accelerators whose capabilities intersect with those of the source device (e.g., instruction set compatibility, resource sufficiency, and compatible hardware/software) before migrating kernels or state to ensure successful execution. The motivation would have been to enable successful migration of VMs between source host and destination host (Zhao paragraph [0078]).
Bernat in view of Zhao does not teach logically map, without user intervention, the queue to the second hardware device in the data parallel programming runtime. Krishnamurthy teaches logically map, without user intervention, the queue to the second hardware device in the data parallel programming runtime (paragraph [0033]: the hybrid computing environment 100 implements a cross-platform parallel programming environment such as, but not limited to, an OpenCL (Open Compute Language) environment; paragraph [0042]: the workload manager 118, at T2, migrates the workload 304 from the accelerator 104 to a compute core (or accelerator) 310 at the server system 102 where the workload 304 can perform its operations with the required data at the server system 102; claim 3: mapping the set of queues to a memory that is shared between the at least one server system and the set of accelerator systems). It would have been obvious to one of ordinary skill in the art at the time the claimed invention was effectively filed to Krishnamurthy’s queue-to-shared-memory mapping after migrating the workload from a accelerator to the other accelerator with Bernat (in view of Zhao)’s orchestrator to effect automatic logical remapping of the queues to the destination accelerator within the runtime, enabling continued processing without user input. The motivation would have been to move workloads to locations where data exchange is most efficient, secure, and/or where the data is available (Krishnamurthy paragraph [0044]).
Regarding claim 11 referring to claim 1, Bernat discloses A method comprising: … (See the rejection for claim 1).
Regarding claim 16 referring to claim 1, Bernat discloses A non-transitory machine readable storage medium comprising instructions that, when executed, cause at least one processor to at least: … (paragraph [0029]: The disclosed embodiments may also be implemented as instructions carried by or stored on a transitory or non-transitory machine-readable (e.g., computer-readable) storage medium, which may be read and executed by one or more processors).
Regarding claim 2, Bernat discloses
wherein the change in the device status comprises a change in device status of the original hardware device (paragraph [0120]: As shown, the method 2600 begins in block 2602, in which the orchestrator server 1616 detects a trigger to initiate a scale-out operation of one or more kernels associated with a workload. An example trigger includes a determination that resource usage in a given accelerator device or sled is relatively high).
Regarding claim 4, Bernat discloses
wherein the change in device status comprises a change in device status of the new hardware device (paragraph [0113]: To improve latency of the current workload, the affected accelerator device may be cloned as a device instance and migrated to newly available accelerator resources; paragraph [0121]: In block 2604, the orchestrator server 1616 determines, as a function of a policy (e.g., a SLA, QoS requirements, a load balancing policy, user-defined specifications in an application associated with the workload, etc.), one or more accelerator devices. Each accelerator device may correspond to one of multiple types (e.g., an FPGA, VPU, ASIC, GPU, etc.). Further, in block 2606, the orchestrator server 1616 may determine a configuration of accelerator devices to satisfy one or more QoS requirements, in which the configuration specifies accelerator devices of differing types; paragraph [0122]: the orchestrator server 1616 generates, from each kernel to be scaled-out to a given accelerator device, a corresponding kernel bit stream that is compatible with the accelerator device).
Regarding claims 5, 13, and 18, Bernat discloses
wherein the change in the device status of the new hardware device comprises at least one of adding the new hardware device to a computing environment of the data parallel programming runtime, higher performance of the new hardware device as compared to the original hardware device, or higher performance per power of the new hardware device as compared to the original hardware device (paragraph [0113]: To improve latency of the current workload, the affected accelerator device may be cloned as a device instance and migrated to newly available accelerator resources; paragraph [0121]: In block 2604, the orchestrator server 1616 determines, as a function of a policy (e.g., a SLA, QoS requirements, a load balancing policy, user-defined specifications in an application associated with the workload, etc.), one or more accelerator devices. Each accelerator device may correspond to one of multiple types (e.g., an FPGA, VPU, ASIC, GPU, etc.). Further, in block 2606, the orchestrator server 1616 may determine a configuration of accelerator devices to satisfy one or more QoS requirements, in which the configuration specifies accelerator devices of differing types; paragraph [0122]: a kernel may be created using the OpenCL framework, which allows the kernel to be compiled into a bit stream compatible with a desired device).
Regarding claims 6, 14, and 19, Bernat discloses
wherein the analysis of the resource intersections comprises analyzing the resources intersections of at least one of: device capabilities of the original hardware device and the new hardware device, compatible context between the original hardware device and the new hardware device, or architecture class of the original hardware device the new hardware device (paragraph [0113]: To improve latency of the current workload, the affected accelerator device may be cloned as a device instance and migrated to newly available accelerator resources; paragraph [0121]: In block 2604, the orchestrator server 1616 determines, as a function of a policy (e.g., a SLA, QoS requirements, a load balancing policy, user-defined specifications in an application associated with the workload, etc.), one or more accelerator devices. Each accelerator device may correspond to one of multiple types (e.g., an FPGA, VPU, ASIC, GPU, etc.). Further, in block 2606, the orchestrator server 1616 may determine a configuration of accelerator devices to satisfy one or more QoS requirements, in which the configuration specifies accelerator devices of differing types; paragraph [0122]: a kernel may be created using the OpenCL framework, which allows the kernel to be compiled into a bit stream compatible with a desired device).
Regarding claim 7, Bernat discloses
… at least one of device type, device make, device model, device features, device attributes, or device telemetry metrics (paragraph [0107]: each kernel may be associated with telemetry monitors; paragraph [0113]: To improve latency of the current workload, the affected accelerator device may be cloned as a device instance and migrated to newly available accelerator resources; paragraph [0121]: In block 2604, the orchestrator server 1616 determines, as a function of a policy (e.g., a SLA, QoS requirements, a load balancing policy, user-defined specifications in an application associated with the workload, etc.), one or more accelerator devices. Each accelerator device may correspond to one of multiple types (e.g., an FPGA, VPU, ASIC, GPU, etc.). Further, in block 2606, the orchestrator server 1616 may determine a configuration of accelerator devices to satisfy one or more QoS requirements, in which the configuration specifies accelerator devices of differing types; paragraph [0122]: a kernel may be created using the OpenCL framework, which allows the kernel to be compiled into a bit stream compatible with a desired device).
Bernat does not teach wherein the at least one of the feature or the attributes comprise at least one of device type, device make, device model, device features, device attributes, or device telemetry metrics. Zhao teaches wherein the at least one of the feature or the attributes comprise at least one of device type, device make, device model, device features, device attributes, or device telemetry metrics (paragraph [0078]: A target host, e.g. HOST-T 451, can be of a same or similar hardware and software configuration as HOST-S 401. Accelerators 410 and accelerators 460 should be of a same or similar type, such as having a compatible instruction set for their respective processors. The HOST-T 451 should have, quantitatively, sufficient resources available as may be required by VM-S so that VM1-S may be migrated to VM1-T. Qualitatively, HOST-S 401 and HOST T-451 should have compatible operating hardware and software. For example, HOST-S 401 accelerators 410 may be of a same manufacturer, and a compatible model, as the accelerators ACC2 460 on HOST-T 451, else the migration may not be successful; Note: Zhao teaches that successful migration depends on selecting destination devices with compatible capabilities and sufficient resources). It would have been obvious to one of ordinary skill in the art at the time the claimed invention was effectively filed to incorporate Zhao’s compatibility analysis between accelerators for migration into Bernat’s orchestrator workflow to select destination accelerators whose capabilities intersect with those of the source device (e.g., instruction set compatibility, resource sufficiency, and compatible hardware/software) before migrating kernels or state to ensure successful execution. The motivation would have been to enable successful migration of VMs between source host and destination host (Zhao paragraph [0078]).
Regarding claim 8, Bernat discloses
wherein the device telemetry metrics comprise at least one of power consumption, load, resource utilization, memory utilization, network bandwidth, network throughput, or network latency (paragraph [0082]: the orchestrator server 1520 may receive telemetry data indicative of performance conditions (e.g., throughput, latency, instructions per second, etc.) in each sled 400 of the managed node 1570 and compare the telemetry data to the quality of service targets to determine whether the quality of service targets are being satisfied; paragraph [0083]: the orchestrator server 1520 may identify trends in the resource utilization of the workload (e.g., the application 1532) … the orchestrator server 1520 may utilize a model that accounts for the performance of resources on the sleds 400 (e.g., FPGA performance, memory access latency, etc.) and the performance (e.g., congestion, latency, bandwidth) of the path through the network to the resource (e.g., FPGA).; paragraph [0089]: The kernel analysis and decision logic unit 1617 may be embodied as any device or circuitry to obtain telemetry data indicative of resource usage and power consumption of the accelerator sleds 1610, 1612 and the compute sled 1617).
Regarding claim 9, Bernat discloses
wherein the plurality of heterogeneous hardware accelerator devices comprise at least one a graphic processing unit (GPU), a central processing unit (CPU), or a programmable integrated circuit (IC) (paragraph [0121]: Each accelerator device may correspond to one of multiple types (e.g., an FPGA, VPU, ASIC, GPU, etc.)).
Regarding claim 10, Bernat discloses
wherein the programmable IC comprises at least one of a field programmable gate array (FPGA), a programmable array logic (PAL), a programmable logic array (PLA), a field programmable logic array (FPLA), an electrically programmable logic device (EPLD), an electrically erasable programmable logic device (EEPLD), a logic cell array (LCA), or a complex programmable logic devices (CPLD) (paragraph [0121]: Each accelerator device may correspond to one of multiple types (e.g., an FPGA, VPU, ASIC, GPU, etc.)).
Regarding claims 15 and 20, Bernat discloses
… at least one of device type, device make, device model, device features, device attributes, or device telemetry metrics (paragraph [0107]: each kernel may be associated with telemetry monitors; paragraph [0113]: To improve latency of the current workload, the affected accelerator device may be cloned as a device instance and migrated to newly available accelerator resources; paragraph [0121]: In block 2604, the orchestrator server 1616 determines, as a function of a policy (e.g., a SLA, QoS requirements, a load balancing policy, user-defined specifications in an application associated with the workload, etc.), one or more accelerator devices. Each accelerator device may correspond to one of multiple types (e.g., an FPGA, VPU, ASIC, GPU, etc.). Further, in block 2606, the orchestrator server 1616 may determine a configuration of accelerator devices to satisfy one or more QoS requirements, in which the configuration specifies accelerator devices of differing types; paragraph [0122]: a kernel may be created using the OpenCL framework, which allows the kernel to be compiled into a bit stream compatible with a desired device), and wherein the device telemetry metrics comprise at least one of power consumption, load, resource utilization, memory utilization, network bandwidth, network throughput, or network latency (paragraph [0082]: the orchestrator server 1520 may receive telemetry data indicative of performance conditions (e.g., throughput, latency, instructions per second, etc.) in each sled 400 of the managed node 1570 and compare the telemetry data to the quality of service targets to determine whether the quality of service targets are being satisfied; paragraph [0083]: the orchestrator server 1520 may identify trends in the resource utilization of the workload (e.g., the application 1532) … the orchestrator server 1520 may utilize a model that accounts for the performance of resources on the sleds 400 (e.g., FPGA performance, memory access latency, etc.) and the performance (e.g., congestion, latency, bandwidth) of the path through the network to the resource (e.g., FPGA).; paragraph [0089]: The kernel analysis and decision logic unit 1617 may be embodied as any device or circuitry to obtain telemetry data indicative of resource usage and power consumption of the accelerator sleds 1610, 1612 and the compute sled 1617).
Bernat does not teach wherein the at least one of the feature or the attributes comprise at least one of device type, device make, device model, device features, device attributes, or device telemetry metrics. Zhao teaches wherein the at least one of the feature or the attributes comprise at least one of device type, device make, device model, device features, device attributes, or device telemetry metrics (paragraph [0078]: A target host, e.g. HOST-T 451, can be of a same or similar hardware and software configuration as HOST-S 401. Accelerators 410 and accelerators 460 should be of a same or similar type, such as having a compatible instruction set for their respective processors. The HOST-T 451 should have, quantitatively, sufficient resources available as may be required by VM-S so that VM1-S may be migrated to VM1-T. Qualitatively, HOST-S 401 and HOST T-451 should have compatible operating hardware and software. For example, HOST-S 401 accelerators 410 may be of a same manufacturer, and a compatible model, as the accelerators ACC2 460 on HOST-T 451, else the migration may not be successful; Note: Zhao teaches that successful migration depends on selecting destination devices with compatible capabilities and sufficient resources). It would have been obvious to one of ordinary skill in the art at the time the claimed invention was effectively filed to incorporate Zhao’s compatibility analysis between accelerators for migration into Bernat’s orchestrator workflow to select destination accelerators whose capabilities intersect with those of the source device (e.g., instruction set compatibility, resource sufficiency, and compatible hardware/software) before migrating kernels or state to ensure successful execution. The motivation would have been to enable successful migration of VMs between source host and destination host (Zhao paragraph [0078]).
Claims 3, 12, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Bernat et al. (US 2020/0409748, hereinafter Bernat) in view of Zhao et al. (US 2022/0214902, hereinafter Zhao) and Krishnamurthy et al. (US 2012/0054771, hereinafter Krishnamurthy) as applied to claims 1, 11, and 16, and further in view of Xing et al. (US 2023/0367648, hereinafter Xing).
Regarding claims 3, 12, and 17, Bernat in view of Zhao and Krishnamurthy does not teach wherein the change in the device status of the original hardware device comprises at least one of de-provisioning of the original hardware device from a computing environment of the data parallel programming runtime or removal of the original hardware device from the computing environment. Xing teaches wherein the change in the device status of the original hardware device comprises at least one of de-provisioning of the original hardware device from a computing environment of the data parallel programming runtime or removal of the original hardware device from the computing environment (paragraph [0120]: the second container may run on any edge device, and the second container may be run by any accelerator, without any necessity of developing different applications for different edge devices or different accelerators or configuring a running environment for the target application on edge devices; paragraph [0126]: after the accelerator on the second edge device is removed, the control server may migrate the second container on the second edge device to other edge devices). It would have been obvious to one of ordinary skill in the art at the time the claimed invention was effectively filed to modify the teaching of Hansen in view of Bernat in view of Zhao and Krishnamurthy by incorporating Xing’s teaching of migrating the second container running by accelerator on the second edge device to run the second container, by any accelerator, without any necessity of developing different applications for different edge devices or different accelerators or configuring a running environment for the target application on edge devices (Xing paragraph [0120]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SISLEY N. KIM whose telephone number is (571)270-7832. The examiner can normally be reached M-F 11:30AM -7:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, April Y. Blair can be reached on (571)270-1014. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SISLEY N KIM/Primary Examiner, Art Unit 2196 12/26/2025