DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
In view of Applicant’s amendment, the objections are withdrawn.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: “hardware logic” in claims 1, 6-12, 17-20, 25-27.
Because this claim limitation is being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it is being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof in paragraphs 00341, 00356, and 00406 of the specification.
If applicant does not intend to have this limitation interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation to avoid it being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation recites sufficient structure to perform the claimed function so as to avoid it being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
In view of Applicant’s amendment, the previous 112(a) and 112(b) rejections are withdrawn. However, see the new 112(b) rejection below in view of the amendment.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 9 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 9 recites the limitation “the virtualization hardware logic”. There is insufficient antecedent basis for this limitation in the claim.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-7, 12-18, and 20-26 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-7, 9-15, and 17-23 of copending Application No. 17/881,540 (“‘540“) in view of Shah et al. US 20200410628 A1 (“Shah”), further in view of Zada et al. US 20150154039 A1 (“Zada”) and Coon et al. US 20090240860 (“Coon”). This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Although the claims at issue are not identical, they are not patentably distinct from each other because ‘540 is commensurate in scope and where it differs it is narrower and anticipates the instant claims as discussed in the art rejections below.
Claim Rejections - 35 USC § 101
In view of Applicant’s amendment, the 101 rejection is withdrawn.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 9, 11, 15, 20, and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Shah et al. US 20200410628 A1 (“Shah”) in view of Zada et al. US 20150154039 A1 (“Zada”) and Coon et al. US 20090240860 (“Coon”).
Regarding claim 1, Shah teaches An apparatus comprising: first hardware logic ([0001]: “This invention relates generally to the field of graphics processors. More particularly, the invention relates to an apparatus and method for provisioning virtualized multi-tile graphics processing hardware.”) comprising parallel execution resources to concurrently execute a number of workloads; ([0069]: “In one embodiment, the execution units additionally include general-purpose logic that is programmable to perform parallel general purpose computational operations, in addition to graphics processing operations. The general purpose logic can perform processing operations in parallel or in conjunction with general purpose logic within the processor core(s) 107 of FIG. 1 or core 202A-202N as in FIG. 2.”; [0244]: “In current virtualized graphics processing implementations, system administrators configure and partition the graphics processing resources such that they can meet the requirements of each virtual machine. These requirements may include a guaranteed quality of service, virtual machine isolation, and guaranteed latency requirements for each assigned workload.”; Examiner notes, execution resources are used to execute workloads.)
second hardware logic to allocate the parallel execution resources between a number of virtual machines, ([0148]: “In addition, in one embodiment, the virtualization mediator 1622 implements a GPU scheduler 1626, which runs concurrently with the CPU scheduler 1616 in the hypervisor 1610, to share the physical GPU 1600 among the VMs 1631-1632.”; [0251]: “In one embodiment, each tile 2591-2594 comprises a scheduler (e.g., a GuC scheduler) 2580 to schedule access by virtual machines to the processing resources.”) each virtual machine to execute a workload on its allocated portion of the execution resources concurrently with workloads executed by one or more other virtual machines executed on corresponding other allocated portions of the execution resources; ([0176]: “GPU scheduler 1712 for GPU 1720 may be separated from the scheduler for CPU in apparatus 1700. To take the advantage of the hardware parallelism in some embodiments, GPU scheduler 1712 may schedule the workloads separately for different GPU engines, e.g., 3D render engine 1722, blitter render engine 1724, video command streamer (VCS) render engine 1726, and video enhanced command streamer (VECS) render engine 1728. For example, VM 1730 may be 3D intensive, and 3D workload 1732 may need to be scheduled to 3D render engine 1722 at a moment. Meanwhile, VM 1740 may be media intensive, and media workload 1744 may need to be scheduled to VCS render engine 1726 and/or VECS render engine 1728. In this case, GPU scheduler 1712 may schedule 3D workload 1732 from VM 1730 and media workload 1744 from VM 1740 separately.”)
Shah does not teach and programmable performance monitoring circuitry to be dynamically partitioned based on the number of virtual machines and a portion of the execution resources allocated to each virtual machine, the programmable performance monitoring circuitry to differentiate between performance monitoring data of different virtual machines based on one or more unique identifiers associated with each of the allocated portions of execution resources.
However, in analogous art, Zada teaches and programmable performance monitoring circuitry to be dynamically partitioned based on the number of virtual machines and a portion of the execution resources allocated to each virtual machine, (Abstract: “…automatically installing an agent on the virtual machine when the virtual machine is identified as having a designated configuration, automatically identifying, via the agent, a resource associated with the virtual machine and, based on the resource, applying a monitoring policy to the virtual machine…”; [0045]: “The example system 100 of FIG. 1 may include multiple performance monitoring servers 106 to monitor agents 132 that are configured differently based on, for example, the resource(s) being monitored by the respective agents 132. Multiple performance monitoring servers 106 may additionally or alternatively be used to monitor different deployment environments 104, different applications 102, and/or different VMs 114. The example agents 132 may be initially configured by the agent installer 126 to communicate with a designated one of the performance monitoring servers 106.”; Fig. 11 – Examiner notes, agent installation in progress, allocation of agents can be changed. Further, “resources” in Zada refer to both software and hardware resources (see at least [0030] and [0032]).) the programmable performance monitoring circuitry to differentiate between performance monitoring data of different virtual machines based on one or more unique identifiers associated with each of the allocated portions of execution resources. (Fig. 5; [0071]: “The example agent monitor 130 of FIG. 1 receives resource report(s) for the new VM(s) from the installed agents (block 312). The resource reports may be generated by the agents 132 of FIG. 1 periodically, aperiodically, and/or in response to an event (e.g., on initial installation and/or execution of the agents 132). The resource reports list the resources and/or provide information about the resources (e.g., platform, servers, services, etc.) executing on the VMs 114. Based on the resource report(s) from the agents 132, the example agent monitor 130 registers the identified resources of the VMs 114 (block 314). For example, the agent monitor 130 may add the resources to an inventory and/or include the resources in a listing or catalog of resources available for monitoring.”)
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the monitoring circuitry for monitoring software and hardware resources of a VM of Zada, with the virtualized tiled-graphics processing hardware system of Shah. As a result, Shah would utilize agents partitioned to each VM, to monitor their specific configuration of execution resources that have been allocated to them. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to more effectively monitor the resources used by virtual machines, as Zada states in [0020]: “Cloud environments are dynamic by nature. Resources, such as virtual machines…may be created, used, moved and/or destroyed with relatively high frequency. In such dynamic environments, monitoring the resources (e.g., virtual machines, applications, services, etc.) can be challenging because, among other things, new resources are often created, resources change their location, and/or the configurations of the resources change. Such changes result in changes to monitoring requirements for the resources.” Therefore, with Zada’s teachings, Shah would more effectively automate the monitoring of resources and responding to necessary changes during a virtual environment’s operation (see also at least [0021]-[0022]).
The combination of Shah/Zada does not teach the one or more unique identifiers comprising an identifier of a respective hardware context that comprises instruction processing and data processing resources to perform a corresponding portion of single-instruction multiple thread (SIMT) or single instruction multiple data (SIMD) operations.
Coon teaches the one or more unique identifiers comprising an identifier of a respective hardware context that comprises instruction processing and data processing resources to perform a corresponding portion of single-instruction multiple thread (SIMT) or single instruction multiple data (SIMD) operations (paragraph 0043).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine Shah/Zada’s processing system with Coon’s processing engine. The motivation for doing so would have been to provide improved locking mechanism for accessing shared memory.
Regarding claim 9, Shah in view of Zada and Coon teaches the apparatus of claim 1.
Shah in view of Zada and Coon further teaches wherein the first hardware logic is integrated on a tile (Shah, [0245]: “One embodiment of the invention comprises a highly configurable graphics processing architecture which includes tile-based partitioning and single-root input-output virtualization (SR-IOV). This embodiment may be implemented in virtualization software within a graphics driver and/or configuration circuitry within the graphics processor. Specifically, one embodiment of the invention utilizes a hardware-based SR-IOV implementation in combination with multi-tile graphics processing hardware and firmware-based scheduling to create a highly configurable virtualized graphics system.”)
comprising the programmable performance monitoring circuitry, (Zada, [0045]: “The example system 100 of FIG. 1 may include multiple performance monitoring servers 106 to monitor agents 132 that are configured differently based on, for example, the resource(s) being monitored by the respective agents 132. Multiple performance monitoring servers 106 may additionally or alternatively be used to monitor different deployment environments 104, different applications 102, and/or different VMs 114. The example agents 132 may be initially configured by the agent installer 126 to communicate with a designated one of the performance monitoring servers 106.”;”) wherein the virtualization hardware logic is to virtualize the programmable performance monitoring circuitry (Zada, [0003]: “’Infrastructure-as-a-Service’ (also commonly referred to as "IaaS") generally describes a suite of technologies provided by a service provider as an integrated solution to allow for elastic creation of a virtualized, networked, and pooled computing platform (sometimes referred to as a ‘cloud computing platform’).”; Examiner notes, Shah in view of Zada and Coon installs software agents on a virtual machine, operating in a virtualized environment, thus the agents (monitoring circuitry) are virtualized).
to allocate a first portion of the programmable performance monitoring circuitry and a first processor instance on the tile to a first virtual machine and to allocate a second portion of the programmable performance monitoring circuitry and a second processor instance on the tile to a second virtual machine. (Shah, Fig. 27, [0261]: “Referring to FIG. 27, the graphics processor 2590 is again subdivided into four tiles 2591-2594 in this embodiment. However, instead of mapping each tile to a single VM, each tile is shared among a plurality of VMs. For example, tile 2594 (“Tile 3”) is illustrated being time shared among a set of VMs 2703 and tile 2593 (“Tile 2”) is time shared by another set of VMs 2702. “; Examiner notes, various virtual machines are allocated to each tile, therefore at least a second virtual machine. Furthermore, see at least Fig. 26 and [0256] for an embodiment where one VM may be assigned to only one tile; Zada, [Abstract]: “…automatically installing an agent on the virtual machine when the virtual machine is identified as having a designated configuration, automatically identifying, via the agent, a resource associated with the virtual machine and, based on the resource, applying a monitoring policy to the virtual machine…”; Examiner notes, agents are automatically installed on each VM with a desired configuration, therefore the number of virtual machines being assigned to tiles will have monitoring agents (monitoring circuitry)).
Regarding claim 11, Shah in view of Zada and Coon teaches the apparatus of claim 1.
Shah in view of Zada and Coon further teaches wherein the second hardware logic is to allocate the parallel execution resources of the first hardware logic by assigning each virtual machine a time quantum in which a virtual machine is provided access to the parallel execution resources of the first hardware logic and the programmable performance monitoring circuitry, (Shah, Fig. 28; [0265]: “As mentioned, in one embodiment, each tile 2591-2594 includes an integrated VM scheduler 2580 (e.g., referred to as the tile's GuC in certain implementations) which will perform the VM context switching in accordance with each VM's time quantum. As illustrated FIG. 28, each tile can be loaded differently and each tile scheduler can switch VMs at different rates. In the illustrated example, Scheduler 0 2820 schedules three VMs on Tile 0, where VM1 is allocated a 50% time quantum, VM2 is allocated a 25% time quantum, and VM3 is allocated a 25% time quantum. Scheduler 1 2821 schedules one VM on Tile 1 using a 50% time quantum with the other 50% time quantum being unused. Scheduler 2 2822 schedules two VMs on Tile 2, with each VM allocated a 50% time quantum. Scheduler 3 of Tile 3 does not schedule any VMs (0% tile utilization).”; Examiner notes, Shah in view of Zada and Coon has the monitoring agents installed on the number of virtual machines, thus during its own time quantum, it will have access to the monitoring agent as well. Furthermore, see also at least Shah, Fig. 30 and [0266]-[0271] for an additional embodiment of time quantum-based sharing on tiles.)
wherein at the end of the time quantum, the performance monitoring data collected by the programmable performance monitoring circuitry is to be saved to a memory and performance monitoring data associated with a new virtual machine is to be restored from memory, (Shah, Fig. 28 – 2850; [0265]: “As illustrated, there is a switching overhead 2850 associated with the context switching performed by the schedulers 2580 (e.g., to save the current execution state of the prior VM and restore the execution state of the next VM to be executed on the tile).”; Examiner notes, Shah in view of Zada and Coon also discloses storing the information discovered by the monitoring agents (see at least Zada, [0066]), thus when the execution state is saved, the resource monitoring data would be saved as well, as the agents are a part of the VM.)
the new virtual machine to be run and provided with access to the parallel execution resources and the programmable performance monitoring circuitry, including the restored performance monitoring data, for a new time quantum. (Shah, Fig. 28; [0265]: “As mentioned, in one embodiment, each tile 2591-2594 includes an integrated VM scheduler 2580 (e.g., referred to as the tile's GuC in certain implementations) which will perform the VM context switching in accordance with each VM's time quantum. As illustrated FIG. 28, each tile can be loaded differently and each tile scheduler can switch VMs at different rates. In the illustrated example, Scheduler 0 2820 schedules three VMs on Tile 0, where VM1 is allocated a 50% time quantum, VM2 is allocated a 25% time quantum, and VM3 is allocated a 25% time quantum. Scheduler 1 2821 schedules one VM on Tile 1 using a 50% time quantum with the other 50% time quantum being unused. Scheduler 2 2822 schedules two VMs on Tile 2, with each VM allocated a 50% time quantum. Scheduler 3 of Tile 3 does not schedule any VMs (0% tile utilization). As illustrated, there is a switching overhead 2850 associated with the context switching performed by the schedulers 2580 (e.g., to save the current execution state of the prior VM and restore the execution state of the next VM to be executed on the tile).”; Examiner notes, Shah in view of Zada and Coon has the monitoring agents installed on the number of virtual machines, during its own time quantum, it will have access to the monitoring agent as well, as its execution state would be restored during its own time quantum. Furthermore, see also at least Shah, Fig. 30 and [0266]-[0271] for an additional embodiment of time quantum-based sharing on tiles).
Regarding claim 12, Shah teaches A method comprising: allocating ([0001]: “This invention relates generally to the field of graphics processors. More particularly, the invention relates to an apparatus and method for provisioning virtualized multi-tile graphics processing hardware.”) parallel execution resources of first hardware logic between a number of virtual machines by second hardware logic; ([0148]: “In addition, in one embodiment, the virtualization mediator 1622 implements a GPU scheduler 1626, which runs concurrently with the CPU scheduler 1616 in the hypervisor 1610, to share the physical GPU 1600 among the VMs 1631-1632.”; [0251]: “In one embodiment, each tile 2591-2594 comprises a scheduler (e.g., a GuC scheduler) 2580 to schedule access by virtual machines to the processing resources.”; [0069]: “In one embodiment, the execution units additionally include general-purpose logic that is programmable to perform parallel general purpose computational operations, in addition to graphics processing operations. The general purpose logic can perform processing operations in parallel or in conjunction with general purpose logic within the processor core(s) 107 of FIG. 1 or core 202A-202N as in FIG. 2.”)
executing a workload of each virtual machine on its allocated portion of the execution resources concurrently with workloads executed by one or more other virtual machines executed on corresponding other allocated portions of the execution resources; ([0176]: “GPU scheduler 1712 for GPU 1720 may be separated from the scheduler for CPU in apparatus 1700. To take the advantage of the hardware parallelism in some embodiments, GPU scheduler 1712 may schedule the workloads separately for different GPU engines, e.g., 3D render engine 1722, blitter render engine 1724, video command streamer (VCS) render engine 1726, and video enhanced command streamer (VECS) render engine 1728. For example, VM 1730 may be 3D intensive, and 3D workload 1732 may need to be scheduled to 3D render engine 1722 at a moment. Meanwhile, VM 1740 may be media intensive, and media workload 1744 may need to be scheduled to VCS render engine 1726 and/or VECS render engine 1728. In this case, GPU scheduler 1712 may schedule 3D workload 1732 from VM 1730 and media workload 1744 from VM 1740 separately.”)
Shah does not teach and dynamically partitioning programmable performance monitoring circuitry based on the number of virtual machines and a portion of the execution resources allocated to each virtual machine, the programmable performance monitoring circuitry to differentiate between performance monitoring data of different virtual machines based on one or more unique identifiers associated with each of the allocated portions of execution resources.
However, in analogous art, Zada teaches and dynamically partitioning programmable performance monitoring circuitry based on the number of virtual machines and a portion of the execution resources allocated to each virtual machine, (Abstract: “…automatically installing an agent on the virtual machine when the virtual machine is identified as having a designated configuration, automatically identifying, via the agent, a resource associated with the virtual machine and, based on the resource, applying a monitoring policy to the virtual machine…”; [0045]: “The example system 100 of FIG. 1 may include multiple performance monitoring servers 106 to monitor agents 132 that are configured differently based on, for example, the resource(s) being monitored by the respective agents 132. Multiple performance monitoring servers 106 may additionally or alternatively be used to monitor different deployment environments 104, different applications 102, and/or different VMs 114. The example agents 132 may be initially configured by the agent installer 126 to communicate with a designated one of the performance monitoring servers 106.”; Fig. 11 – Examiner notes, agent installation in progress, allocation of agents can be changed. Further, “resources” in Zada refer to both software and hardware resources (see at least [0030] and [0032]).) the programmable performance monitoring circuitry to differentiate between performance monitoring data of different virtual machines based on one or more unique identifiers associated with each of the allocated portions of execution resources. (Fig. 5; [0071]: “The example agent monitor 130 of FIG. 1 receives resource report(s) for the new VM(s) from the installed agents (block 312). The resource reports may be generated by the agents 132 of FIG. 1 periodically, aperiodically, and/or in response to an event (e.g., on initial installation and/or execution of the agents 132). The resource reports list the resources and/or provide information about the resources (e.g., platform, servers, services, etc.) executing on the VMs 114. Based on the resource report(s) from the agents 132, the example agent monitor 130 registers the identified resources of the VMs 114 (block 314). For example, the agent monitor 130 may add the resources to an inventory and/or include the resources in a listing or catalog of resources available for monitoring.”)
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the monitoring circuitry for monitoring software and hardware resources of a VM of Zada, with the virtualized tiled-graphics processing hardware system of Shah. As a result, Shah would utilize agents partitioned to each VM, to monitor their specific configuration of execution resources that have been allocated to them. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to more effectively monitor the resources used by virtual machines, as Zada states in [0020]: “Cloud environments are dynamic by nature. Resources, such as virtual machines…may be created, used, moved and/or destroyed with relatively high frequency. In such dynamic environments, monitoring the resources (e.g., virtual machines, applications, services, etc.) can be challenging because, among other things, new resources are often created, resources change their location, and/or the configurations of the resources change. Such changes result in changes to monitoring requirements for the resources.” Therefore, with Zada’s teachings, Shah would more effectively automate the monitoring of resources and responding to necessary changes during a virtual environment’s operation (see also at least [0021]-[0022]).
The combination of Shah/Zada does not teach the one or more unique identifiers comprising an identifier of a respective hardware context that comprises instruction processing and data processing resources to perform a corresponding portion of single-instruction multiple thread (SIMT) or single instruction multiple data (SIMD) operations.
Coon teaches the one or more unique identifiers comprising an identifier of a respective hardware context that comprises instruction processing and data processing resources to perform a corresponding portion of single-instruction multiple thread (SIMT) or single instruction multiple data (SIMD) operations (paragraph 0043).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine Shah/Zada’s processing system with Coon’s processing engine. The motivation for doing so would have been to provide improved locking mechanism for accessing shared memory.
Regarding claim 15, Shah in view of Zada and Coon teaches the method of claim 12.
Shah in view of Zada and Coon further teaches wherein the programmable performance monitoring circuitry is to perform performance monitoring operations for a selected hardware context (Zada, [0019]: “As used herein, the term "agent" refers to software installed on a machine (virtual or physical) that provides an interface to enable remote control of the machine for execution of commands, scripts, and/or other code. The agents monitor designated aspects of the machine (e.g., hardware, software, and/or firmware aspects) and report data related to the monitored aspects to a monitoring server.”) based on commands received from a command streamer associated with the selected hardware context. (Shah, [0067]: “In some embodiments, GPE 410 couples with or includes a command streamer 403, which provides a command stream to the 3D pipeline 312 and/or media pipelines 316. In some embodiments, command streamer 403 is coupled with memory, which can be system memory, or one or more of internal cache memory and shared cache memory. In some embodiments, command streamer 403 receives commands from the memory and sends the commands to 3D pipeline 312 and/or media pipeline 316. The commands are directives fetched from a ring buffer, which stores commands for the 3D pipeline 312 and media pipeline 316.”; Examiner notes, agents comprise an interface for accepting instructions on a VM for performing monitoring of the hardware, software, or firmware, therefore would be monitoring the hardware when the graphics processing engine is receiving commands from a command streamer).
Regarding claim 20, Shah teaches A non-transitory machine-readable medium (Fig. 1 – 120) having program code stored thereon which, when executed by a machine, causes the machine to perform the operations of: (Fig. 1 – 120, 121; [0314]: “Embodiments of the invention may include various steps, which have been described above. The steps may be embodied in machine-executable instructions which may be used to cause a general-purpose or special-purpose processor to perform the steps. Alternatively, these steps may be performed by specific hardware components that contain hardwired logic for performing the steps, or by any combination of programmed computer components and custom hardware components.”)
allocating ([0001]: “This invention relates generally to the field of graphics processors. More particularly, the invention relates to an apparatus and method for provisioning virtualized multi-tile graphics processing hardware.”), parallel execution resources of first hardware logic between a number of virtual machines by second hardware logic; ([0148]: “In addition, in one embodiment, the virtualization mediator 1622 implements a GPU scheduler 1626, which runs concurrently with the CPU scheduler 1616 in the hypervisor 1610, to share the physical GPU 1600 among the VMs 1631-1632.”; [0251]: “In one embodiment, each tile 2591-2594 comprises a scheduler (e.g., a GuC scheduler) 2580 to schedule access by virtual machines to the processing resources.”; [0069]: “In one embodiment, the execution units additionally include general-purpose logic that is programmable to perform parallel general purpose computational operations, in addition to graphics processing operations. The general purpose logic can perform processing operations in parallel or in conjunction with general purpose logic within the processor core(s) 107 of FIG. 1 or core 202A-202N as in FIG. 2.”)
executing a workload of each virtual machine on its allocated portion of the execution resources concurrently with workloads executed by one or more other virtual machines executed on corresponding other allocated portions of the execution resources; ([0176]: “GPU scheduler 1712 for GPU 1720 may be separated from the scheduler for CPU in apparatus 1700. To take the advantage of the hardware parallelism in some embodiments, GPU scheduler 1712 may schedule the workloads separately for different GPU engines, e.g., 3D render engine 1722, blitter render engine 1724, video command streamer (VCS) render engine 1726, and video enhanced command streamer (VECS) render engine 1728. For example, VM 1730 may be 3D intensive, and 3D workload 1732 may need to be scheduled to 3D render engine 1722 at a moment. Meanwhile, VM 1740 may be media intensive, and media workload 1744 may need to be scheduled to VCS render engine 1726 and/or VECS render engine 1728. In this case, GPU scheduler 1712 may schedule 3D workload 1732 from VM 1730 and media workload 1744 from VM 1740 separately.”)
Shah does not teach and dynamically partitioning programmable performance monitoring circuitry based on the number of virtual machines and a portion of the execution resources allocated to each virtual machine, the programmable performance monitoring circuitry to differentiate between performance monitoring data of different virtual machines based on one or more unique identifiers associated with each of the allocated portions of execution resources.
However, in analogous art, Zada teaches and dynamically partitioning programmable performance monitoring circuitry based on the number of virtual machines and a portion of the execution resources allocated to each virtual machine, (Abstract: “…automatically installing an agent on the virtual machine when the virtual machine is identified as having a designated configuration, automatically identifying, via the agent, a resource associated with the virtual machine and, based on the resource, applying a monitoring policy to the virtual machine…”; [0045]: “The example system 100 of FIG. 1 may include multiple performance monitoring servers 106 to monitor agents 132 that are configured differently based on, for example, the resource(s) being monitored by the respective agents 132. Multiple performance monitoring servers 106 may additionally or alternatively be used to monitor different deployment environments 104, different applications 102, and/or different VMs 114. The example agents 132 may be initially configured by the agent installer 126 to communicate with a designated one of the performance monitoring servers 106.”; Fig. 11 – Examiner notes, agent installation in progress, allocation of agents can be changed. Further, “resources” in Zada refer to both software and hardware resources (see at least [0030] and [0032]).) the programmable performance monitoring circuitry to differentiate between performance monitoring data of different virtual machines based on one or more unique identifiers associated with each of the allocated portions of execution resources. (Fig. 5; [0071]: “The example agent monitor 130 of FIG. 1 receives resource report(s) for the new VM(s) from the installed agents (block 312). The resource reports may be generated by the agents 132 of FIG. 1 periodically, aperiodically, and/or in response to an event (e.g., on initial installation and/or execution of the agents 132). The resource reports list the resources and/or provide information about the resources (e.g., platform, servers, services, etc.) executing on the VMs 114. Based on the resource report(s) from the agents 132, the example agent monitor 130 registers the identified resources of the VMs 114 (block 314). For example, the agent monitor 130 may add the resources to an inventory and/or include the resources in a listing or catalog of resources available for monitoring.”)
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the monitoring circuitry for monitoring software and hardware resources of a VM of Zada, with the virtualized tiled-graphics processing hardware system of Shah. As a result, Shah would utilize agents partitioned to each VM, to monitor their specific configuration of execution resources that have been allocated to them. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to more effectively monitor the resources used by virtual machine, as Zada states in [0020]: “Cloud environments are dynamic by nature. Resources, such as virtual machines…may be created, used, moved and/or destroyed with relatively high frequency. In such dynamic environments, monitoring the resources (e.g., virtual machines, applications, services, etc.) can be challenging because, among other things, new resources are often created, resources change their location, and/or the configurations of the resources change. Such changes result in changes to monitoring requirements for the resources.” Therefore, with Zada’s teachings, Shah would more effectively automate the monitoring of resources and responding to necessary changes during a virtual environment’s operation (see also at least [0021]-[0022]).
The combination of Shah/Zada does not teach the one or more unique identifiers comprising an identifier of a respective hardware context that comprises instruction processing and data processing resources to perform a corresponding portion of single-instruction multiple thread (SIMT) or single instruction multiple data (SIMD) operations.
Coon teaches the one or more unique identifiers comprising an identifier of a respective hardware context that comprises instruction processing and data processing resources to perform a corresponding portion of single-instruction multiple thread (SIMT) or single instruction multiple data (SIMD) operations (paragraph 0043).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine Shah/Zada’s processing system with Coon’s processing engine. The motivation for doing so would have been to provide improved locking mechanism for accessing shared memory.
Regarding claim 23, Shah in view of Zada and Coon teaches the non-transitory machine-readable medium of claim 20.
Shah in view of Zada and Coon further teaches wherein the programmable performance monitoring circuitry is to perform performance monitoring operations for a selected hardware context (Zada, [0019]: “As used herein, the term "agent" refers to software installed on a machine (virtual or physical) that provides an interface to enable remote control of the machine for execution of commands, scripts, and/or other code. The agents monitor designated aspects of the machine (e.g., hardware, software, and/or firmware aspects) and report data related to the monitored aspects to a monitoring server.”) based on commands received from a command streamer associated with the selected hardware context. (Shah, [0067]: “In some embodiments, GPE 410 couples with or includes a command streamer 403, which provides a command stream to the 3D pipeline 312 and/or media pipelines 316. In some embodiments, command streamer 403 is coupled with memory, which can be system memory, or one or more of internal cache memory and shared cache memory. In some embodiments, command streamer 403 receives commands from the memory and sends the commands to 3D pipeline 312 and/or media pipeline 316. The commands are directives fetched from a ring buffer, which stores commands for the 3D pipeline 312 and media pipeline 316.”; Examiner notes, agents comprise an interface for accepting instructions on a VM for performing monitoring of the hardware, software, or firmware, therefore would be monitoring the hardware when the graphics processing engine is receiving commands from a command streamer).
Claims 2-4, 13-14, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Shah et al. US 20200410628 A1 (“Shah”) in view of Zada et al. US 20150154039 A1 (“Zada”) and Coon et al. US 20090240860 (“Coon”), in view of BERNAT US 20200241926 A1 (“Bernat”), in view of Wright, David R CSC216 Finite State Machines. David R. Wright website N. Carolina State Univ, 2005 (“Wright”).
Regarding claim 2, Shah in view of Zada and Coon teaches the apparatus of claim 1.
Shah in view of Zada and Coon further teaches a selected hardware context of a particular virtual machine (Fig. 31; [0274]: “In FIG. 31, commands for compute contexts from VM1 and VM3 are shown within the compute queue 3170 and commands for 3D context 0 from VM1 and VM3 are shown within the render queue 3171.”)
Shah in view of Zada and Coon does not teach wherein the programmable performance monitoring circuitry includes a hardware security state machine to enter into a selected context-specific state indicated by a selected unique identifier associated with a selected hardware context of a particular virtual machine, the programmable performance monitoring circuitry to provide performance monitoring data associated with the selected hardware context only in response to requests from at least one of the particular virtual machine the selected hardware context. (Examiner notes, though Shah in view of Zada and Coon teaches the programmable performance monitoring circuity (at least Zada [0045]), hardware context, (at least Shah Fig. 31, [0274] and [0276], and virtual machines (at least Shah Fig. 31), it does not teach a state machine nor the above recited functional limitations).
However, in analogous art, Bernat teaches wherein the programmable performance monitoring circuitry includes a hardware security validation block (Fig. 17B)
to enter into a selected context-specific state indicated by a selected unique identifier associated with a selected hardware context of a particular virtual machine, the programmable performance monitoring circuitry to provide performance monitoring data associated with the selected hardware context only in response to requests from at least one of the particular virtual machine the selected hardware context. ([0114]: “A service or job request can include: target service ID, composite node ID, tenant ID, a request certificate that identifies the requester. For a received service request, system address decoder 1752 can perform the following: (1) validate the service request certificate and the corresponding tenant using validation block 1756 based on shared keys, decryption, or hashing; (2) validate that the service requester has enough privilege to access that particular service (otherwise a flag is generated); (3) using a table 1754, translate the target service ID, composite node ID and tenant ID to provision the service request to the actual hardware elements that perform the service request; (4) modify the actual request to the real/actual target semantic used by the underlying real fabric (e.g., Internet Protocol (IP)); and (5) cause network interface 1750 to transmit the service request to addresses associated with the actual hardware and software elements that perform the request and that embody the composite node. Table 1754 can track identifiers of IP and MAC addresses or other manners of communicating with the actual hardware devices and software elements that make up the composite node associated with the composite node ID. Table 1754 can provide such identifiers so that that the service request is sent to the proper destination.”; [0097]: “Various embodiments provide for validation, security, control, and visibility of the actual hardware and software components allocated in a composite node for a service, tenant, sub-tenant, or customer. Some embodiments provide for resource certification in which hardware and software resources expose an interface to allow obtaining its meta-data and the corresponding certification of its resources (e.g., CPUs, memories, storage, network interface, accelerators, applications, and so forth).”; Examiner notes, before providing the data of the hardware of the particular VM (see [0104]), the request is first verified, upon successful verification, the rest of the steps may be completed. The previous process is a transition from various states considering an input and previous output (analyze the requestor ID, then move to different states if verification was successful or not etc.)).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the hardware verification process in Bernat, with the agents installed on the number of virtual machines in Shah in view of Zada and Coon. As a result, Shah in view of Zada and Coon would utilize Bernat’s process to allow the monitoring circuitry (agent) on a virtual machine, to verify and collect information of a specific hardware context for that virtual machine it is installed on. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to provide the scheduler in Shah in view of Zada and Coon with detailed information about the hardware each hardware context corresponds to, therefore can more effectively be scheduled on hardware with stronger affinity masks (see at least Shah [0276], Fig. 31).
Shah in view of Zada and Coon in view of Bernat does not teach state machine.
However, in analogous art, Wright teaches state machine. (Pg. 14: Figure 20, Table 5; Examiner notes, there is source code for the implementation on pg. 24-25).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the validation block in Shah in view of Zada and Coon in view of Bernat (Bernat, Fig. 17B) with a finite state machine implementation in Wright. As a result, the hardware context validation procedure, and its steps, would be implemented via a finite state machine. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to reliably implement the states of the verification block. Wright says on pg. 1: “There are many ways of modeling the behavior of systems, and the use of state machines is one of the oldest and best known. State machines allow us to think about the “state” of a system at a particular point in time and characterize the behavior of the system based on that state. We can use a similar technique to model and design software systems by identifying what states the system can be in, what inputs or events trigger state transitions, and how the system will behave in each state. In this model, we view the execution of the software as a sequence of transitions that move the system through its various states.”
Regarding claim 3, Shah in view of Zada and Coon, in view of Bernat, in view of Wright teaches the apparatus of claim 2.
Shah in view of Zada and Coon, in view of Bernat, in view of Wright further teaches wherein the programmable performance monitoring circuitry comprises a command streamer interface to couple the programmable performance monitoring circuitry (Zada, [0019]: “As used herein, the term "agent" refers to software installed on a machine (virtual or physical) that provides an interface to enable remote control of the machine for execution of commands, scripts, and/or other code. The agents monitor designated aspects of the machine (e.g., hardware, software, and/or firmware aspects) and report data related to the monitored aspects to a monitoring server.”) to command streamers of the selected hardware contexts. (Shah, [0067]: “In some embodiments, GPE 410 couples with or includes a command streamer 403, which provides a command stream to the 3D pipeline 312 and/or media pipelines 316. In some embodiments, command streamer 403 is coupled with memory, which can be system memory, or one or more of internal cache memory and shared cache memory. In some embodiments, command streamer 403 receives commands from the memory and sends the commands to 3D pipeline 312 and/or media pipeline 316. The commands are directives fetched from a ring buffer, which stores commands for the 3D pipeline 312 and media pipeline 316.”; Examiner notes, agents comprise an interface for accepting instructions on a VM for performing monitoring of the hardware software or firmware).
Regarding claim 4, Shah in view of Zada and Coon, in view of Bernat, in view of Wright teaches the apparatus of claim 3.
Shah in view of Zada and Coon, in view of Bernat, in view of Wright further teaches wherein the programmable performance monitoring circuitry is to perform performance monitoring operations for a selected hardware context (Zada, [0019]: “As used herein, the term "agent" refers to software installed on a machine (virtual or physical) that provides an interface to enable remote control of the machine for execution of commands, scripts, and/or other code. The agents monitor designated aspects of the machine (e.g., hardware, software, and/or firmware aspects) and report data related to the monitored aspects to a monitoring server.”) based on commands received from a command streamer associated with the selected hardware context. (Shah, [0067]: “In some embodiments, GPE 410 couples with or includes a command streamer 403, which provides a command stream to the 3D pipeline 312 and/or media pipelines 316. In some embodiments, command streamer 403 is coupled with memory, which can be system memory, or one or more of internal cache memory and shared cache memory. In some embodiments, command streamer 403 receives commands from the memory and sends the commands to 3D pipeline 312 and/or media pipeline 316. The commands are directives fetched from a ring buffer, which stores commands for the 3D pipeline 312 and media pipeline 316.”; Examiner notes, agents comprise an interface for accepting instructions on a VM for performing monitoring of the hardware, software, or firmware, therefore would be monitoring the hardware when the graphics processing engine is receiving commands from a command streamer).
Regarding claim 13, Shah in view of Zada and Coon teaches the method of claim 12.
Shah in view of Zada and Coon further teaches a selected hardware context of a particular virtual machine (Fig. 31; [0274]: “In FIG. 31, commands for compute contexts from VM1 and VM3 are shown within the compute queue 3170 and commands for 3D context 0 from VM1 and VM3 are shown within the render queue 3171.”).
Shah in view of Zada and Coon wherein the programmable performance monitoring circuitry includes a hardware security state machine to enter into a selected context-specific state indicated by a selected unique identifier associated with a selected hardware context of a particular virtual machine, the programmable performance monitoring circuitry to provide performance monitoring data associated with the selected hardware context only in response to requests from at least one of the particular virtual machine the selected hardware context. (Examiner notes, though Shah in view of Zada and Coon teaches the programmable performance monitoring circuity (at least Zada [0045]), hardware context, (at least Shah Fig. 31, [0274] and [0276], and virtual machines (at least Shah Fig. 31), it does not teach a state machine nor the above recited functional limitations).
However, in analogous art, Bernat teaches wherein the programmable performance monitoring circuitry includes a hardware security validation block (Fig. 17B)
to enter into a selected context-specific state indicated by a selected unique identifier associated with a selected hardware context of a particular virtual machine, the programmable performance monitoring circuitry to provide performance monitoring data associated with the selected hardware context only in response to requests from at least one of the particular virtual machine the selected hardware context. ([0114]: “A service or job request can include: target service ID, composite node ID, tenant ID, a request certificate that identifies the requester. For a received service request, system address decoder 1752 can perform the following: (1) validate the service request certificate and the corresponding tenant using validation block 1756 based on shared keys, decryption, or hashing; (2) validate that the service requester has enough privilege to access that particular service (otherwise a flag is generated); (3) using a table 1754, translate the target service ID, composite node ID and tenant ID to provision the service request to the actual hardware elements that perform the service request; (4) modify the actual request to the real/actual target semantic used by the underlying real fabric (e.g., Internet Protocol (IP)); and (5) cause network interface 1750 to transmit the service request to addresses associated with the actual hardware and software elements that perform the request and that embody the composite node. Table 1754 can track identifiers of IP and MAC addresses or other manners of communicating with the actual hardware devices and software elements that make up the composite node associated with the composite node ID. Table 1754 can provide such identifiers so that that the service request is sent to the proper destination.”; [0097]: “Various embodiments provide for validation, security, control, and visibility of the actual hardware and software components allocated in a composite node for a service, tenant, sub-tenant, or customer. Some embodiments provide for resource certification in which hardware and software resources expose an interface to allow obtaining its meta-data and the corresponding certification of its resources (e.g., CPUs, memories, storage, network interface, accelerators, applications, and so forth).”; Examiner notes, before providing the data of the hardware of the particular VM (see [0104]), the request is first verified, upon successful verification, the rest of the steps may be completed. The previous process is a transition from various states considering an input and previous output (analyze the requestor ID, then move to different states if verification was successful or not etc.)).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the hardware verification process in Bernat, with the agents installed on the number of virtual machines in Shah in view of Zada and Coon. As a result, Shah in view of Zada and Coon would utilize Bernat’s process to allow the monitoring circuitry (agent) on a virtual machine, to verify and collect information of a specific hardware context for that virtual machine it is installed on. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to provide the scheduler in Shah in view of Zada and Coon with detailed information about the hardware each hardware context corresponds to, therefore can more effectively be scheduled on hardware with stronger affinity masks (see at least Shah [0276], Fig. 31).
Shah in view of Zada and Coon in view of Bernat does not teach state machine.
However, in analogous art, Wright teaches state machine. (Pg. 14: Figure 20, Table 5; Examiner notes, there is source code for the implementation on pg. 24-25).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the validation block in Shah in view of Zada and Coon in view of Bernat (Bernat, Fig. 17B) with a finite state machine implementation in Wright. As a result, the hardware context validation procedure, and its steps, would be implemented via a finite state machine. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to reliably implement the states of the verification block. Wright says on pg. 1: “There are many ways of modeling the behavior of systems, and the use of state machines is one of the oldest and best known. State machines allow us to think about the “state” of a system at a particular point in time and characterize the behavior of the system based on that state. We can use a similar technique to model and design software systems by identifying what states the system can be in, what inputs or events trigger state transitions, and how the system will behave in each state. In this model, we view the execution of the software as a sequence of transitions that move the system through its various states.”
Regarding claim 14, Shah in view of Zada and Coon, in view of Bernat, in view of Wright teaches the method of claim 13.
Shah in view of Zada and Coon, in view of Bernat, in view of Wright further teaches wherein the programmable performance monitoring circuitry comprises a command streamer interface to couple the programmable performance monitoring circuitry (Zada, [0019]: “As used herein, the term "agent" refers to software installed on a machine (virtual or physical) that provides an interface to enable remote control of the machine for execution of commands, scripts, and/or other code. The agents monitor designated aspects of the machine (e.g., hardware, software, and/or firmware aspects) and report data related to the monitored aspects to a monitoring server.”) to command streamers of the selected hardware contexts. (Shah, [0067]: “In some embodiments, GPE 410 couples with or includes a command streamer 403, which provides a command stream to the 3D pipeline 312 and/or media pipelines 316. In some embodiments, command streamer 403 is coupled with memory, which can be system memory, or one or more of internal cache memory and shared cache memory. In some embodiments, command streamer 403 receives commands from the memory and sends the commands to 3D pipeline 312 and/or media pipeline 316. The commands are directives fetched from a ring buffer, which stores commands for the 3D pipeline 312 and media pipeline 316.”; Examiner notes, agents comprise an interface for accepting instructions on a VM for performing monitoring of the hardware software or firmware).
Regarding claim 21, Shah in view of Zada and Coon teaches the non-transitory machine-readable medium of claim 20.
Shah in view of Zada and Coon further teaches a selected hardware context of a particular virtual machine (Fig. 31; [0274]: “In FIG. 31, commands for compute contexts from VM1 and VM3 are shown within the compute queue 3170 and commands for 3D context 0 from VM1 and VM3 are shown within the render queue 3171.”)
Shah in view of Zada and Coon does not teach wherein the programmable performance monitoring circuitry includes a hardware security state machine to enter into a selected context-specific state indicated by a selected unique identifier associated with a selected hardware context of a particular virtual machine, the programmable performance monitoring circuitry to provide performance monitoring data associated with the selected hardware context only in response to requests from at least one of the particular virtual machine the selected hardware context. (Examiner notes, though Shah in view of Zada and Coon teaches the programmable performance monitoring circuity (at least Zada [0045]), hardware context, (at least Shah Fig. 31, [0274] and [0276], and virtual machines (at least Shah Fig. 31), it does not teach a state machine nor the above recited functional limitations).
However, in analogous art, Bernat teaches wherein the programmable performance monitoring circuitry includes a hardware security validation block (Fig. 17B)
to enter into a selected context-specific state indicated by a selected unique identifier associated with a selected hardware context of a particular virtual machine, the programmable performance monitoring circuitry to provide performance monitoring data associated with the selected hardware context only in response to requests from at least one of the particular virtual machine the selected hardware context. ([0114]: “A service or job request can include: target service ID, composite node ID, tenant ID, a request certificate that identifies the requester. For a received service request, system address decoder 1752 can perform the following: (1) validate the service request certificate and the corresponding tenant using validation block 1756 based on shared keys, decryption, or hashing; (2) validate that the service requester has enough privilege to access that particular service (otherwise a flag is generated); (3) using a table 1754, translate the target service ID, composite node ID and tenant ID to provision the service request to the actual hardware elements that perform the service request; (4) modify the actual request to the real/actual target semantic used by the underlying real fabric (e.g., Internet Protocol (IP)); and (5) cause network interface 1750 to transmit the service request to addresses associated with the actual hardware and software elements that perform the request and that embody the composite node. Table 1754 can track identifiers of IP and MAC addresses or other manners of communicating with the actual hardware devices and software elements that make up the composite node associated with the composite node ID. Table 1754 can provide such identifiers so that that the service request is sent to the proper destination.”; [0097]: “Various embodiments provide for validation, security, control, and visibility of the actual hardware and software components allocated in a composite node for a service, tenant, sub-tenant, or customer. Some embodiments provide for resource certification in which hardware and software resources expose an interface to allow obtaining its meta-data and the corresponding certification of its resources (e.g., CPUs, memories, storage, network interface, accelerators, applications, and so forth).”; Examiner notes, before providing the data of the hardware of the particular VM (see [0104]), the request is first verified, upon successful verification, the rest of the steps may be completed. The previous process is a transition from various states considering an input and previous output (analyze the requestor ID, then move to different states if verification was successful or not etc.)).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the hardware verification process in Bernat, with the agents installed on the number of virtual machines in Shah in view of Zada and Coon. As a result, Shah in view of Zada and Coon would utilize Bernat’s process to allow the monitoring circuitry (agent) on a virtual machine, to verify and collect information of a specific hardware context for that virtual machine it is installed on. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to provide the scheduler in Shah in view of Zada and Coon with detailed information about the hardware each hardware context corresponds to, therefore can more effectively be scheduled on hardware with stronger affinity masks (see at least Shah [0276], Fig. 31).
Shah in view of Zada and Coon in view of Bernat does not teach state machine.
However, in analogous art, Wright teaches state machine. (Pg. 14: Figure 20, Table 5; Examiner notes, there is source code for the implementation on pg. 24-25).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the validation block in Shah in view of Zada and Coon in view of Bernat (Bernat, Fig. 17B) with a finite state machine implementation in Wright. As a result, the hardware context validation procedure, and its steps, would be implemented via a finite state machine. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to reliably implement the states of the verification block. Wright says on pg. 1: “There are many ways of modeling the behavior of systems, and the use of state machines is one of the oldest and best known. State machines allow us to think about the “state” of a system at a particular point in time and characterize the behavior of the system based on that state. We can use a similar technique to model and design software systems by identifying what states the system can be in, what inputs or events trigger state transitions, and how the system will behave in each state. In this model, we view the execution of the software as a sequence of transitions that move the system through its various states.”
Regarding claim 22, Shah in view of Zada and Coon, in view of Bernat, in view of Wright teaches the non-transitory machine-readable medium of claim 21.
Shah in view of Zada and Coon, in view of Bernat, in view of Wright further teaches wherein the programmable performance monitoring circuitry comprises a command streamer interface to couple the programmable performance monitoring circuitry (Zada, [0019]: “As used herein, the term "agent" refers to software installed on a machine (virtual or physical) that provides an interface to enable remote control of the machine for execution of commands, scripts, and/or other code. The agents monitor designated aspects of the machine (e.g., hardware, software, and/or firmware aspects) and report data related to the monitored aspects to a monitoring server.”) to command streamers of the selected hardware contexts. (Shah, [0067]: “In some embodiments, GPE 410 couples with or includes a command streamer 403, which provides a command stream to the 3D pipeline 312 and/or media pipelines 316. In some embodiments, command streamer 403 is coupled with memory, which can be system memory, or one or more of internal cache memory and shared cache memory. In some embodiments, command streamer 403 receives commands from the memory and sends the commands to 3D pipeline 312 and/or media pipeline 316. The commands are directives fetched from a ring buffer, which stores commands for the 3D pipeline 312 and media pipeline 316.”; Examiner notes, agents comprise an interface for accepting instructions on a VM for performing monitoring of the hardware software or firmware).
Claims 5-7 are rejected under 35 U.S.C. 103 as being unpatentable over Shah et al. US 20200410628 A1 (“Shah”) in view of Zada et al. US 20150154039 A1 (“Zada”) and Coon et al. US 20090240860 (“Coon”), in view of BERNAT US 20200241926 A1 (“Bernat”), in view of Wright, David R CSC215 Class Notes. David R. Wright website N. Carolina State Univ 2014-03-27 (“Wright”), in view of Glossner et al. US 20150220347 A1 (“Glossner”).
Regarding claim 5, Shah in view of Zada and Coon, in view of Bernat, in view of Wright teaches the apparatus of claim 4.
Shah further teaches hardware contexts (Fig. 31 – VM1: HW context 0, HW context 1; [0274]: “In FIG. 31, commands for compute contexts from VM1 and VM3 are shown within the compute queue 3170 and commands for 3D context 0 from VM1 and VM3 are shown within the render queue 3171.”) of the number of virtual machines. (Fig. – 31: VM1 – VM63).
Shah in view of Zada and Coon, in view of Bernat, in view of Wright does not teach wherein the programmable performance monitoring circuitry includes hardware context logic to individually monitor performance of each of the selected hardware contexts of the number of virtual machines. (Examiner notes, though Shah in view of Zada and Coon, in view of Bernat, in view of Wright does teach circuitry for monitoring hardware contexts (see previous claim rejections), and does teach multiple hardware contexts of each VM (see current rejection, immediate above), the reference is silent as to whether the monitoring agents can monitor each of the multiple hardware contexts assigned to a VM).
However, in analogous art, Glossner teaches wherein the programmable performance monitoring circuitry includes hardware context logic to individually monitor performance of each of the selected hardware contexts of the number of virtual machines. (Fig. 9; [0060]: “As shown in FIG. 9, an ordered list of identifiers, for example, an array of bits in an AllocatedContextIDArray ("ACID array") register 900, may be configured to keep track of how many hardware contexts (e.g., 602a-602n) are actively executing software threads. The ACID array register 900 may contain one bit for each hardware context (e.g., 602a-602n). A bit may be set corresponding to a hardware context number in the set of hardware contexts (e.g., 602a-602n).”).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the monitoring of multiple hardware contexts in Glossner, with the monitoring agents of Shah in view of Zada and Coon, in view of Bernat, in view of Wright, resulting in the agents partitioned to certain VMs in Shah in view of Zada and Coon, in view of Bernat, in view of Wright, to monitor each individual hardware context for a VM if more than one hardware context is assigned. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to avoid a decrease in single threaded performance in a system with multi-threaded processors. Glossner states in [0054]: “One problem with typical designs of multi-threaded processors is that single threaded performance may suffer. In examples of the present disclosure, a barrel-threaded processor is guaranteed a dispatch slot providing deterministic behavior. Other threads may also dispatch to a context using a specific dispatch policy that provides enhanced single threaded performance in a deterministic manner without causing dead-lock.”
Regarding claim 6, Shah in view of Zada and Coon, in view of Bernat, in view of Wright, in view of Glossner teaches the apparatus of claim 5.
Shah in view of Zada and Coon, in view of Bernat, in view of Wright, in view of Glossner further teaches wherein the programmable performance monitoring circuitry further includes render context hardware logic (Shah, Fig. 31; [0274]: “For example, scheduler 0 2820 (highlighted in FIG. 31) includes a single compute queue 3170 to store commands directed to the compute engine of tile 2591 from all VMs and a single render queue 3171 to store commands directed to the 3D engine of tile 2591 from all VMs. In FIG. 31, commands for compute contexts from VM1 and VM3 are shown within the compute queue 3170 and commands for 3D context 0 from VM1 and VM3 are shown within the render queue 3171. Similarly, scheduler 0 2820 draws from other engine queues 3172 to store commands/data for the other graphics processor engines (e.g., BLT engine 2503, media engine 2504, and command streamer engine 2505).”) to individually monitor performance of a plurality of render hardware contexts allocated to the number of virtual machines. (Glossner, Fig. 9; [0060]: “As shown in FIG. 9, an ordered list of identifiers, for example, an array of bits in an AllocatedContextIDArray ("ACID array") register 900, may be configured to keep track of how many hardware contexts (e.g., 602a-602n) are actively executing software threads. The ACID array register 900 may contain one bit for each hardware context (e.g., 602a-602n). A bit may be set corresponding to a hardware context number in the set of hardware contexts (e.g., 602a-602n).”; Examiner notes, Shah in view of Zada and Coon, in view of Bernat, in view of Wright, in view of Glossner apply to both render and compute hardware contexts, as both utilize the processors and threads but in different contexts).
Regarding claim 7, Shah in view of Zada and Coon, in view of Bernat, in view of Wright, in view of Glossner teaches the apparatus of claim 6.
Shah in view of Zada and Coon, in view of Bernat, in view of Wright, in view of Glossner further teaches wherein the programmable performance monitoring circuitry further includes global performance monitoring hardware logic to globally monitor performance of the apparatus. (Shah, [0163]: “In various embodiments, hypervisor 1710 may track, manage resources and lifecycles of the vGPUs 1760A and 1760B as described herein.”).
Claims 8, 16-19, and 24-27 are rejected under 35 U.S.C. 103 as being unpatentable over Shah et al. US 20200410628 A1 (“Shah”) in view of Zada et al. US 20150154039 A1 (“Zada”) and Coon et al. US 20090240860 (“Coon”), in view of Glossner et al. US 20150220347 A1 (“Glossner”).
Regarding claim 8, Shah in view of Zada and Coon teaches the apparatus of claim 1.
Shah in view of Zada and Coon further teaches a change in the number of virtual machines to a new number of virtual machines, and wherein the programmable performance monitoring circuitry is to be dynamically repartitioned based on the new number of virtual machines (Zada, Fig. 3, Fig. 5, [0070]: “The example agent installer 126 of FIG. 1 deploys agent(s) to the new VM(s) 114 based on the configuration(s) and based on a management policy (block 310)…Example instructions to implement block 310 of FIG. 3 are disclosed below with reference to FIG. 5.”; Examiner notes, new VMs identified and agents are repartitioned to them; Abstract: “…automatically installing an agent on the virtual machine when the virtual machine is identified as having a designated configuration, automatically identifying, via the agent, a resource associated with the virtual machine and, based on the resource, applying a monitoring policy to the virtual machine…”; Fig 11: Examiner notes, agent installation in progress, allocation of agents can be changed. Further, agents can be automatically installed when new VMs are identified.) and to perform parallel performance monitoring operations to monitor performance of each virtual machine of the new number of virtual machines. (Zada, [0071]: “The example agent monitor 130 of FIG. 1 receives resource report(s) for the new VM(s) from the installed agents (block 312). The resource reports may be generated by the agents 132 of FIG. 1 periodically, aperiodically, and/or in response to an event (e.g., on initial installation and/or execution of the agents 132). The resource reports list the resources and/or provide information about the resources (e.g., platform, servers, services, etc.) executing on the VMs 114. Based on the resource report(s) from the agents 132, the example agent monitor 130 registers the identified resources of the VMs 114 (block 314). For example, the agent monitor 130 may add the resources to an inventory and/or include the resources in a listing or catalog of resources available for monitoring.”).
Shah in view of Zada and Coon does not teach wherein the first hardware logic is to be reallocated in response to a change in the number of virtual machines to a new number of virtual machines. (Examiner notes, though Shah in view of Zada and Coon does teach changing the number of virtual machines (see current rejection, immediate above), the reference is silent as to whether the first hardware logic is reallocated in response to a change in the number of virtual machines occurs).
However, in analogous art, Glossner teaches wherein the first hardware logic is to be reallocated in response to a change in the number of software threads to a new number of software threads. ([0069]: “Referring again to FIGS. 6-10, in one example, the instruction issue logic 608 of the processing device 600 (e.g., the multithreaded processor 600) may be configured to identify a set of software threads having instructions waiting to issue. For each software thread in the set of the software threads, the instruction issue logic 608 may be configured to bind the software thread to an available hardware context (e.g., 602a) in a set of hardware contexts (e.g., 602a-602n).”; [0071]: “In an example, if the processing device 600 (e.g., the multithreaded processor 600) adds a new software thread to the software threads having instructions waiting to issue, the instruction issue logic 608 may be configured to bind the new software thread to an available hardware context (e.g., 602b) in a set of available hardware contexts (e.g., 602a-602n) and to store an identifier (e.g., a CID 800a) of the available hardware context (e.g., 602b) bound to the new software thread to a next available entry in an ordered list of identifiers 1008 (e.g., the ACID array register 900).”).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the reallocation of hardware contexts in response to new software threads in Glossner with the repartitioning of agents in response to a change of the number of virtual machines in Shah in view of Zada and Coon. As a result, when the number of virtual machines changes, the hardware contexts will be reallocated to the new number of VMs, and monitoring agents repartitioned to the new number of VMs as well. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to avoid the performance decrease from certain conventional multi-threading technologies when there are more hardware contexts on a processor available then software threads (see at least Glossner, [0015]-[0017] and [0021]).
Regarding claim 16, Shah in view of Zada and Coon teaches the method of claim 15.
Shah further teaches hardware contexts (Fig. 31 – VM1: HW context 0, HW context 1; [0274]: “In FIG. 31, commands for compute contexts from VM1 and VM3 are shown within the compute queue 3170 and commands for 3D context 0 from VM1 and VM3 are shown within the render queue 3171.”) of the number of virtual machines. (Fig. – 31: VM1 – VM63).
Shah in view of Zada and Coon does not teach wherein the programmable performance monitoring circuitry includes hardware context logic to individually monitor performance of each of the selected hardware contexts of the number of virtual machines. (Examiner notes, though Shah in view of Zada and Coon does teach circuitry for monitoring hardware contexts (see previous claim rejections), and does teach multiple hardware contexts of each VM (see current rejection, immediate above), the reference is silent as to whether the monitoring agents can monitor each of the multiple hardware contexts assigned to a VM).
However, in analogous art, Glossner teaches wherein the programmable performance monitoring circuitry includes hardware context logic to individually monitor performance of each of the selected hardware contexts of the number of virtual machines. (Fig. 9; [0060]: “As shown in FIG. 9, an ordered list of identifiers, for example, an array of bits in an AllocatedContextIDArray ("ACID array") register 900, may be configured to keep track of how many hardware contexts (e.g., 602a-602n) are actively executing software threads. The ACID array register 900 may contain one bit for each hardware context (e.g., 602a-602n). A bit may be set corresponding to a hardware context number in the set of hardware contexts (e.g., 602a-602n).”).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the monitoring of multiple hardware contexts in Glossner, with the monitoring agents of Shah in view of Zada and Coon, resulting in the agents partitioned to certain VMs in Shah in view of Zada and Coon, to monitor each individual hardware context for a VM if more than one hardware context is assigned. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to avoid a decrease in single threaded performance in a system with multi-threaded processors. Glossner states in [0054]: “One problem with typical designs of multi-threaded processors is that single threaded performance may suffer. In examples of the present disclosure, a barrel-threaded processor is guaranteed a dispatch slot providing deterministic behavior. Other threads may also dispatch to a context using a specific dispatch policy that provides enhanced single threaded performance in a deterministic manner without causing dead-lock.”
Regarding claim 17, Shah in view of Zada and Coon, in view of Glossner teaches the method of claim 16.
Shah in view of Zada and Coon, in view of Glossner further teaches wherein the programmable performance monitoring circuitry further includes render context hardware logic (Shah, Fig. 31; [0274]: “For example, scheduler 0 2820 (highlighted in FIG. 31) includes a single compute queue 3170 to store commands directed to the compute engine of tile 2591 from all VMs and a single render queue 3171 to store commands directed to the 3D engine of tile 2591 from all VMs. In FIG. 31, commands for compute contexts from VM1 and VM3 are shown within the compute queue 3170 and commands for 3D context 0 from VM1 and VM3 are shown within the render queue 3171. Similarly, scheduler 0 2820 draws from other engine queues 3172 to store commands/data for the other graphics processor engines (e.g., BLT engine 2503, media engine 2504, and command streamer engine 2505).”) to individually monitor performance of a plurality of render hardware contexts allocated to the number of virtual machines. (Glossner, Fig. 9; [0060]: “As shown in FIG. 9, an ordered list of identifiers, for example, an array of bits in an AllocatedContextIDArray ("ACID array") register 900, may be configured to keep track of how many hardware contexts (e.g., 602a-602n) are actively executing software threads. The ACID array register 900 may contain one bit for each hardware context (e.g., 602a-602n). A bit may be set corresponding to a hardware context number in the set of hardware contexts (e.g., 602a-602n).”; Examiner notes, Shah in view of Zada and Coon, in view of Glossner apply to both render and compute hardware contexts, as both utilize the processors and threads but in different contexts).
Regarding claim 18, Shah in view of Zada and Coon, in view of Glossner teaches the method of claim 17.
Shah in view of Zada and Coon, in view of Glossner further teaches wherein the programmable performance monitoring circuitry further includes global performance monitoring hardware logic to globally monitor performance (Shah, [0163]: “In various embodiments, hypervisor 1710 may track, manage resources and lifecycles of the vGPUs 1760A and 1760B as described herein.”).
Regarding claim 19, Shah in view of Zada and Coon teaches the method of claim 12.
Shah in view of Zada and Coon further teaches a change in the number of virtual machines to a new number of virtual machines, and wherein the programmable performance monitoring circuitry is to be dynamically repartitioned based on the new number of virtual machines (Zada, Fig. 3, Fig. 5, [0070]: “The example agent installer 126 of FIG. 1 deploys agent(s) to the new VM(s) 114 based on the configuration(s) and based on a management policy (block 310)…Example instructions to implement block 310 of FIG. 3 are disclosed below with reference to FIG. 5.”; Examiner notes, new VMs identified and agents are repartitioned to them; Abstract: “…automatically installing an agent on the virtual machine when the virtual machine is identified as having a designated configuration, automatically identifying, via the agent, a resource associated with the virtual machine and, based on the resource, applying a monitoring policy to the virtual machine…”; Fig 11: Examiner notes, agent installation in progress, allocation of agents can be changed. Further, agents can be automatically installed when new VMs are identified.) and to perform parallel performance monitoring operations to monitor performance of each virtual machine of the new number of virtual machines. (Zada, [0071]: “The example agent monitor 130 of FIG. 1 receives resource report(s) for the new VM(s) from the installed agents (block 312). The resource reports may be generated by the agents 132 of FIG. 1 periodically, aperiodically, and/or in response to an event (e.g., on initial installation and/or execution of the agents 132). The resource reports list the resources and/or provide information about the resources (e.g., platform, servers, services, etc.) executing on the VMs 114. Based on the resource report(s) from the agents 132, the example agent monitor 130 registers the identified resources of the VMs 114 (block 314). For example, the agent monitor 130 may add the resources to an inventory and/or include the resources in a listing or catalog of resources available for monitoring.”).
Shah in view of Zada and Coon does not teach wherein the first hardware logic is to be reallocated in response to a change in the number of virtual machines to a new number of virtual machines. (Examiner notes, though Shah in view of Zada and Coon does teach changing the number of virtual machines (see current rejection, immediate above), the reference is silent as to whether the first hardware logic is reallocated in response to a change in the number of virtual machines occurs).
However, in analogous art, Glossner teaches wherein the first hardware logic is to be reallocated in response to a change in the number of software threads to a new number of software threads. ([0069]: “Referring again to FIGS. 6-10, in one example, the instruction issue logic 608 of the processing device 600 (e.g., the multithreaded processor 600) may be configured to identify a set of software threads having instructions waiting to issue. For each software thread in the set of the software threads, the instruction issue logic 608 may be configured to bind the software thread to an available hardware context (e.g., 602a) in a set of hardware contexts (e.g., 602a-602n).”; [0071]: “In an example, if the processing device 600 (e.g., the multithreaded processor 600) adds a new software thread to the software threads having instructions waiting to issue, the instruction issue logic 608 may be configured to bind the new software thread to an available hardware context (e.g., 602b) in a set of available hardware contexts (e.g., 602a-602n) and to store an identifier (e.g., a CID 800a) of the available hardware context (e.g., 602b) bound to the new software thread to a next available entry in an ordered list of identifiers 1008 (e.g., the ACID array register 900).”).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the reallocation of hardware contexts in response to new software threads in Glossner with the repartitioning of agents in response to a change of the number of virtual machines in Shah in view of Zada and Coon. As a result, when the number of virtual machines changes, the hardware contexts will be reallocated to the new number of VMs, and monitoring agents repartitioned to the new number of VMs as well. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to avoid the performance decrease from certain conventional multi-threading technologies when there are more hardware contexts on a processor available then software threads (see at least Glossner, [0015]-[0017] and [0021]).
Regarding claim 24, Shah in view of Zada and Coon teaches the non-transitory machine-readable medium of claim 23.
Shah further teaches hardware contexts (Fig. 31 – VM1: HW context 0, HW context 1; [0274]: “In FIG. 31, commands for compute contexts from VM1 and VM3 are shown within the compute queue 3170 and commands for 3D context 0 from VM1 and VM3 are shown within the render queue 3171.”) of the number of virtual machines. (Fig. – 31: VM1 – VM63).
Shah in view of Zada and Coon does not teach wherein the programmable performance monitoring circuitry includes hardware context logic to individually monitor performance of each of the selected hardware contexts of the number of virtual machines. (Examiner notes, though Shah in view of Zada and Coon does teach circuitry for monitoring hardware contexts (see previous claim rejections), and does teach multiple hardware contexts of each VM (see current rejection, immediate above), the reference is silent as to whether the monitoring agents can monitor each of the multiple hardware contexts assigned to a VM).
However, in analogous art, Glossner teaches wherein the programmable performance monitoring circuitry includes hardware context logic to individually monitor performance of each of the selected hardware contexts of the number of virtual machines. (Fig. 9; [0060]: “As shown in FIG. 9, an ordered list of identifiers, for example, an array of bits in an AllocatedContextIDArray ("ACID array") register 900, may be configured to keep track of how many hardware contexts (e.g., 602a-602n) are actively executing software threads. The ACID array register 900 may contain one bit for each hardware context (e.g., 602a-602n). A bit may be set corresponding to a hardware context number in the set of hardware contexts (e.g., 602a-602n).”).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the monitoring of multiple hardware contexts in Glossner, with the monitoring agents of Shah in view of Zada and Coon, resulting in the agents partitioned to certain VMs in Shah in view of Zada and Coon, to monitor each individual hardware context for a VM if more than one hardware context is assigned. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to avoid a decrease in single threaded performance in a system with multi-threaded processors. Glossner states in [0054]: “One problem with typical designs of multi-threaded processors is that single threaded performance may suffer. In examples of the present disclosure, a barrel-threaded processor is guaranteed a dispatch slot providing deterministic behavior. Other threads may also dispatch to a context using a specific dispatch policy that provides enhanced single threaded performance in a deterministic manner without causing dead-lock.”
Regarding claim 25, Shah in view of Zada and Coon, in view of Glossner teaches the non-transitory machine-readable medium of claim 24.
Shah in view of Zada and Coon, in view of Glossner further teaches wherein the programmable performance monitoring circuitry further includes render context hardware logic (Shah, Fig. 31; [0274]: “For example, scheduler 0 2820 (highlighted in FIG. 31) includes a single compute queue 3170 to store commands directed to the compute engine of tile 2591 from all VMs and a single render queue 3171 to store commands directed to the 3D engine of tile 2591 from all VMs. In FIG. 31, commands for compute contexts from VM1 and VM3 are shown within the compute queue 3170 and commands for 3D context 0 from VM1 and VM3 are shown within the render queue 3171. Similarly, scheduler 0 2820 draws from other engine queues 3172 to store commands/data for the other graphics processor engines (e.g., BLT engine 2503, media engine 2504, and command streamer engine 2505).”) to individually monitor performance of a plurality of render hardware contexts allocated to the number of virtual machines. (Glossner, Fig. 9; [0060]: “As shown in FIG. 9, an ordered list of identifiers, for example, an array of bits in an AllocatedContextIDArray ("ACID array") register 900, may be configured to keep track of how many hardware contexts (e.g., 602a-602n) are actively executing software threads. The ACID array register 900 may contain one bit for each hardware context (e.g., 602a-602n). A bit may be set corresponding to a hardware context number in the set of hardware contexts (e.g., 602a-602n).”; Examiner notes, Shah in view of Zada and Coon, in view of Glossner apply to both render and compute hardware contexts, as both utilize the processors and threads but in different contexts).
Regarding claim 26, Shah in view of Zada and Coon, in view of Glossner teaches the non-transitory machine-readable medium of claim 25.
Shah in view of Zada and Coon, in view of Glossner further teaches wherein the programmable performance monitoring circuitry further includes global performance monitoring hardware logic to globally monitor performance (Shah, [0163]: “In various embodiments, hypervisor 1710 may track, manage resources and lifecycles of the vGPUs 1760A and 1760B as described herein.”).
Regarding claim 27, Shah in view of Zada and Coon teaches the non-transitory machine-readable medium of claim 20.
Shah in view of Zada and Coon further teaches a change in the number of virtual machines to a new number of virtual machines, and wherein the programmable performance monitoring circuitry is to be dynamically repartitioned based on the new number of virtual machines (Zada, Fig. 3, Fig. 5, [0070]: “The example agent installer 126 of FIG. 1 deploys agent(s) to the new VM(s) 114 based on the configuration(s) and based on a management policy (block 310)…Example instructions to implement block 310 of FIG. 3 are disclosed below with reference to FIG. 5.”; Examiner notes, new VMs identified and agents are repartitioned to them; Abstract: “…automatically installing an agent on the virtual machine when the virtual machine is identified as having a designated configuration, automatically identifying, via the agent, a resource associated with the virtual machine and, based on the resource, applying a monitoring policy to the virtual machine…”; Fig 11: Examiner notes, agent installation in progress, allocation of agents can be changed. Further, agents can be automatically installed when new VMs are identified.) and to perform parallel performance monitoring operations to monitor performance of each virtual machine of the new number of virtual machines. (Zada, [0071]: “The example agent monitor 130 of FIG. 1 receives resource report(s) for the new VM(s) from the installed agents (block 312). The resource reports may be generated by the agents 132 of FIG. 1 periodically, aperiodically, and/or in response to an event (e.g., on initial installation and/or execution of the agents 132). The resource reports list the resources and/or provide information about the resources (e.g., platform, servers, services, etc.) executing on the VMs 114. Based on the resource report(s) from the agents 132, the example agent monitor 130 registers the identified resources of the VMs 114 (block 314). For example, the agent monitor 130 may add the resources to an inventory and/or include the resources in a listing or catalog of resources available for monitoring.”).
Shah in view of Zada and Coon does not teach wherein the first hardware logic is to be reallocated in response to a change in the number of virtual machines to a new number of virtual machines. (Examiner notes, though Shah in view of Zada and Coon does teach changing the number of virtual machines (see current rejection, immediate above), the reference is silent as to whether the first hardware logic is reallocated in response to a change in the number of virtual machines occurs).
However, in analogous art, Glossner teaches wherein the first hardware logic is to be reallocated in response to a change in the number of software threads to a new number of software threads. ([0069]: “Referring again to FIGS. 6-10, in one example, the instruction issue logic 608 of the processing device 600 (e.g., the multithreaded processor 600) may be configured to identify a set of software threads having instructions waiting to issue. For each software thread in the set of the software threads, the instruction issue logic 608 may be configured to bind the software thread to an available hardware context (e.g., 602a) in a set of hardware contexts (e.g., 602a-602n).”; [0071]: “In an example, if the processing device 600 (e.g., the multithreaded processor 600) adds a new software thread to the software threads having instructions waiting to issue, the instruction issue logic 608 may be configured to bind the new software thread to an available hardware context (e.g., 602b) in a set of available hardware contexts (e.g., 602a-602n) and to store an identifier (e.g., a CID 800a) of the available hardware context (e.g., 602b) bound to the new software thread to a next available entry in an ordered list of identifiers 1008 (e.g., the ACID array register 900).”).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the reallocation of hardware contexts in response to new software threads in Glossner with the repartitioning of agents in response to a change of the number of virtual machines in Shah in view of Zada and Coon. As a result, when the number of virtual machines changes, the hardware contexts will be reallocated to the new number of VMs, and monitoring agents repartitioned to the new number of VMs as well. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to avoid the performance decrease from certain conventional multi-threading technologies when there are more hardware contexts on a processor available then software threads (see at least Glossner, [0015]-[0017] and [0021]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Shah et al. US 20200410628 A1 (“Shah”) in view of Zada et al. US 20150154039 A1 (“Zada”) and Coon et al. US 20090240860 (“Coon”), in view of Current et al. US 11410984 B1 (“Current”).
Regarding claim 10, Shah in view of Zada and Coon teaches the apparatus of claim 1.
Shah further teaches wherein the first hardware logic comprises tiles integrated on a processor package, (Fig. 25; [0249]: “FIG. 25 illustrates one embodiment in which the graphics processing resources of the graphics processor 2590 are subdivided into a plurality of tiles 2591-2594 where each tile comprises a set of graphics processing resources, as specified by tile configuration circuitry 2595. In the illustrated implementation, each tile 2591-2594 is directly coupled to a high bandwidth memory (HBM) 2596-2599, respectively, operating in accordance with the HBM2 standard. However, the underlying principles of the invention are not limited to any particular memory interface standard. In one embodiment, a set of interconnected schedulers 2580 schedule operations on the processing resources associated with each respective tile 2591-2594.”)
and at least some of the tiles interconnected horizontally through a bridge, ([0316]: “The coupling of the set of processors and other components is typically through one or more busses and bridges (also termed as bus controllers).”)
one or more of the tiles comprising interconnects or interfaces to couple the processor package to another processor package. ([0316]: “The coupling of the set of processors and other components is typically through one or more busses and bridges (also termed as bus controllers).”).
Shah in view of Zada and Coon does not teach at least some of the tiles stacked vertically and interconnected through vertical interconnects.
However, in analogous art, Current teaches at least some of the tiles stacked vertically and interconnected through vertical interconnects. (Fig. 17; Col. 20, lines 39-48: “In FIG. 17, the upper device layer 140 is shown in a face-up orientation, so that the substrate of the upper device layer faces the INL 130 and circuitry of the upper device layer faces away from the INL. Accordingly, the upper device layer 140 is electrically coupled to INL 130 through vertical connection vias 144. The process S222 of forming upper layers may include forming openings in the upper device layer and depositing a metal material in the openings to form the vertical connection vias 144.”).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to combine the 3DIC stacking fabrication technique in Current with the tiles of Shah in view of Zada and Coon, resulting in the tiles being vertically stacked and interconnected. A person having ordinary skill in the art would have been motivated to make this combination, with a reasonable expectation of success, to overcome the “limitations for performance, thermal, economic[,] component availability”, and internal stress in conventional 2.5D packaging techniques (see at least col. 1, line 35 – col. 2, line 5). In solving some of the previously listed problems, Current states: “To address the challenges outlined above, high bandwidth and concomitant high signal transfer rates in 3D multi-chip assemblies call for lateral (as well as vertical) signal interconnect networks which span beyond the lateral dimensions of the individual chip level circuits. The lateral dimensions of an individual chip circuit are constrained by the maximum field of view of available high-resolution optical lithography, at present, 2.6 by 3.3 cm. Embodiments of the layer transfer methods described by the present disclosure for formation of high bandwidth interconnections within a 3DIC stack are well suited for wafer-level transfers. This property provides the ability to fabricate signal interconnect layers which extend over the combined lateral dimensions of even the largest chip components by use of available lithographic edge of view alignment methods.” (col. 2, lines 40-54).
Response to Arguments
Applicant’s arguments regarding the 103 rejections have been fully considered and are persuasive. Therefore, the rejections have been withdrawn. However, upon further consideration, new ground(s) of rejection are made in view of Coon et al. US 20090240860 (“Coon”).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Arpan P. Savla whose telephone number is (571)272-1077. The examiner can normally be reached M-F, 10AM-6PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Cottingham can be reached at 571-272-1400. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Arpan P. Savla/Supervisory Patent Examiner, Art Unit 2137
Prosecution Insights