DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to under 37 CFR 1.83(a). The drawings must show every feature of the invention specified in the claims. Therefore, the:
Plurality of virtual GPU instances with workload distribution and resource allocation.
Compute fences and memory barriers.
must be shown or the feature(s) canceled from the claim(s). No new matter should be entered.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 10-17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Sundberg et al. (US 20200250254 A1, hereinafter “Sundberg”) in view of Suresh (US 20190019267 A1, hereinafter “Suresh”).
Regarding claim 11,
Sundberg teaches:
A non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors associated with a cloud-based system to perform steps of (Sundberg: Claim 1, “A non-transitory computer-readable medium that provides instructions that, when executed by a processor, cause the processor to perform operations, comprising. . .”; ¶21, “. . . the separate computing system may be hosted entirely in one or more server computing devices that are located in a different premises as the rendering device (e.g., using cloud computing for the one or more server computing devices). . . .”; ¶24, “. . . cloud based server infrastructure mechanism, may be used for securely communicating with the client device. . .”):
receiving a rendering request from a client device (Sundberg: ¶51, “. . . In block 401, a web server (for example, Web Server 101) receives a client request for web browsing from a client computing device. . .”; NOTE: The server receives a request from a client for web browsing. A request to view a web browser is a rendering request to view the browser. Therefore, the rendering request is the web browsing request of the client computing device.);
analyzing the rendering request (Sundberg: ¶51, “. . . an administrator associated with the client computing device (also referred to as an endpoint) may configure settings associated with a network on which the client computing device resides to automatically redirect browsing requests to the web server. The request typically includes one or more of: a URL associated with a web page that the user intends to browse, an indicator of an operating system of the client computing device, an indicator of a type of browser or other application through which the user initiated the request, an indicator of a current status or state of one or more caches on the client computing device or content in the one or more caches (for example, a resource cache or others), an indicator (e.g., a cookie) that identifies whether the user or client computing device is associated with one or more browsing sessions contemporaneously executed on the web server or previously executed on the web server and now terminated, or an indicator of one or more portions of settings or configuration information of the client computing device, the browser or another application on the client computing device, or the network in which the client computing device resides. . .”; NOTE: The webserver analyzes the rendering request (web browser request) by determining the URL of the requested web page, browser type, cache status, cookie, network.);
generating rendering instructions (Sundberg: ¶68, “. . . In one example remote browser 601, the layer code for layers 606a-b is intercepted, as described above with respect to FIG. 5, to extract draw commands for what is to be rendered . . .”; NOTE: The draw commands constitute rendering instructions because they are instructions for what is to be rendered); and
pushing the rendering instructions to the client device (Sundberg: ¶68, “. . . The remote browser 601 then potentially encodes the rendering output (the draw commands for regions such as quads to be rendered) in block 607a, caches desired encoded rendering output in block 607b, and compresses the encoded output in block 607c. This encoding, caching, and/or compressing can be performed according to a protocol 607 to be used to transport the rendered output across the secure connection. . .”; NOTE: Figure 6 illustrates the flow of the rendering instructions (draw commands) from being generated at the server step 605, packaged at step 607, and pushed to the local client device via internet/network).
However, Sundberg fails to teach: determining a workload distribution across a plurality of virtual Graphics Processing Unit (GPU) instances based on the analyzing; and executing rendering tasks across the plurality of virtual GPU instances.
The analogous art Suresh teaches:
determining a workload distribution across a plurality of virtual Graphics Processing Unit (GPU) instances based on the analyzing (Suresh: ¶47, “. . . Each virtual machine 332 may have a virtual disk 326A-C, a virtual processor 328A-C, and in some instances, one or more virtual graphics processing devices . . .”; ¶71, “. . . virtual GPU manager 522 may generate a first super-GPU view which may be a first logical linkage of a first group of available physical GPUs, a second super-GPU view which may be a second logical linkage of a second group of available physical GPUs, a third super-GPU view which may be a third logical linkage of a third group of available physical GPUs, and so on.. . . .”; ¶99-101,“. . . step 615, again in instances in which computing device 501 is one or more server computers and endpoint device 530 is a user computing device, graphics data streamer 528 may receive the graphical processing request from endpoint device 530 . . . virtual GPU manager 522 may receive the graphical processing request from the operating system. . . step 616, virtual GPU manager 522 may map the graphical processing request from either the endpoint device 530 or the operating system and/or another application operating on computing device 501 to the logical GPU by way of the one or more virtual graphics driver(s) 526 based on one or more of the received queried information. . . aspects of the graphical processing request that require high processing capacity (e.g., collision detection, animation, morphing, acceleration techniques using spatial subdivision schemes, model and camera transformation, lighting, projection, clipping, window/viewport transformation, and rasterization) may be mapped to one or more of the discreet GPUs 514A-514N and/or integrated CPU/GPU(s) 512 with high processing capacity within the logical GPU and aspects of the graphical processing request that do not require high processing capacity (e.g., pixel formatting, frame optimization, hardware encoding, and image processing techniques such as sharpening and watermarking, and the like) may be mapped to one or more of the discreet GPUs 514A-514N and/or integrated CPU/GPU(s) 512 with light processing capacity within the logical GPU”; ¶104, “. . . aspects of the graphical processing request requiring high processing capacity may be mapped by virtual GPU manager 522 to a first group of one or more of the super-GPU views of the multiple super-GPU model and aspects of the graphical processing request that do not require high processing capacity may be mapped by virtual GPU manager 522 to a second group of one or more of the super-GPU views of the multiple super-GPU model. . .”; NOTE: The server creates virtual machines with virtual GPUs (super-GPUs) created by the virtual GPU manager (¶47, 71). When a rendering request (graphical processing request) is received by the server, the server analyzes the rendering request and determines the workload for high processing capacity and light processing capacity and distributes the workload to a first group/second group of super-GPUs based on analyzing the rendering request);
and executing rendering tasks across the plurality of virtual GPU instances and generating rendering instructions (Suresh: ¶72, “. . . During performance of graphical rendering requests, the first logical grouping may be responsible for performing rendering operations requiring high-load processing capacity, whereas the second logical grouping may be responsible for performing post-processing operations requiring light-load processing capacity. For instance, the first logical grouping may perform processes such as collision detection, animation, morphing, acceleration techniques using spatial subdivision schemes, model and camera transformation, lighting, projection, clipping, window/viewport transformation, rasterization, and the like. The second logical grouping may perform processes such as pixel formatting, frame optimization, hardware encoding, and image processing techniques such as sharpening and watermarking. . .”; NOTE: The different groups corresponding to the super-GPUs executes different rendering tasks such as for example, animation in the first group, and frame optimization in the second group of super-GPUs. These rendering tasks executed by the super GPU’s creates rendering instructions such as animation and frame optimization.).
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Sundberg, and Suresh and include: determining a workload distribution across a plurality of virtual Graphics Processing Unit (GPU) instances based on the analyzing; and executing rendering tasks across the plurality of virtual GPU instances. (NOTE: Instantiate super-GPU’s as taught by Suresh >> Receive rendering request as taught by Sundberg/Suresh >> distribute rendering tasks to multiple VGPU to generate rendering instructions as taught by Suresh >> Intercepting rendering instructions to be packaged and pushed to the client as taught by Sundberg.)
The reason for doing so is for “leveraging multiple graphics processors, by a virtual GPU manager, to optimize the rendering of graphics” (Suresh: ¶1) because “The underutilization of the total available graphics processing power results in a sub-optimal scenario where extra CPU cycles are spent on handling the data flow through the graphics processing pipeline. The rendering operations are serialized and the graphics processing pipeline may stall when heavy-duty workload is executed leading to deteriorating graphics performance and quality.” (Suresh: ¶2).
Regarding claim 12, depending on 11,
The combination of Sundberg, and Suresh teaches:
The non-transitory computer-readable medium of claim 11,
Sundberg further teaches:
wherein the steps comprise, prior to the receiving, initiating a browser isolation session between the client device and a server associated with the cloud-based system (Sundberg: ¶51, “FIG. 4 is an example overview flow diagram of the logic for initiating an isolated remote web browser session using an example Adaptive Rendering Application Isolation System. In block 401, a web server (for example, Web Server 101) receives a client request for web browsing from a client computing device (for example, one of client devices 104a-104d). The request may be an HTTP/HTTPS request. The request may be initiated by the user visiting or logging into a particular web page that results in the client computing device transmitting or forwarding of the request to the web server. . .”; NOTE: The client must make connection first to the server before the server knows what webpage the client is requesting, therefore, the browser isolation session is initiated prior to receiving the rendering request (web browsing request to generate and render a web page)).
Regarding claim 13, depending on 11,
The combination of Sundberg, and Suresh teaches:
The non-transitory computer-readable medium of claim 11,
However, Sundberg fails to teach: wherein the steps comprise, prior to the receiving, initializing a plurality of virtual GPU instances within one or more servers of the cloud-based system.
Suresh further teaches:
wherein the steps comprise, prior to the receiving, initializing a plurality of virtual GPU instances within one or more servers of the cloud-based system (Suresh: ¶104, “. . . the logical GPU created by virtual GPU manager 522 is of a multiple super-GPU model, . . .”; ¶119, “. . . At step 710, virtual GPU manager may query each of the plurality of physical GPUs to identify processing performance variables of each of the plurality of physical GPUs. At step 715, the virtual GPU manager may generate a logical GPU corresponding to one or more of the plurality of physical GPUs. At step 720, the virtual GPU manager may receive a rendering request . . .”; ¶47, “. . . Each virtual machine 332 may have a virtual disk 326A-C, a virtual processor 328A-C, and in some instances, one or more virtual graphics processing devices . . . “; NOTE: In reference to Suresh: Fig. 7, the Logical GPU, which is of a multiple super-GPU, constituting to the plurality of virtual GPU instances, is created by the virtual GPU manager at step 715. Step 715 is a prior to step 720, where step 720 is when the rendering request is being received. Therefore, the super-GPUs are initialized prior to the receiving of the rendering request.)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Sundberg, and Suresh and include: wherein the steps comprise, prior to the receiving, initializing a plurality of virtual GPU instances within one or more servers of the cloud-based system.
The reason for doing so is for “leveraging multiple graphics processors, by a virtual GPU manager, to optimize the rendering of graphics” (Suresh: ¶1) because “The underutilization of the total available graphics processing power results in a sub-optimal scenario where extra CPU cycles are spent on handling the data flow through the graphics processing pipeline. The rendering operations are serialized and the graphics processing pipeline may stall when heavy-duty workload is executed leading to deteriorating graphics performance and quality.” (Suresh: ¶2).
Regarding claim 14 depending on 13,
The combination of Sundberg, and Suresh teaches:
The non-transitory computer-readable medium of claim 13,
Suresh further teaches:
wherein the steps comprise allocating computational resources including memory (Suresh: ¶8, “. . . the first logical grouping and second logical grouping share a common memory allocation. . .”; ¶73, “. . . Virtual GPU manager 522 may further be configured to create shared memory heaps 518 (e.g., cross-shared memory heaps) as one or more memory elements in physical memory 516. Shared memory heaps 518 may be a shared memory space associated with graphics runtime 524 which may serve as a commonly and/or mutually accessible data allocation area for the available physical GPUs corresponding to the logical GPU(s). . .”)
and processing resources to each of the plurality of virtual GPU instances (Suresh ¶71, “. . . Virtual GPU manager 522 may distribute processing power equally across each of the super-GPU views or, alternatively, may allocate the available physical GPUs between the super-GPU views based the processing performance variables in a task specific manner. . .”; ).
Regarding claim 15, depending on 11,
The combination of Sundberg, and Suresh teaches:
The non-transitory computer-readable medium of claim 11,
Suresh further teaches:
wherein the steps comprise distributing the rendering tasks in parallel across the plurality of virtual GPU instances (Suresh: ¶71, “. . . a plurality of super-GPU views may be generated in order to facilitate graphically computational intensive applications such as split-screen rendering. . .”; NOTE: Split screen rendering constitute in parallel rendering. Because both screens are rendered simultaneously, a virtual GPU instance (super-GPU instance) processes the first screen, and another instance for the second screen. In order for the split-screen to be rendered at the same time, the rendering tasks for the first and second screen are distributed in parallel to the super-GPU instances.
Regarding claim 16, depending on 11
The combination of Sundberg, and Suresh teaches:
The non-transitory computer-readable medium of claim 11,
Sundberg further teaches:
wherein the executing includes performing all read operations locally within the cloud-based system (Sundberg: ¶4, “. . . securing an application is to execute the application remotely on a server instead of locally on a client. . .”; ¶32, “. . . Once the secure connection 120 is established, the general flow of data between the web application 105 and the remote application instance 107 in the secure container 103 is that input received at the web application 105, such as keystrokes, mouse, and other cursor and input events, is intercepted by the remoting code previously integrated into the web application 105 and forwarded to the remote application instance 107 via the secure connection 120. The remote application instance 107 then performs whatever execution is being requested, including, for example, downloading web pages or content via a third party website 130, applying stylesheet definitions (.css), and executing JavaScript for web browser applications, to generate rendering output. The rendering output is then packaged, including optional encoding, optimization, and enhancing, and forwarded via the secure connection 120 back to the web application. . .”; NOTE: Executing the application on a server instead of on a client constitutes to the server performing all read operations of the application being requested (i.e web browser) locally in the server side instead of the client device. Also, the flow of data is one way. In reference to Fig. 6, the client only requests a content, such as web browsing. The server performs all read operations of the requested content such as reading the DOM tree at 604, intercepts the draw commands at 606, and only transmits the intercepted draw commands to the client. The client never performed any readings on the content.).
Regarding claim 17, depending on 11,
The combination of Sundberg, and Suresh teaches:
The non-transitory computer-readable medium of claim 11,
Sundberg further teaches:
wherein the rendering request is associated with any of a remote application (Sundberg: ¶72, “The techniques described with respect to FIG. 6 may also be incorporated in remoting applications that are not web browser based”; ),
a browser (Sundberg: ¶51, “. . . initiating an isolated remote web browser session. . .”),
and an entire desktop environment (Sundberg: ¶3, “. . . applications are run and to “remote” the desktop so that it runs in a protected space (e.g., a sandbox or virtual machine) on a server computing system . . .”).
Regarding claim 20, depending on 11,
The combination of Sundberg, and Suresh teaches:
The non-transitory computer-readable medium of claim 11,
Sundberg further teaches:
wherein only rendering instructions to render an output are provided to the client device (Sundberg: ¶48, “. . . extract draw commands that correspond to the portions of the page that would otherwise be rasterized. This type of interception results in a system that can forward packets of drawing commands (high level of graphics interception) instead of pixel pushing . . .”; NOTE: The draw commands constitute rendering instructions. In reference to Fig 6., only the draw commands are intercepted, extracted, encoded, and then transmitted to the client device so it can render an output at 614, and 617.)
Regarding method claims 1-7, and 10,
Method claims 1-7, and 10 are drawn to the methods corresponding to the instructions of using same as claimed in CRM claims 11-17, and 20. Therefore, method claim 1-7, and 10 correspond to the instructions in the CRM of claim claims 11-17, and 20, and are rejected for the same reasons of obviousness as used above.
Claims 8-9, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Sundberg in view of Suresh further in view of Bolz et al. (US 20110063313 A1, hereinafter “Bolz”)
Regarding claim 18, depending on 11,
The combination of Sundberg, and Suresh teaches:
The non-transitory computer-readable medium of claim 11,
Although the combination of Sundberg, and Suresh teaches analyzing rendering request such as web browser remoting (Sundberg), and distributing rendering task of the rendering request to multiple vGPU for optimization of resources (Suresh), intercepting and generating GPU/draw command stream transmitted to the client to be rendered on the clients device (Sundberg paragraph 48, packets of drawing commands forwarded to the client.), and although Sundberg uses proprietary synchronous client-server computing techniques (Sundberg paragraph 48), the combination fails to teach: wherein the steps comprise synchronizing states of the plurality of virtual GPU instances and a remote physical GPU of the client device.
The analogous art Bolz teaches:
Synchronization of multi-threaded graphics pipeline, alleviating synchronization inaccuracies of graphics command streams. (Bolz: ¶6, “. . . in a system that processes multiple primitives in parallel, a store operation issued by a shader when working on a first primitive might complete after a store operation for a second primitive, even if the first primitive was specified prior to the second primitive. This is problematic when the second primitive is dependent upon the first primitive. “ ¶34-36, “Programmable graphics pipelines typically execute a plurality of vertex, geometry, and fragment shader programs within the vertex shader 152, the geometry shader 154, and the fragment shader 156, respectively. The simultaneous execution of one or more programs within a processor is known in the art as "multi-threading." Such multi-threading results in execution order and synchronization inaccuracies of graphics command streams received . . . memory operations that are required by particular commands may be reordered for execution by the GPU in an order that was not originally specified within the graphics command stream . . . To alleviate the aforementioned synchronization inaccuracies while avoiding the overhead of the automatic synchronization mechanism discussed above, we can provide explicit synchronization commands into the received graphics command streams. Explicit synchronization ensures that the effects of buffer and texture data stores performed by one or more shader programs to a portion of memory are visible to subsequent commands that access the same portion of memory. For example, a graphics command stream may include one or more memory operations that are completed in an undefined order. To provide a defined order of execution, the GPU 150 may perform an explicit synchronization at various points within the graphics command stream. This can be accomplished by configuring the GPU 150 to track the execution state of each of the commands in order to effectively determine whether all commands have completed in execution”)
(NOTE: Suresh paragraph 48 discloses that the first group logical GPU that executes rendering operations and stores the data generated which are the rendering instructions in a shared memory heaps, where the second group of logical GPU have access to the stored data in the shared memory heap for post-processing. In reference to Bolz paragraph 6 as cited above, Bolz addresses a problem with parallel processing of primitives (i.e. Suresh multi super-GPU model). Suresh alleviates synchronization inaccuracies by embedding synchronization commands into the command stream (i.e. the draw commands packet generated by Sundberg) and track execution state to determine whether all commands have completed in execution.
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Sundberg, Suresh, and Bolz and include: wherein the steps comprise synchronizing states of the plurality of virtual GPU instances and a remote physical GPU of the client device. (NOTE: Instantiate super-GPU’s as taught by Suresh >> Receive rendering request as taught by Sundberg/Suresh >> distribute rendering tasks to multiple VGPU to generate rendering instructions as taught by Suresh >> Intercepting rendering instructions to be packaged as taught by Sundberg + embed synchronization command into the draw command stream as taught by Bolz >> push draw commands to the client for rendering as taught by Sundberg)
The reason for doing so is to provide “a mechanism for ensuring the coherency of memory read and write operations in a highly-pipelined system” (Bolz: ¶7) and to improve control of memory access operations (Bolz: 8) because “in a system that processes multiple primitives in parallel, a store operation issued by a shader when working on a first primitive might complete after a store operation for a second primitive, even if the first primitive was specified prior to the second primitive. This is problematic when the second primitive is dependent upon the first primitive” (Bolz: ¶6)
Regarding claim 19, depending on 18,
The combination of Sundberg, Suresh, and Bolz teaches:
The non-transitory computer-readable medium of claim 18,
Bolz further teaches:
wherein the synchronizing is performed via any of compute fences (Bolz: ¶38, “. . . alleviate the aforementioned synchronization inaccuracies while avoiding the overhead of the automatic synchronization mechanism discussed above, we can provide explicit synchronization commands into the received graphics command streams. . . To provide a defined order of execution, the GPU 150 may perform an explicit synchronization at various points within the graphics command stream. . .”; NOTE: In reference to paragraph 81 of applicant’s specification, “Compute fences help coordinate the execution order of the GPU tasks”, therefore, Bolz synchronization commands functions as the claimed compute fence as it provides a defined order of execution, constituting to coordinating execution order.)
and memory barriers (Bolz: ¶40, “The memory barrier command provides stronger ordering of read and write operations performed by a single thread. When a memory barrier command is executed, any memory operations issued by the thread prior to the memory barrier command are guaranteed to be completed before any subsequent memory operations are performed. Memory barrier commands are needed for algorithms that allow multiple threads to access the same memory location. . .”; ).
Regarding method claims 8-9,
Method claims 8-9 are drawn to the methods corresponding to the instructions of using same as claimed in CRM claims 18-19. Therefore, method claims 8-9 correspond to the instructions in the CRM of claims 18-19, and are rejected for the same reasons of obviousness as used above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PATRICK GALERA whose telephone number is (571)272-5070. The examiner can normally be reached Mon-Fri 0800-1700 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PATRICK P GALERA/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights