DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments/Arguments
Applicant’s amendments/arguments filed on 5/11/2026 have been fully considered.
Applicant’s arguments with respect to method claims 1, and 3, and corresponding CRM claims 11, and 13 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-5, 7-12, and 14-15, 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Sundberg et al. (US 2020/0250254 A1, hereinafter “Sundberg”) in view of Li et al. (US 20100214301 A1, hereinafter “Li”).
Regarding claim 1, Sundberg discloses:
a method comprising steps of:
initiating, by a cloud-based system, a browser isolation session between a user device and a server associated with the cloud-based system (Sundberg: Fig. 4, par. 51, "overview flow diagram of the logic for initiating an isolated remote web browser session using an example Adaptive Rendering Application Isolation System"; Sundberg: par. 5, “One manner of securing an application is to execute the application remotely on a server instead of locally on a client device where the hosted remoted application can be protected inside of a sandbox, such as a virtual machine. When the application is a web browser, this is sometimes referred to as “browser isolation. . .”; Sundberg: par. 24, “the web server 101 securely communicates (e.g., via public/private key encryption) with the client device 104a. For instance, the load balancer 106 or other secure component provided, for example by AWS, AZURE, or another cloud based server infrastructure mechanism, may be used for securely communicating with the client device 104a. . .”; NOTE: Sundberg’s ARAIS is a browser isolation system. It is cloud-based system as its web server uses cloud based server infrastructure mechanism. The user device is the client device 104a.);
receiving, at the server, a request for a resource from the user device (Sundberg: Fig. 4, 401, 404, par. 51, " In block 401, a web server (for example, Web Server 101) receives a client request for web browsing from a client computing device (for example, one of client devices 104a-104d). The request may be an HTTP/HTTPS request. The request may be initiated by the user visiting or logging into a particular web page that results in the client computing device transmitting or forwarding of the request to the web server"; par. 53, "In block 404, the orchestrator server determines the appropriate graphics level to use for transporting rendering data to the client computing device (see, for example, FIG. 7) and instantiates a remote web browser application in a secure container (such as secure containers 206a-c) based upon one or more characteristics of the client computing device (for example, the client operating system, client application or version, network settings or configurations enforced by an IT network administrator for the network in which the client computing device resides, bandwidth of the connection with the client computing device, or others)."; NOTE: The resource requested by the user using a user device is the client request for web browsing based on the user device’s OS, app version, network settings as described in par 53.);
executing, at the server, code associated with the resource within a secure, sandboxed execution environment of the cloud-based system (Sundberg: ¶21, “. . ., in example ARAIS environments, the isolated execution environment may be any type of separate environment that allows the isolation-enabled application to be executed independently from where its output is ultimately rendered. . . the isolation execution environment may be located on a separate computing system from the rendering device, and the separate computing system may be hosted entirely in one or more server . . . (e.g., using cloud computing for the one or more server computing devices). . . When using the same computing system, a separate instance of the application could be executed in a sandboxed (e.g., containerized or virtualized) environment and used to perform code execution and send rendering data to the instance of the application ultimately responsible for rendering output. Such an architecture allows the application code to be executed safely and separately (NOTE: Sundberg’s system receives request from a client/user device for web browsing >> instantiates remote web browser application in a secure container which is a sandboxed environment based on the client’s device characteristics >> the code associated with the resource, is the application code executed in the sandbox safely. Safely = secure),
Further, Sundberg’s ARAIS is configured to:
compile shaders (Sundberg: par. 97, “. . . providing intermediate render commands . . . combined with compiled shaders to draw the curves. . .”),
process graphical instructions (Sundberg: ¶86, “. . . the graphics engines or libraries can be directly inspected or modified, and the highest-level graphics library of the remote application is modified to intercept and output called draw commands and corresponding position information to facilitate providing (forwarding, transmitting, sending, etc.) those draw commands to the client computing device. . .”; NOTE: The graphical instructions are the draw commands processed so it can be forwarded to the client device),
and generate GPU command streams corresponding to rendering operations (Sundberg: ¶34, “. . . transport vector commands (draw commands), or even commands translated to a lower level in the graphics pipeline (such as using OpenGL, WebGL, DirectX, Vulkan, or other GPU commands). . .”; Sundberg: par. 86, “. . . the graphics engines . . . intercept and output called draw commands and corresponding position information to facilitate providing (forwarding, transmitting, sending, etc.) those draw commands to the client computing device. . .”; Sundberg: par. 71, “The local browser renderer 614 then executes the modified draw commands 616 using the standard browser's rendering code (or ARAIS code in the integrated Web Assembly code 611) to render composited draw commands 616 to the WebGL or other canvas 617. . .” ; NOTE: The generated GPU commands are the outputted draw commands streamed, or transmitted to the client device for rendering. The draw commands correspond to rendering operations because once the client device receives the draw commands processed and streamed from the server, the draw commands are executed in the client’s device for rendering. Sundberg’s FIG. 6 illustrates the process of processing draw commands in in remote browser 601 >> encoded in the server >> stream over internet/network to client’s device >> decoded by the client device >> execute draw commands for rendering),
producing no framebuffer output and no pixel data at the server (Sundberg: ¶48, “ARAIS, the Chromium code is altered to capture the draw calls that relate to “tiles” that are being composited and would otherwise be rasterized (into pixels/bitmaps) . . . This type of interception results in a system that can forward packets of drawing commands (high level of graphics interception) instead of pixel pushing”; ¶86, “. . . the remote application can be modified to prevent rasterization of pixels to reduce expensive computations. . .”; NOTE: Sundberg intercepts the draw calls to prevent pixel rasterization, resulting in producing no pixel data, if there is no pixel data that is rasterized, then there are no pixels written into the framebuffer for rendering, and therefore no framebuffer output is produced. Also, Fig. 5 shows the interception of the draw commands.),
without performing rasterization, bitmap generation, or any graphical rendering at the server (Sundberg: ¶86, “. . . the remote application can be modified to prevent rasterization of pixels to reduce expensive computations while still obtaining position information for the intercepted draw commands . . .”; NOTE: The remote application of Sundberg is in a server. Because Sundberg’s remote application prevents rasterization of pixels, therefore Sundberg’s system processes draw commands without performing rasterization, bitmap generation, or any graphical rendering at the server to reduce expensive computations.);
encoding the GPU command streams at the server (Sundberg: Fig. 6 reference 607a, par. 68, “The remote browser 601 then potentially encodes the rendering output (the draw commands for regions such as quads to be rendered) in block 607a. . .” NOTE: Sundberg’s encoding happens at the server as illustrated in Fig. 6., block 607 where encoding is done, is within the remote browser 601.),
transmitting the encoded GPU command streams to the user device over a secure network connection established by the cloud-based system (Sundberg: par. 24, “. . . All communication between applications running on the client device 104a and the remote execution environment 100 is through the web server 101 using HTTP/HTTPS until the orchestration server 102 is able to establish a secure connection 120 between the web application 105 executing on the client device 104a and the corresponding remote application instance 107 within the secure container 103 . . .”; Sundberg: par. 68, “. . . The remote browser 601 then potentially encodes the rendering output (the draw commands for regions such as quads to be rendered) in block 607a, . . . This encoding, . . . to be used to transport the rendered output across the secure connection . . .”);
decoding, by the user device, the encoded GPU command streams using a GPU command decoder that maps graphical resources (Sundberg: Fig. 6 ref 613b, par. 70, “The corresponding protocol 612 on the local browser . . . to decode, decompress, and/or recall data such as in blocks 613a-c. Specifically, a cache (for example, a resource cache or a local cached library) on the local browser 610 (not shown) stores resources, such as images, textures, fonts, shapes, . . . Once the local browser 610 receives an identifier in one of the decode and decompressed (and depacketized) messages, it can retrieve the corresponding cached item from its cache”; NOTE: Sundberg’s decode block is executed inside the Local Client Browser 610 as illustrated in Fig. 6. Draw commands processed and encoded at the server >> draw commands streamed over secure network >> Local Client Browser at the user device receives draw commands >> decompress >> decode draw commands >> local cache stores graphical resources such as images, textures, fonts >> protocol 612 then maps the graphical resources using an identifier to retrieve the graphical resources),
and synchronizes execution order (Sundberg: par. 25, “. . . the secure connection 120 is implemented using TCP/IP connections (which uses sequenced packets). . .”; Sundberg: par. 69, “. . . Each draw command is embedded in a message that contains the minimal amount of data needed to replicate that draw command on the local browser 610. . .”; par. 32, “. . . input received at the web application 105, such as keystrokes, mouse, and other cursor and input events, . . . performs whatever execution is being requested, . . . to generate rendering output. The rendering output is then packaged, including optional encoding, optimization, and enhancing, and forwarded via the secure connection 120 back to the web application 105 to be rendered on a display of the client device 104a . . .”; NOTE: Fig 6. Remote Browser 601 encodes and embeds the draw command in a message with ordered or sequenced packets >> streamed via secure network to the client >> client browser decodes the message with the sequenced packets >> render at 617. Since the draw commands are sent using sequenced or ordered packets, the local client device receives and decodes the draw commands in sequenced order including keystrokes and mouse events for rendering, this inherently synchronizes the execution of the decoded draw commands in order so it can be replicated on the local browser. If the execution is not in synchronized order, then the keyboard or mouse events will not move or display as intended.),
and rendering, within a browser executing a Web Graphics (WebGL) environment, graphics based on the decoded GPU command streams (Sundberg: Fig. 6 614 renderer, 617 WebGL canvas; Sundberg: par 71, “. . . The local browser renderer 614 then executes the modified draw commands . . . to render composited draw commands 616 to the WebGL or other canvas 61 . . .”; Sundberg: par. 83, ". . . the client application is configured to render visual output of the remote application instance based on draw commands received from the remote application. . .”).
Although Sundberg teaches system and methods for executing, at the server, code associated with the resource within a secure, sandboxed execution environment of the cloud-based system that compile shaders, process graphical instructions, and generate GPU command streams corresponding to rendering operations, without performing rasterization, bitmap generation, or any graphical rendering at the server, by intercepting draw commands from the application code, Sundberg fails to disclose “using a simulated Graphics Processing Unit” configured to perform the above functions, and therefore fails to teach: using a simulated Graphics Processing Unit (GPU) configured to compile shaders, process graphical instructions, and generate GPU command streams corresponding to rendering operations, that exposes a graphics application programming interface (API) to the executed code, the simulated GPU compiling shaders, processing graphical instructions received through the graphics API, and generating GPU command streams corresponding to rendering operations from graphics calls made through the graphics API by the executed code.
The analogous art Li teaches:
a simulated Graphics Processing Unit (GPU) (Li: ¶44, “FIG. 4 shows an exemplary layered architecture 303 that includes a VGPU emulator 400 to emulate a GPU (see, e.g., the GPU 150). . .”; ¶15, “An exemplary GPU emulation framework, referred to as a virtual GPU (VGPU), can emulate functionality of a specific, target GPU. . .”; NOTE: Because the VGPU is emulating a specific GPU, then it is simulating a GPU.),
that exposes a graphics application programming interface (API) to the executed code (Li: ¶44-47, “. . . the VGPU emulator 400 emulates a guest GPU (e.g., GPU 150) and relies on the API/runtime layer 320 to control the host GPU. . . The API/runtime layer 320 abstracts and exposes host GPU functionality. . . The VGPU emulator 400 emulates a GPU at the binary level. The inputs of VGPU are the low level command buffers, shaders and graphics resources generated by the application 310 code at run-time. . . In the VGPU emulator 400, the command processor 410 parses low level graphics commands and generates intermediate VGPU commands. . .”; NOTE: The API/runtime layer 320 is exposed by the VGPU to the application 310 code during runtime in order to parse low level graphics command for further processing.),
the simulated GPU compiling shaders (Li: ¶53, “. . . VGPU 400 is responsible for accessing the shader or shader commands (e.g., optionally shader code) and transferring the shader or compiling the shader via the shader compiler 438 and placing the shader in the shader cache 434, which becomes accessible by the back-end 408 of the VGPU 400. . .”),
processing graphical instructions received through the graphics API (In the VGPU emulator 400, the command processor 410 parses low level graphics commands and generates intermediate VGPU commands. The draw engine 420 receives the intermediate VGPU commands and then generates the resulting host Direct3D.RTM. calls),
and generating GPU command streams corresponding to rendering operations from graphics calls made through the graphics API by the executed code (Li: ¶16, “. . .the VGPU parses emitted graphics commands and converts them to intermediate representations. In essentially concurrent operation, the shader and other graphics resources (e.g., vertex and texture data) are loaded and converted from guest GPU formats to host GPU formats. The graphics resources and shaders are correctly linked and packed in a commands stream for the host GPU. The VGPU executes intermediate representations (e.g., graphics commands) in the form of host graphics commands on the host platform. . .”; ¶51, “To arrive at the rendered result . . . transmits low level graphics commands to a command buffer (see, e.g., the packed commands 130 of FIG. 1 and the commands of the GPU video memory 160 of FIG. 2. . .”;).
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Sundberg, and Li such that Sundberg’s browser isolation method by: using a simulated Graphics Processing Unit (GPU) configured to compile shaders, process graphical instructions, and generate GPU command streams corresponding to rendering operations, that exposes a graphics application programming interface (API) to the executed code, the simulated GPU compiling shaders, processing graphical instructions received through the graphics API, and generating GPU command streams corresponding to rendering operations from graphics calls made through the graphics API by the executed code, the simulated GPU producing no framebuffer output and no pixel data at the server.
The reason for doing so is “to produce the same rendering results in about the same amount of time regardless of the GPU used” and “allow for real time emulation of a target GPU and can be used to assess results and performance of a graphics application without physically porting the application to a computing platform with the actual target GPU”. (Li: ¶2-3).
Regarding claim 2, depending on claim 1,
The combination of Sundberg and Lei teaches:
The method of claim 1,
Sundberg further teaches:
wherein the steps further comprise displaying the graphics within a browser operating on the user device (Sundberg: par. 32 "The remote application instance 107 then performs whatever execution is being requested, including, for example, downloading web pages or content via a third party website 130, applying stylesheet definitions (.css), and executing JavaScript for web browser applications, to generate rendering output. The rendering output is then packaged, including optional encoding, optimization, and enhancing, and forwarded via the secure connection 120 back to the web application 105 to be rendered on a display of the client device 104a using the remoting code previously integrated into the web application 10").
Regarding claim 4, depending on claim 1,
The combination of Sundberg and Lei teaches:
The method of claim 1,
Sundberg further teaches:
wherein all code associated with the resource is executed at the server prior to the sending (Sundberg: Fig. 6, par. 21, "Although this disclosure typically refers to the isolated execution environment as a server computing system and the local rendering environment as a client computing system, in example ARAIS environments, the isolated execution environment may be any type of separate environment that allows the isolation-enabled application to be executed independently from where its output is ultimately rendered”; par. 21, “When using the same computing system, a separate instance of the application could be executed in a sandboxed (e.g., containerized or virtualized) environment and used to perform code execution and send rendering data to the instance of the application ultimately responsible for rendering output. Such an architecture allows the application code to be executed safely and separately from the process responsible for rendering output, for example, to an end user. In addition, separation of code execution from output rendering can lead to performance gains in some applications. For example, code execution could be executed on a machine with significantly higher network or CPU resources").
Regarding claim 5, depending on claim 1,
The combination of Sundberg and Lei teaches:
The method of claim 1,
Sundberg further teaches:
wherein code associated with the resource is executed in a sandbox at the server (Sundberg: par. 21, "When using the same computing system, a separate instance of the application could be executed in a sandboxed (e.g., containerized or virtualized) environment and used to perform code execution").
Regarding claim 6, depending on 1,
The combination of Sundberg and Lei teaches:
The method of claim 1,
Sundberg further teaches:
wherein the GPU command streams are transmitted over the network to a browser executing on the user device (Sundberg par. 86, “. . . output called draw commands and corresponding position information to facilitate providing (forwarding, transmitting, sending, etc.) those draw commands to the client computing device. . .”; Also see Fig 6. NOTE: As illustrated in Fig 6, draw commands, or GPU commands are outputted by 605 within the remote browser 601 >> draw commands packaged at 607 >> transmitted via internet/network >> draw commands received by local client browser which is the browser executing on the user device.),
and wherein the GPU command streams commands are decoupled from and independent of code associated with the resource (Sundberg: par. 46, “. . . Separating the execution actions of the web browser from the rendering allows the ARAIS to minimize vulnerabilities potentially caused by downloading web pages or content via a third party website, execution of code such as JavaScript (embedded or as separately provided as a downloaded code file) . . .”; NOTE: Sundberg’s ARAIS system separates execution actions such as Javascript embedded on a web page. The Javascript code is the code associated with the resource. Separating the Javascript code from the rendering is decoupling the Javascript code from the rendering or draw commands which are the GPU commands making it independent.),
the GPU command streams being executed within a Web Graphics Library (WebGL) rendering context environment in the browser (Sundberg: par. 71, “. . . to render composited draw commands 616 to the WebGL or other canvas 617 . . .”; NOTE: Fig. 6. The local Client browser 610 executes draw commands at 616 and render to a WebGL canvas 617. Therefore, the draw commands, or the GPU command streams are being executed within a Web Graphics Library (WebGL) rendering context environment in the browser.),
while the code associated with the resource is executed exclusively within the secure, sandboxed execution environment of the cloud-based system (Sundberg: par. 46-47, “. . . secure container 300 contains an attribute and rules determiner 301 (module, engine, code logic, or the like), and a remote application instance 302 which may correspond to the intercepted request for a page by the client web browser . . . This instance runs in the isolation environment provided by the secure container 300 to restrict any malicious code or effects from the remote application instance 302 to that container. Thus, other client computing systems' remote instances are not infected by anything that may occur in remote application instance 302. . .”; NOTE: Paragraph 46 discloses decoupling of Javascript codes which could be malicious. When a client request web browsing, the server instantiates an instance of the web browser in a secure container. The secure, sandboxed execution environment of the cloud-based system is the secure container 300. ARAIS decouples or separates Javascript codes from the draw commands, only sending the draw commands to the client for rendering. The Javascript codes are the code associated with the resource. If the actions of the user on the web browser (i.e. clicking a link) has malicious Javascript code, it is restricted within the secure container, which is executing the malicious code within the secure sandboxed environment, and so the client systems are not infected by anything that may occur in the remote application instance, the remote application instance which is within the secure container. Therefore, any malicious codes that are associated with the resource such as a web browser for a client’s device are exclusively executed within the secure container to not infect the client’s system because the client’s system only receives the draw commands.),
the sandbox having restricted network access limited to managed communications of the isolation session (Sundberg: par. 24, “. . . All communication between applications running on the client device 104a and the remote execution environment 100 is through the web server 101 using HTTP/HTTPS until the orchestration server 102 is able to establish a secure connection 120 between the web application 105 executing on the client device 104a and the corresponding remote application instance 107 within the secure container 103 (isolated execution environment. . .”; Sundberg: par 38, “. . . an orchestrator server 200 may comprise remoting level determination logic and rules engine 201; . . . connection support logic 204; data repositories (storage) 208 for objects such as a render cache, whitelists, blacklists, client privileges, and application information; and one or more secure containers running remote application instances. . .”; Sundberg: par 45, “. . . the data repositories 208 may be used to store other information related to performance and/or security of the execution environments running in the secure containers 206a-c. For example, the data repositories 208 may store whitelists and/or blacklists of users, third party websites or content, privileges associated with client computing systems and/or users, and application information for applications that are able to be remote-enabled. . .”; NOTE: The isolation session is the remote application instance within the secure container. The sandbox is the secure container. The orchestration server manages the communication and restricts the communication between the client and the isolated execution environment until secure connection is established. Therefore, the sandbox, or the secure container have restricted network access until the orchestration server that manages the communications between the client and the isolated execution environment establishes a secure connection. Paragraph 38 and 45 further discloses whitelists, blacklists, and privileges. If an orchestrator have an object in the blacklist, then the secure container have restricted access to the blacklisted objects.).
Regarding claim 7, depending on claim 1,
The combination of Sundberg and Lei teaches:
The method of claim 1,
Sundberg further teaches:
wherein the resource is any of a remote application, a browser, or an entire desktop environment (Sundberg: par. 20, "the ARAIS provides a self-contained delivery mechanism for enabling existing applications such as web browsers to be “isolation-ready” such that they can immediately utilize the secure ARAIS environment"; par. 25, “In one example ARAIS, the secure connection 120 is implemented using WebRTC. WebRTC allows both UDP/IP connections (which uses non-sequenced packets) and TCP/IP connections (which uses sequenced packets) and thus provides great flexibility. In addition, when using UDP, the secure connection 120 is typically faster. Other secure connections 120 can be used, including for example WebSockets, RDP, and proprietary or other standard known solutions”; par. 29, “In conjunction with configuring a corresponding remote application instance 107 within the secure container 103, the orchestration server 102 causes remoting code/logic and potentially other resources) to be downloaded to the application 105 running on the client device 104a via communications paths 110 and 111”; Sundberg: ¶22, “FIG. 1 is an example block diagram of an example secure computing environment for executing remoted applications in isolation using an example Adaptive Rendering Application Isolation System. The remote execution environment 100 comprises one or more client devices 104a-d (e.g., a laptop, mobile devices, desktops, or other systems, such as a virtual-reality or augmented/mixed reality headset), a web server 101, one or more third party websites 130 and 131 or data repositories from which requested content is retrieved, and one or more orchestrator servers 102. One or more of these servers can be part of each other and located physically remotely from the client devices 104a-d or can be located on the same physical machine as the client endpoint, or in any other architectural layout amenable to isolation (e.g., via sandboxing, containerization, or virtualization”).
Regarding claim 8, depending on claim 1,
The combination of Sundberg and Lei teaches:
The method of claim 1,
wherein the steps further comprise simulating a GPU device in the server for executing code thereon.
Although Sundberg teaches components and methods that functionally simulates a GPU rendering pipeline so it can render draw commands to a client device using for example, Chromium and SKIA as its rendering library (Sundberg: ¶19, 47, 59, Fig. 5), Sundberg’s ARAIS allows the use of massive and expensive amount of hardware such as GPU farms, this means that Sundberg’s ARAIS can perform a less expensive server execution instead (Sundberg Abstract). Therefore, the ARAIS system can perform what a hardware GPU can perform.
However Sundberg fails to disclose wherein the steps further comprise simulating a GPU device in the server for executing code thereon.
The analogous art Lei teaches:
simulating a GPU device (GPU) (Lei: par. 9, “. . . providing an efficient simulated GPU for a virtual machine . . .”; Lei: par. 43, “. . . a GPU simulation structure and a data flow for simulating a client GPU . . .”)
NOTE: A virtual machine a sandboxed execution environment. Sundberg’s remote browser is executed in a sandboxed execution environment.
It would have been an obvious design choice to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention between using a simulated GPU or not using a simulated GPU. Sundberg already teaches an ARAIS system that performs a rendering pipeline at the server instead of using massive amounts of hardware GPU . It would also have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Sundberg and incorporate Lei’s simulated GPU and include: wherein the steps further comprise simulating a GPU device in the server for executing code thereon.
The reason for doing so is to provide an efficient simulated GPU for a virtual machine and improve simulation speed, the GPU simulation speed is improved, and the implementation difficulty and complexity of the method are effectively controlled. (Lei: ¶9).
Regarding claim 9, depending on claim 1,
The combination of Sundberg and Li teaches:
The method of claim 1,
Sundberg further teaches:
wherein no rendering is performed on the server (Sundberg: par 19, “. . . the ARAIS solution allows the use of less expensive execution (e.g., server) machines that do not require massive and expensive amounts of hardware (such as GPU farms) for rasterization and video encoding . . .”; Sundberg: par. 57, “the client web browser application is configured to rasterize visual output of the remote web browser application based on draw commands received from the remote web browser application without bitmap rasterizations of the draw commands. . .”; Sundberg par 86, “. . . draw commands for the application's highest-level library can be intercepted when executing the remote application instance, and the remote application can be modified to prevent rasterization of pixels to reduce expensive computations while still obtaining position information for the intercepted draw commands . . .”; NOTE: The rendering is the rasterization. Sundberg ARAIS do not require GPU farms for rasterization and video encoding, so rasterization is not a priority for the ARAIS. Paragraph 57 further disclose that the client is the one configured to rasterize visual output, the remote web browser which is at the server is only sending draw commands to the client, without bitmap rasterization. Because the remote web browser only sends draw commands without bitmap rasterization at the server, therefore, no rendering is performed on the server.
Regarding claim 10, depending on claim 1, Sundberg further teaches:
The method of claim 1, wherein the commands are sent over the network to the user device utilizing Transmission Control Protocol (TCP) (Sundberg: ¶39, “. . .The connection support logic may be implemented for any type of secure connection over a network/Internet, including for example . . . which uses TCP and provides client/server style connectivity. . .”; Sundberg: par. 112, “A variety of distributed computing techniques are appropriate for implementing the components of the illustrated embodiments in a distributed manner including but not limited to TCP/IP sockets, RPC, RMI, HTTP, Web Services (XML-RPC, JAX-RPC, SOAP, etc.), WebRTC, WebSockets, and the like. Other variations are possible. Also, other functionality could be provided by each component/module, or existing functionality could be distributed amongst the components/modules in different ways, yet still achieve the functions of an orchestrator server").
Regarding claims 11, 12, 14, 15 and 17-20, Sundberg teaches:
A non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to perform steps of (Sundberg: par. 113, “. . . (e.g., as executable or other machine-readable software instructions or structured data) on a computer-readable medium . . . The contents (e.g., machine-readable software instructions) may be provided to one or more processors to enable the orchestrator server to perform at least some of the described techniques. Some or all of the components and/or data structures may be stored on tangible, non-transitory storage mediums including non-transitory computer-readable mediums”)
Claims 11, 12, 14, 15, 17-20 are drawn to a non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to execute the method of claims 1, 2, 4, 5, 7-10 respectively. Therefore, claims 11, 12, 14, 15, 17-20 correspond to method claims 1, 2, 4, 5, 7-10 respectively, and are rejected for the same reasons of obviousness as used above.
Claims 3, and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Sundberg in view of Li and Swift et al US 2008/0303835.
Regarding claim 3, depending on claim 1,
The combination of Sundberg and Li teaches:
The method of claim 1,
wherein the encoding uses a minimal binary protocol and compression algorithm specifically configured for encoding GPU command sequences generated by the simulated GPU (Sundberg: par. 69, “In some example remote browsers 601 . . . to encode, optimize, and/or compress data. For example, one remote browser 601 uses Google Protocol Buffers to apply compression to the draw commands and uses gzip or other similar compression technique to minimize the size of the data being transmitted. Each draw command is embedded in a message that contains the minimal amount of data needed to replicate that draw command on the local browser 610. . .”; NOTE: Sundberg uses Google protocol buffer as the binary protocol. Since Sundberg’s encoding results in an optimized with minimal amount of data needed to replicate the draw command on the client’s device, therefore, Sundberg’s encoding uses a minimal binary protocol. The compression algorithm is Gzip. Since Sundberg’s encoding includes to encode, optimize, and compress the draw commands, which are the GPU commands, therefore they are specifically configured for encoding GPU command sequences as taught by Sundberg, generated by the simulated GPU as taught by Li.),
and the decoding at the user device is performed by a GPU command decoder configured to map cached textures, fonts, and other graphical resources identified by resource identifiers (Sundberg: par. 69, “. . . images and fonts are only transferred once and stored in a cache identified by an identifier on both the remote and local computing systems. Thereafter, use of the font/image resource will be transmitted using only the identifier, and then recalled from the local browser's cache later. . .”; Sundberg: par. 70, “The corresponding protocol 612 on the local browser side 610 . . . ways to decode, decompress, and/or recall data such as in blocks 613a-c. Specifically, a cache (for example, a resource cache or a local cached library) on the local browser 610 (not shown) stores resources, such as images, textures, fonts, shapes, curves, draw commands, predefined combinations thereof, or others, that are reusable. Once the local browser 610 receives an identifier in one of the decode and decompressed (and depacketized) messages, it can retrieve the corresponding cached item from its cache”); NOTE: The mapping is identifying the corresponding cached item using an identifier. The GPU command decoder is the protocol 612. The local browser is at the user device as illustrated in Fig. 6 reference 610, local client browser.)
and to synchronize command buffers at the user device for correct execution order before rendering (Sundberg: par. 25, “. . . the secure connection 120 is implemented using TCP/IP connections (which uses sequenced packets). . .”; Sundberg: par. 69, “. . . Each draw command is embedded in a message . . . to replicate that draw command on the local browser 610. . .”; par. 32, “. . . input received at the web application 105, such as keystrokes, mouse, and other cursor and input events, . . . performs whatever execution is being requested, . . . to generate rendering output. The rendering output is then packaged, including optional encoding, optimization, and enhancing, and forwarded via the secure connection 120 back to the web application 105 to be rendered on a display of the client device 104a . . .”
NOTE: Fig. 6 reference 615 illustrates the draw commands stored in a storage or buffer symbolized by a cylinder, this is the command buffer which is before the rendering in WebGL 617 (615 draw command buffer >> 616 Execute draw commands >> 617 render at the client device).
NOTE: Fig 6. Remote Browser 601 encodes and embeds the draw commands in a message with ordered or sequenced packets >> streamed via secure network to the client >> client browser decodes the message with the sequenced packets >> buffer draw commands at 615 >> execute draw commands render at 617. Because the packets are sequenced, they are received at the client and cached in 615 in a sequenced order including keystrokes and mouse events, this inherently ensures correct execution order synchronized with the draw commands generated by the remote browser 601 in order to so it can be replicate the draw commands on the local browser. If the execution is not in synchronized order, then the keyboard or mouse events will not move or display as intended.)
Although Sundberg teaches encoding and compressing encoding GPU commands using a minimal binary protocol. Sundberg fails to disclose that: the minimal binary protocol encoding GPU command opcodes, parameter types, and resource handles defined by the graphics API of the simulated GPU.
Li further teaches:
the minimal binary protocol (Li: ¶15, “. . . the VGPU can emulate a target graphics subsystem at the binary level to provide for portability and compatibility testing with real time performance. . .”; ¶19, “. . . when the application 105 executes on the CPU 110, it generates low level graphics commands and transmits them in one or more packed command buffers 130 to the GPU. . .”; NOTE: Li executes an API layer as discussed in the rejection of claim 1, so that the VGPU process the GPU commands and encodes it at a binary level for transmission. The process of packing the low level graphics command is the encoding to a binary level. Li’s encoding is also a minimal binary protocol as its goal is to provide portability.)
encoding GPU command opcodes, parameter types, and resource handles defined by the graphics API of the simulated GPU (Li: ¶70, “In an issue block 804, the guest CPU issues the following commands (e.g., low level graphics commands) to a target GPU: Use Texture T1, Use Shader S1, Draw Object O1, Use Texture T2, Use Shader S1, and Draw Object O2. In a reception block 808, the VGPU front-end command processor receives the commands as issued by the guest CPU. In a parse block 812, the command processor of the front-end of the VGPU parses the received commands”; NOTE: low level graphics command that are subsequently encoded and packed by the VGPU includes for example, “Use Texture T1”, the opcode is “Use”, the parameter type is “Texture”, and the resource handle or identifier is T1. They are defined by the graphics API layer because API calls are executed at runtime as discussed in the rejection of claim 1.)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Sundberg, and Li to include: the minimal binary protocol encoding GPU command opcodes, parameter types, and resource handles defined by the graphics API of the simulated GPU.
The reason for doing so is “to provide for portability and compatibility testing with real time performance” (Li: ¶15).
And although Sundberg teaches decoding GPU commands and synchronizing command buffers at the user device for correct execution order before rendering, Sundberg fails to disclose decoding: by resolving inter-command dependencies among the decoded GPU command streams.
The analogous art Swift teaches:
resolving inter-command dependencies among the decoded GPU command streams (Swift: Fig.3, ¶44, “. . . command fetching module 309 in the graphics driver module 117 may retrieve graphics commands from the command stream buffer. . . retrieve the next graphics command from the command stream buffer according to a status of the associated graphics processing unit 119 and/or dependency conditions identified from previously received graphics commands. . . The command fetching module 309 may proceed fetching the next graphics command as instructed by a dependency checking module 307“; ¶45, “A dependency checking module 307 may extract header data from a retrieved graphics command to determine if an associated graphics resource has been allocated in a memory. . . compare the graphics instructions carried by the retrieved graphics command with graphics instructions from previously retrieved graphics commands to determine if a dependency condition exists for the current graphics command.; NOTE: The decoded GPU command streams are the commands illustrated in Fig. 3 reference 315, 317, and 319. The dependency checking module resolves the inter-command dependencies (dependency conditions) and instruct the fetching module 309 so it can proceed to fetch the next graphics command.)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Sundberg, Li, and Swift and apply to the decoder of Sundberg: resolving inter-command dependencies among the decoded GPU command streams, such that: the decoding at the user device is performed by a GPU command decoder configured to map cached textures, fonts, and other graphical resources identified by resource identifiers and to synchronize command buffers at the user device by resolving inter-command dependencies among the decoded GPU command streams for correct execution order before rendering.
The reason for doing so is because “multiple media processors and a display device coupled with a common memory system may require synchronization. For example, more than one component of a graphics content may arrive asynchronously from separate media processors to be displayed in a display device for a single application. Executing a graphics command to display the graphics content may depend on when each component is properly rendered and ready to display. Apparently, it is necessary to ensure maximum parallelism among multiple media processors to allow rendering different components for the same graphics content in a synchronous manner” (Swift: ¶4)
Regarding claim 13,
Claim 13 is drawn to a non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to execute the method of claim 3. Therefore, claim 13 corresponds to method of claim 3 and is rejected for the same reasons of obviousness as used above.
Allowable Subject Matter
Claims 6, and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PATRICK GALERA whose telephone number is (571)272-5070. The examiner can normally be reached Mon-Fri 0800-1700 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PATRICK P GALERA/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights