DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In view of the appeal filed on 3/25/2026, PROSECUTION IS HEREBY REOPENED. New grounds of rejection is set forth below.
To avoid abandonment of the application, appellant must exercise one of the following two options:
(1) file a reply under 37 CFR 1.111 (if this Office action is non-final) or a reply under 37 CFR 1.113 (if this Office action is final); or,
(2) initiate a new appeal by filing a notice of appeal under 37 CFR 41.31 followed by an appeal brief under 37 CFR 41.37. The previously paid notice of appeal fee and appeal brief fee can be applied to the new appeal. If, however, the appeal fees set forth in 37 CFR 41.20 have been increased since they were previously paid, then appellant must pay the difference between the increased fees and the amount previously paid.
A Supervisory Patent Examiner (SPE) has approved of reopening prosecution by signing below:
/BRADLEY A TEETS/Supervisory Patent Examiner, Art Unit 2197
Response to Amendment
3. Applicant's arguments with respect to claims 1-20 have been considered but are moot in view of the new ground(s) of rejection.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-7, 10-17 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Brockmann et al. (Pub 20200128282) (hereafter Brockmann) in view of Devereaux et al. (Pub 20140279581) (hereafter Devereaux).
As per claim 1, Brockmann teaches:
A method comprising:
receiving, by a host virtual machine (VM) executing on a computing system, a notification from at least one guest VM that a rendered frame from one or more applications executing on the at least one guest VM is available for display; ([Paragraph 10], One form of virtualization, sometimes referred to as “remote” virtualization, involves the interaction of two computing environments; one of which is a host server environment in which resides the actual computing resources to be virtualized, and the other in a remote guest or client device environment from which these resources are exploited. In this context, one can say that the “virtual” resources reside on the client device, while the actual resources reside in the host environment (although the host might also include additional virtual resources, such as virtual machines). [Paragraph 11], Commercially available remote desktop virtualization software currently enables users to remotely control applications running, by way of example, on their desktop computers at home or office from a client device, such as a smartphone or laptop, or from a remote location such as a hotel room. Such software enables such remote users to input events like keystrokes or screen touch events into a smartphone and see the results of such interactions on that device, while the applications being controlled are in fact executing on the host computer at the home or office. Thus, remote virtualization enables users to access, from a remote location, computing resources that, for a variety of reasons, are not otherwise available at the remote location. However, these solutions either require a high bandwidth and low latency connection between the server and client (such as is available on a Local Area Network or LAN), or do not support the more complex and immersive interactivity of modern media applications. [Paragraph 84], In some embodiments, the applications include user interface elements rendered via a graphics API (e.g., OpenGL) with full-screen video and/or partial-screen video (e.g., managed via a video playback API such as OpenMAX). The applications are meant to be ported, installed and run locally on the client device. Instead, in some embodiments, methods are provided for running the application as, or similar to, unmodified Virtual Client Virtual Machines (VCVM) running on application servers in a different domain than the client's or central facility's domain. By virtualizing the used APIs, such as OpenGL and OpenMAX, application functionality can be separated from the rendering functionality. [Paragraph 108], As shown in FIG. 7, multiple third-party applications 101a, 101b and/or 101c may be active concurrently (e.g., have open sessions with) on a client device 130 from application servers that are not necessarily synchronized with each other (e.g., are associated with distinct providers), yet are presented to the user as a single, seamlessly integrated application. In some embodiments, the orchestrator 110 determines, based on control 112 from the client device, control instructions 111a-111c to communicate to VCVMs 103a-103c, respectively. For example, if a user of client device 130 requests first content (e.g., or inputs a playback control for the first content) from a first provider associated with third-party application 101a, the orchestrator 110 sends control instruction 111a to VCVM 103a. Orchestrator 110 also communicates Smart Graphics Media Proxy (SGMP) information to the VCVM such that the VCVMs 103a-103c are updated regarding the graphics state of client device 130. For example, a program guide application generates information for displaying information (e.g., a small overlaid notification or icon) indicating that a sports broadcast (e.g., from an application that is distinct from the program guide application) is about to begin.)
responsive to receiving the notification, retrieving the rendered frame from the at least one guest VM; and
generating, by the host VM, a composited frame for display based at least in part on the rendered frame retrieved from the at least one guest VM. ([Paragraph 97], In some embodiments, the orchestrator 110 invokes an instance of VCVM 103 (e.g., in response to an input on a client device (e.g., thin client 130)). [Paragraph 103], FIGS. 4, 5 and 6 depict embodiments similar to the embodiments represented in, respectively, FIGS. 1, 2 and 3, modified for support of client devices (e.g., Legacy Client 430) that do not have a GPU. In some embodiments, to support client devices that do not have a GPU, the orchestrator 110 includes virtual GPU 403, as shown in FIGS. 4-6. [Paragraph 142], instance of a virtual-next-generation-set-top 931 local to the orchestrator that substitutes the functionality normally available on the next generation client and provide a composite output of the one or more applications and/or video from their respective CDNs 105 to client 430, by means of a VGPU 932 and optionally a stitcher instance 933, to produce a single stream output through network interface 934 that can be displayed by a legacy set-top 430. [Paragraph 107], FIG. 7 is a top-level diagram depicting a session with multiple concurrent applications connected to a single client under the control of the orchestrator, in accordance with some embodiments. In some embodiments, as shown in FIG. 7, the orchestrator 110 invokes a plurality of instances of distinct VCVMs (e.g., including VCVM 103a, VCVM 103b and VCVM 103c), each VCVM associated with a respective third-party CDN (e.g., including CDN 105), and provides a client device (e.g., thin client 130) with instructions to integrate (e.g., display concurrently) media content (e.g., audio/video and image assets) from the plurality of CDNs. In this manner, the client device is enabled to access media content from a plurality of CDNs without running a plurality of distinct applications (each application associated with a distinct CDN of the plurality of CDNs) at the client device. Each instance of the VCVM 103a-103c has a respective SGMP 102a-102c and a respective third party application 101a-101c executing within the VCVM. [Paragraph 133], FIG. 9 illustrates an embodiment in which orchestrator 110 is augmented by a legacy-set-top-support-stack 930 that hosts one or more virtual-next-generation-set-top (VNGST) 931 instances (e.g., instances 931a, 931b, and 931c) for each legacy client session. [Paragraph 174], In some embodiments, the digest segment is transmitted separately from the indication of the graphic state of the client device. [Paragraph 46], In some embodiments, the indication of the graphics state of the client device includes an indication of a state of media content, displayed by the client device, that corresponds to the respective applications of the multiple applications. [Paragraph 50], The digest segment corresponds to a first media stream segment received by the client device… [Paragraph 84], In some embodiments, the applications include user interface elements rendered via a graphics API (e.g., OpenGL) with full-screen video and/or partial-screen video (e.g., managed via a video playback API such as OpenMAX).)
Although Brockmann discloses of receiving a notification (graphics state) and rendered frames are transmitted/received separately. ([Paragraph 174], In some embodiments, the digest segment is transmitted separately from the indication of the graphic state of the client device. [Paragraph 46], In some embodiments, the indication of the graphics state of the client device includes an indication of a state of media content, displayed by the client device, that corresponds to the respective applications of the multiple applications. [Paragraph 50], The digest segment corresponds to a first media stream segment received by the client device…)
Brockmann does not explicitly disclose the exact sequence of receiving a notification from at least one guest VM that a rendered frame from one or more applications executing on the at least one guest VM is available for display; and
responsive to receiving the notification, retrieving the rendered frame from the at least one guest VM.
Devereaux teaches the exact sequence of receiving a notification from at least one guest VM that a rendered frame from one or more applications executing on the at least one guest VM is available for display; and
responsive to receiving the notification, retrieving the rendered frame from the at least one guest VM. ([Paragraph 86], Operations also include receiving the frame as rendered from the files provided to the remote server (408). In some implementations, when the rendering involves multiple frames, the frames can be received sequentially as and when they are produced. Alternatively, multiple frames can be received together. Receiving the frame can include receiving a notification that a rendered frame is available for downloading. A particular remote location can then be accessed to download the rendered frame. [Paragraph 5], The method also includes causing the applications to run on the virtual machines, and causing each of the jobs to be run on one of the virtual machines. [Paragraph 7], In another aspect, a method includes making available to a customer an application or a virtual machine for rendering frames of jobs and permitting the customer to execute two or more instances of the application or virtual machine, respectively. )
It would have been obvious to a person with ordinary skills in the art, before the effective filing date of the invention, to combine the teachings of Brockmann wherein a notification of state of rendered frame(s) is/are provided by guest virtual machines are received and separate transmissions of rendered frames are composited and displayed by a host VM, into teachings of Devereaux wherein the notification includes information that the rendered frame(s) is/are available/ready for retrieval/download, because this would enhance the teachings of Brockmann wherein by providing the information of availability of rendered frame(s) with the state information allows multiple rendered frames provided by multiple virtual machines to be received and composited as it becomes available, sequentially and/or together. [Devereaux paragraph 86]
As per claim 2, rejection of claim 1 is incorporated:
Brockmann teaches wherein retrieving the rendered frame from the at least one guest VM includes retrieving the rendered frame by accessing a frame memory associated with a virtual function of the at least one guest VM. ([Paragraph 178], Memory 2306 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. [Paragraph 11], Thus, remote virtualization enables users to access, from a remote location, computing resources that, for a variety of reasons, are not otherwise available at the remote location. [Paragraph 89], The resulting user experience is essentially equivalent to running these applications on the local client device, even when these devices require access to remote server resources such as various graphics rendering and other resources. [Paragraph 95], In some embodiments, third party-application 101 is configured to generate a media stream by combining user interface graphics (e.g. rendered via a Graphics API 801 (shown in FIG. 8)) with media assets (e.g., managed via a media playback API 810 (FIG. 8)). In some embodiments, the application 101 executes within a virtual-client-virtual-machine 103 (VCVM) on an application server 2300 (FIG. 23) of the third-party data center. In some embodiments, VCVM 103 includes a smart-graphics-&-media-proxy 102 (SGMP), described in detail below with regard to FIG. 8. In some embodiments, media assets 140 (e.g., images, audio, and/or video assets) are provided (e.g., via connection 107) to VCVM 103 by a media source (e.g., media stored on one or more servers that are separate from one or more servers on which application 101 executes) such as a third-party CDN server 105. Because third-party application 101 is configured to generate a media stream that includes user interface graphics and media content (e.g., video) received from a content source, the system shown in FIG. 1 does not require an intermediary stitcher (e.g., such as a cable television system headend in domain 2) to mix graphics from an application 101 with video.)
Devereaux also teaches ([Paragraph 86], Operations also include receiving the frame as rendered from the files provided to the remote server (408). In some implementations, when the rendering involves multiple frames, the frames can be received sequentially as and when they are produced. Alternatively, multiple frames can be received together. Receiving the frame can include receiving a notification that a rendered frame is available for downloading. A particular remote location can then be accessed to download the rendered frame. [Paragraph 5], The method also includes causing the applications to run on the virtual machines, and causing each of the jobs to be run on one of the virtual machines. [Paragraph 7], In another aspect, a method includes making available to a customer an application or a virtual machine for rendering frames of jobs and permitting the customer to execute two or more instances of the application or virtual machine, respectively. )
As per claim 3, rejection of claim 1 is incorporated:
Brockmann teaches comprising retrieving multiple rendered frames from multiple guest VMs that include the at least one guest VM, wherein generating the composited frame for display by the host VM comprises compositing at least a portion of two or more rendered frames of the multiple rendered frames for simultaneous display. ([Paragraph 142], instance of a virtual-next-generation-set-top 931 local to the orchestrator that substitutes the functionality normally available on the next generation client and provide a composite output of the one or more applications and/or video from their respective CDNs 105 to client 430, by means of a VGPU 932 and optionally a stitcher instance 933, to produce a single stream output through network interface 934 that can be displayed by a legacy set-top 430. [Paragraph 87], In some embodiments, multiple applications from multiple services are combined by the system to be active concurrently for a single user and presented to the user as a single, seamlessly integrated application. For example, while a user is watching a show in a VOD application, a sports match (e.g., in which a user has indicated an interest) begins. A Program Guide application that is provided by an application that is distinct from the VOD application (and possibly running on another server which might not be related to VOD application), temporarily displays, over the VOD application, an indication (e.g., a small overlaid notification) that the sports broadcast of interest is about to begin. [Paragraph 133], The VNGST has a similar composition to next-generation-set-top 130 as depicted in FIG. 19 (e.g., virtual-GPU 932 (VGPU) of VNGST 931 corresponds to the GPU, frame-buffer and display control component 1940 of next-generation-set-top 130; and VGPU 932 outputs an audio/video stream whereas next-generation-set-top 130 outputs video 1972 and audio 1973).)
Devereaux also teaches ([Paragraph 86], Operations also include receiving the frame as rendered from the files provided to the remote server (408). In some implementations, when the rendering involves multiple frames, the frames can be received sequentially as and when they are produced. Alternatively, multiple frames can be received together. Receiving the frame can include receiving a notification that a rendered frame is available for downloading. A particular remote location can then be accessed to download the rendered frame. [Paragraph 5], The method also includes causing the applications to run on the virtual machines, and causing each of the jobs to be run on one of the virtual machines. [Paragraph 7], In another aspect, a method includes making available to a customer an application or a virtual machine for rendering frames of jobs and permitting the customer to execute two or more instances of the application or virtual machine, respectively. )
As per claim 4, rejection of claim 2 is incorporated:
Brockmann teaches comprising retrieving multiple rendered frames from multiple guest VMs, wherein generating the composited frame for display by the host VM comprises selecting one rendered frame of the multiple rendered frames for display. ([Paragraph 84], In some embodiments, the applications include user interface elements rendered via a graphics API (e.g., OpenGL) with full-screen video and/or partial-screen video (e.g., managed via a video playback API such as OpenMAX). The applications are meant to be ported, installed and run locally on the client device. Instead, in some embodiments, methods are provided for running the application as, or similar to, unmodified Virtual Client Virtual Machines (VCVM) running on application servers in a different domain than the client's or central facility's domain. [Paragraph 108], As shown in FIG. 7, multiple third-party applications 101a, 101b and/or 101c may be active concurrently (e.g., have open sessions with) on a client device 130 from application servers that are not necessarily synchronized with each other (e.g., are associated with distinct providers), yet are presented to the user as a single, seamlessly integrated application. In some embodiments, the orchestrator 110 determines, based on control 112 from the client device, control instructions 111a-111c to communicate to VCVMs 103a-103c, respectively. For example, if a user of client device 130 requests first content (e.g., or inputs a playback control for the first content) from a first provider associated with third-party application 101a, the orchestrator 110 sends control instruction 111a to VCVM 103a. Orchestrator 110 also communicates Smart Graphics Media Proxy (SGMP) information to the VCVM such that the VCVMs 103a-103c are updated regarding the graphics state of client device 130. For example, a program guide application generates information for displaying information (e.g., a small overlaid notification or icon) indicating that a sports broadcast (e.g., from an application that is distinct from the program guide application) is about to begin. [Paragraph 107], FIG. 7 is a top-level diagram depicting a session with multiple concurrent applications connected to a single client under the control of the orchestrator, in accordance with some embodiments. In some embodiments, as shown in FIG. 7, the orchestrator 110 invokes a plurality of instances of distinct VCVMs (e.g., including VCVM 103a, VCVM 103b and VCVM 103c), each VCVM associated with a respective third-party CDN (e.g., including CDN 105), and provides a client device (e.g., thin client 130) with instructions to integrate (e.g., display concurrently) media content (e.g., audio/video and image assets) from the plurality of CDNs. In this manner, the client device is enabled to access media content from a plurality of CDNs without running a plurality of distinct applications (each application associated with a distinct CDN of the plurality of CDNs) at the client device.)
As per claim 5, rejection of claim 1 is incorporated:
Brockmann teaches wherein the rendered frame from the at least one guest VM comprises output from multiple applications composited by the at least one guest VM. ([Paragraph 142], normally available on the next generation client and provide a composite output of the one or more applications and/or video from their respective CDNs 105 to client 430, by means of a VGPU 932 and optionally a stitcher instance 933, to produce a single stream output through network interface 934 that can be displayed by a legacy set-top 430. [Paragraph 143], The application proxy instances are managed by the STAC so that an instance of an application proxy is available for each third-party application currently active for the client's session. The application proxy receives events, such as graphics, primitives, and media events (e.g., as outlined in FIGS. 10 and 12) from the SGMP 1960 The application proxy receives image, audio and video data from CDNs 1961. Based on received data, the application proxy performs one or more operations on behalf of the SGMP. The graphics related primitives are forwarded to the local APIs, such as for example OpenGL and media events to OpenMAX. A local GPU 1940 renders the graphics primitives to a local context and under the control of the window manager 1920, the contexts are composited in the GPUs frame-buffer 1940 to be displayed via the device's video output 1971. Audio is output via an audio device 1950 to the audio output 1972.)
Devereaux also teaches ([Paragraph 86], Operations also include receiving the frame as rendered from the files provided to the remote server (408). In some implementations, when the rendering involves multiple frames, the frames can be received sequentially as and when they are produced. Alternatively, multiple frames can be received together. Receiving the frame can include receiving a notification that a rendered frame is available for downloading. A particular remote location can then be accessed to download the rendered frame. [Paragraph 5], The method also includes causing the applications to run on the virtual machines, and causing each of the jobs to be run on one of the virtual machines. [Paragraph 7], In another aspect, a method includes making available to a customer an application or a virtual machine for rendering frames of jobs and permitting the customer to execute two or more instances of the application or virtual machine, respectively. )
As per claim 6, rejection of claim 1 is incorporated:
Brockmann teaches further comprising providing the composited frame to a remote client device for local display by the remote client device. ([Paragraph 96], In some embodiments; the third party application 101 executed on VCVM 103 comprises an application associated with a media content provider. For example, a user of client device (e.g., thin client 130) may request content from a first media content provider. As shown in FIG. 1, the first media provider associated with third-party CDN 105 sends (e.g., via connection 121) media assets to the client device.)
Devereaux also teaches ([Paragraph 50], The cloud computing system that performs the rendering job is represented by the server 106. The server 106 can include (or communicate with) multiple processors that can process a rendering job under a distributed computing framework. In some implementations, hundreds or thousands of processors can be used, thereby providing subscribers of the cloud computing system with scalable, on-demand computing capabilities. The server 106 can be configured to distribute a given rendering job (as represented by, for example, the scene file 108) to multiple processors and combine the results to produce the rendered frame 110. )
As per claim 7, rejection of claim 1 is incorporated:
Brockmann teaches wherein the host VM is associated with a first set of privileges that includes accessing a physical function of the computing system for displaying the composited frame, and wherein the at least one guest VM is associated with a set of privileges that does not include accessing the physical function. ([Paragraph 97], In some embodiments, orchestrator 110, which is situated in the operator's network (domain 2), is configured to (1) manage client sessions, (2) control playback (e.g., start, end, pause, and/or resume) of media processed by application 101, (3) signal bandwidth settings to the SGMP 102, and/or (4) provide conditional access and digital rights management (DRM) related information to the SGMP 102. In some embodiments, the orchestrator 110 invokes an instance of VCVM 103 (e.g., in response to an input on a client device (e.g., thin client 130)). In some embodiments, the orchestrator 110 receives input, via connection 112, from a user of client device 130 corresponding to a control (e.g., a playback control) and forwards the control, via connection 111, to the VCVM. In some embodiments, the orchestrator processes the control and a command to the VCVM in order to control the graphics output by VCVM 103. [Paragraph 81], In accordance with some embodiments, computer systems provide an environment for third-party applications in which applications can run unmodified in a server environment in the third-party's domain (e.g., in a manner that is transparent to third-party applications that run on a client device). In some embodiments, an “orchestration” function (e.g., in the operator's domain) coordinates one or more third-party applications running in their respective server environments. In some embodiments, a client device (e.g., in the user's domain) provides a seamless experience to the user as though the third-party applications were running locally on the client device. [Paragraph 104] [Fig. 4], As illustrated in FIG. 4, in some embodiments, third-party CDN 105 sends media assets 401 to orchestrator 110 (e.g., instead of directly to the client 130 as shown in FIGS. 1-3) so that GPU 403 can render the graphics. The orchestrator 110 communicates the graphics state of the Virtual GPU to the VCVM via connection 408. In some embodiments, orchestrator 110 receive user input from the legacy client 430 via connection 402. In some embodiments, orchestrator 110 sends rendered media assets (e.g., graphics) to the legacy client 430 via connection 402.)
Devereaux also teaches ([Paragraph 5], In another aspect, a method includes accessing through a network, processing capacity of virtual machines at a remote location and receiving from users through a network, jobs for rendering frames using applications that have features suitable for rendering frames of the respective jobs. The method also includes causing the applications to run on the virtual machines, and causing each of the jobs to be run on one of the virtual machines. When the job is completed, another job for which that virtual machine is suitable, is caused to be run on that virtual machine. [Paragraph 10], The computing device includes memory and a processor, and is configured to access through a network, processing capacity of virtual machines at a remote location and receive from users through a network, jobs for rendering frames using applications that have features suitable for rendering frames of the respective jobs. The computing device is also configured to cause the applications to run on the virtual machines, and cause each of the jobs to be run on one of the virtual machines. When the job is completed, another job for which that virtual machine is suitable, is caused to be run on that virtual machine.)
As per claim 10, rejection of claim 1 is incorporated:
Brockmann teaches wherein the virtual function of the at least one guest VM is a virtual function of a virtualized physical display, and wherein retrieving the rendered frame from the at least one guest VM by accessing the associated frame memory is performed by the host VM via a physical function of the virtualized physical display. ([Paragraph 10], One form of virtualization, sometimes referred to as “remote” virtualization, involves the interaction of two computing environments; one of which is a host server environment in which resides the actual computing resources to be virtualized, and the other in a remote guest or client device environment from which these resources are exploited. In this context, one can say that the “virtual” resources reside on the client device, while the actual resources reside in the host environment (although the host might also include additional virtual resources, such as virtual machines). [Paragraph 133], The VNGST has a similar composition to next-generation-set-top 130 as depicted in FIG. 19 (e.g., virtual-GPU 932 (VGPU) of VNGST 931 corresponds to the GPU, frame-buffer and display control component 1940 of next-generation-set-top 130; and VGPU 932 outputs an audio/video stream whereas next-generation-set-top 130 outputs video 1972 and audio 1973). In some embodiments, VGPU 932 (e.g., VGPU 932a, 932b, and 932c that correspond to VNGST 931a, 931b, and 931c, respectively) implements GPU and video encoding functionality in software, hardware or any hybrid of the two.)
Devereaux also teaches ([Paragraph 5], In another aspect, a method includes accessing through a network, processing capacity of virtual machines at a remote location and receiving from users through a network, jobs for rendering frames using applications that have features suitable for rendering frames of the respective jobs. The method also includes causing the applications to run on the virtual machines, and causing each of the jobs to be run on one of the virtual machines. When the job is completed, another job for which that virtual machine is suitable, is caused to be run on that virtual machine. [Paragraph 10], The computing device includes memory and a processor, and is configured to access through a network, processing capacity of virtual machines at a remote location and receive from users through a network, jobs for rendering frames using applications that have features suitable for rendering frames of the respective jobs. The computing device is also configured to cause the applications to run on the virtual machines, and cause each of the jobs to be run on one of the virtual machines. When the job is completed, another job for which that virtual machine is suitable, is caused to be run on that virtual machine.)
As per claims 11 and 15-17, these are computing system claims corresponding to the method claims 1 and 5-7. Therefore, rejected based on similar rationale.
As per claim 12, rejection of claim 11 is incorporated:
Brockmann teaches wherein the guest VM is one of multiple guest VMs that are executing on the computing system and managed by the host VM, and wherein the host VM retrieves multiple rendered frames from the multiple guest VMs by accessing a respective frame memory associated with each of the multiple guest VMs. ([Paragraph 108], As shown in FIG. 7, multiple third-party applications 101a, 101b and/or 101c may be active concurrently (e.g., have open sessions with) on a client device 130 from application servers that are not necessarily synchronized with each other (e.g., are associated with distinct providers), yet are presented to the user as a single, seamlessly integrated application. In some embodiments, the orchestrator 110 determines, based on control 112 from the client device, control instructions 111a-111c to communicate to VCVMs 103a-103c, respectively. For example, if a user of client device 130 requests first content (e.g., or inputs a playback control for the first content) from a first provider associated with third-party application 101a, the orchestrator 110 sends control instruction 111a to VCVM 103a. Orchestrator 110 also communicates Smart Graphics Media Proxy (SGMP) information to the VCVM such that the VCVMs 103a-103c are updated regarding the graphics state of client device 130. For example, a program guide application generates information for displaying information (e.g., a small overlaid notification or icon) indicating that a sports broadcast (e.g., from an application that is distinct from the program guide application) is about to begin. [Paragraph 107], FIG. 7 is a top-level diagram depicting a session with multiple concurrent applications connected to a single client under the control of the orchestrator, in accordance with some embodiments. In some embodiments, as shown in FIG. 7, the orchestrator 110 invokes a plurality of instances of distinct VCVMs (e.g., including VCVM 103a, VCVM 103b and VCVM 103c), each VCVM associated with a respective third-party CDN (e.g., including CDN 105), and provides a client device (e.g., thin client 130) with instructions to integrate (e.g., display concurrently) media content (e.g., audio/video and image assets) from the plurality of CDNs. In this manner, the client device is enabled to access media content from a plurality of CDNs without running a plurality of distinct applications (each application associated with a distinct CDN of the plurality of CDNs) at the client device. Each instance of the VCVM 103a-103c has a respective SGMP 102a-102c and a respective third party application 101a-101c executing within the VCVM. [Paragraph 146], The media stream segment has a first data size. In some embodiments, the segment header includes data for the plurality of frames. For example, the segment header may include byte information (e.g., a byte range) that specifies (e.g., identifies) the frames in the segment. [Paragraph 150], In some embodiments, the digest segment does not include (2126) image content for at least one respective frame of the plurality of frames. For example, the digest segment includes the representation of the video data (e.g., the segment header), but does not include at least a portion of the video data that corresponds to the plurality of frames. In some embodiments, the representation of the video data of the plurality of frames includes (2128) an indication of a location and/or a data length of at least one frame of the plurality of frames of the media stream. For example, the representation of the video data includes information to indicate a current playback position of the media stream. For example, the representation includes a range, such as an indication of a start location and an indication of an end location of one or more frames of the plurality of frames (e.g., an identifier of a first byte and/or a last byte that correspond to an individual frame). In some embodiments, the representation of the video data includes (2130) a frame identifier for at least one respective frame of the plurality of frames. )
Devereaux also teaches ([Paragraph 5], In another aspect, a method includes accessing through a network, processing capacity of virtual machines at a remote location and receiving from users through a network, jobs for rendering frames using applications that have features suitable for rendering frames of the respective jobs. The method also includes causing the applications to run on the virtual machines, and causing each of the jobs to be run on one of the virtual machines. When the job is completed, another job for which that virtual machine is suitable, is caused to be run on that virtual machine. [Paragraph 10], The computing device includes memory and a processor, and is configured to access through a network, processing capacity of virtual machines at a remote location and receive from users through a network, jobs for rendering frames using applications that have features suitable for rendering frames of the respective jobs. The computing device is also configured to cause the applications to run on the virtual machines, and cause each of the jobs to be run on one of the virtual machines. When the job is completed, another job for which that virtual machine is suitable, is caused to be run on that virtual machine.)
As per claim 13, rejection of claim 12 is incorporated:
Brockmann teaches wherein the composited frame includes at least a portion of two or more rendered frames of the multiple rendered frames for simultaneous display. ([Paragraph 87], In some embodiments, multiple applications from multiple services are combined by the system to be active concurrently for a single user and presented to the user as a single, seamlessly integrated application. For example, while a user is watching a show in a VOD application, a sports match (e.g., in which a user has indicated an interest) begins. A Program Guide application that is provided by an application that is distinct from the VOD application (and possibly running on another server which might not be related to VOD application), temporarily displays, over the VOD application, an indication (e.g., a small overlaid notification) that the sports broadcast of interest is about to begin.)
As per claim 14, rejection of claim 12 is incorporated:
Brockmann teaches wherein to generate the composited frame for display by the host VM includes to select one rendered frame of the multiple rendered frames for display. ([Paragraph 87], In some embodiments, multiple applications from multiple services are combined by the system to be active concurrently for a single user and presented to the user as a single, seamlessly integrated application. For example, while a user is watching a show in a VOD application, a sports match (e.g., in which a user has indicated an interest) begins. A Program Guide application that is provided by an application that is distinct from the VOD application (and possibly running on another server which might not be related to VOD application), temporarily displays, over the VOD application, an indication (e.g., a small overlaid notification) that the sports broadcast of interest is about to begin. [Paragraph 84], In some embodiments, the applications include user interface elements rendered via a graphics API (e.g., OpenGL) with full-screen video and/or partial-screen video (e.g., managed via a video playback API such as OpenMAX). The applications are meant to be ported, installed and run locally on the client device. Instead, in some embodiments, methods are provided for running the application as, or similar to, unmodified Virtual Client Virtual Machines (VCVM) running on application servers in a different domain than the client's or central facility's domain. [Paragraph 108], As shown in FIG. 7, multiple third-party applications 101a, 101b and/or 101c may be active concurrently (e.g., have open sessions with) on a client device 130 from application servers that are not necessarily synchronized with each other (e.g., are associated with distinct providers), yet are presented to the user as a single, seamlessly integrated application. In some embodiments, the orchestrator 110 determines, based on control 112 from the client device, control instructions 111a-111c to communicate to VCVMs 103a-103c, respectively. For example, if a user of client device 130 requests first content (e.g., or inputs a playback control for the first content) from a first provider associated with third-party application 101a, the orchestrator 110 sends control instruction 111a to VCVM 103a. Orchestrator 110 also communicates Smart Graphics Media Proxy (SGMP) information to the VCVM such that the VCVMs 103a-103c are updated regarding the graphics state of client device 130. For example, a program guide application generates information for displaying information (e.g., a small overlaid notification or icon) indicating that a sports broadcast (e.g., from an application that is distinct from the program guide application) is about to begin. [Paragraph 107], FIG. 7 is a top-level diagram depicting a session with multiple concurrent applications connected to a single client under the control of the orchestrator, in accordance with some embodiments. In some embodiments, as shown in FIG. 7, the orchestrator 110 invokes a plurality of instances of distinct VCVMs (e.g., including VCVM 103a, VCVM 103b and VCVM 103c), each VCVM associated with a respective third-party CDN (e.g., including CDN 105), and provides a client device (e.g., thin client 130) with instructions to integrate (e.g., display concurrently) media content (e.g., audio/video and image assets) from the plurality of CDNs. In this manner, the client device is enabled to access media content from a plurality of CDNs without running a plurality of distinct applications (each application associated with a distinct CDN of the plurality of CDNs) at the client device.)
As per claim 20, this is a non-transitory computing readable medium claim corresponding to the method claim 1. Therefore, rejected based on similar rationale.
Claim(s) 8 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Brockmann in view of Devereaux and further in view of Cheng et al. (Pub 20130229421) (hereafter Cheng).
As per claim 8, rejection of claim 1 is incorporated:
Brockmann teaches wherein receiving the notification from the at least one guest VM comprises processing an interrupt caused by a write operation to one or more registers of the host VM. ([Paragraph 112], The state tracker module also maintains a model of the display state of the client. By way of example, for certain events when a call would cause changes to the client's frame buffer 1940 that would be seen by a user, a difference between the virtual state and the state of the real (remote) GPU 1940, maintained by the state tracker 802, is determined and forwarded as an update to the bandwidth manager 804. [Paragraph 114], The following lists reasons why the remote state may need to be updated: (1) the API call causes changes to the client's frame buffer 1940 that are user-perceivable, (2) the API call has related data that can be send ahead, (3) a state accumulation threshold has been met, (4) a timeout has expired.)
However, Brockmann and Devereaux do not explicitly disclose interrupt cause by a write operation.
Cheng teaches interrupt cause by a write operation. ([Paragraph 64], As previously discussed, the vertical synchronization interrupt is a synchronization mechanism that ensures that the new surface is ready to be output to a monitor. Because the virtual function GPU driver retains the existing page flipping model, the GPU driver checks at the emulated vertical synchronization interrupt whether the page flip is complete as shown in operations 708 and 709. [Paragraph 46], The underlying rendering and drawing functionality provided by GPU 520 is shared among physical function 512 and its associated virtual functions, e.g. virtual functions 513. However, each virtual function has its own independent set of resources including register address map and memory address map. In particular, virtual function 513 is configured to have a separate frame buffer segment (e.g. virtual function frame buffer 519) to store and manipulate the graphic surface associated with virtual machine 511.)
It would have been obvious to a person with ordinary skills in the art, before the effective filing date of the invention, to combine the teachings of Brockmann and Devereaux wherein a notification of availability of rendered frame(s) is/are provided by guest virtual machines are received and separate transmissions of rendered frames are composited and displayed by a host VM in response to the notification, into teachings of Cheng wherein new/updates to frame(s) written to a buffer associated with frame(s) rendered by virtual machine(s) causes a synchronization interrupt, because this would enhance the teachings of Brockmann and Devereaux wherein by causing the synchronization interrupt, it allows the rendered frames to be displayed/created/updated based on newly written frame data by the virtual machine(s) for synchronization/composition and display.
As per claim 18, this is a computing system claim corresponding to the method claim 8. Therefore, rejected based on similar rationale.
Claim(s) 9 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Brockmann in view of Devereaux, in view of Cheng and further in view of Khosa et al. (Pub 20150029200) (hereafter Khosa).
As per claim 9, rejection of claim 8 is incorporated:
Brockmann teaches displaying sequence of images and write operation to the one or more registers/buffers.
Cheng teaches each virtual machine having its corresponding registers/buffers.
wherein the write operation to the one or more registers is based at least in part on one or more modifications to a swap chain associated with the one or more applications executing on the at least one guest VM. (i.e. sequence of frames) ([Brockmann paragraph 4], application servers execute an application on the server at a remote location and send the resulting output, usually in the form of a screen image, or sequence of images, or a video stream, to the client device which the user sees and interacts with. [Paragraph 112], The state tracker module also maintains a model of the display state of the client. By way of example, for certain events when a call would cause changes to the client's frame buffer 1940 that would be seen by a user, a difference between the virtual state and the state of the real (remote) GPU 1940, maintained by the state tracker 802, is determined and forwarded as an update to the bandwidth manager 804. [Paragraph 114], The following lists reasons why the remote state may need to be updated: (1) the API call causes changes to the client's frame buffer 1940 that are user-perceivable, (2) the API call has related data that can be send ahead, (3) a state accumulation threshold has been met, (4) a timeout has expired.))
([Cheng paragraph 46], The underlying rendering and drawing functionality provided by GPU 520 is shared among physical function 512 and its associated virtual functions, e.g. virtual functions 513. However, each virtual function has its own independent set of resources including register address map and memory address map. In particular, virtual function 513 is configured to have a separate frame buffer segment (e.g. virtual function frame buffer 519) to store and manipulate the graphic surface associated with virtual machine 511.)
However, Brockmann, Devereaux and Cheng do not explicitly disclose one or more modifications to a swap chain.
Khosa teaches one or more modifications to a swap chain. ([Paragraph 4], Virtualization of a GPU device is useful to allow multiple independent environments shared access to the GPU hardware device. For example, in the context of desktop virtualization, Windows and Linux virtual machines (VMs) may be concurrently running on a computer, each with applications requiring GPU acceleration. In another example, an application running on a remote access server may wish to use the GPU of the client computer that is providing the user interface (UI). [Paragraph 35], n particular, swap chains 404 and 406 are typically used in two ways. First, the application in control of the display hardware (e.g. 322) uses the swap chains to switch from one frame to the next--without display artifacts. In a desktop environment, applications typically only have control over their own windows, not the whole display. The OS compositor typically controls the display as described above. Applications still use double buffering and buffer swapping, which works like swap-chains; however this is usually through a high level API--such as EGL (the API to link OpenGL to the native windowing system). The buffer-swap command is sent to the compositor (e.g., 422, a window manager) once an output window is fully rendered, and the compositor 422 then knows it is safe to use that buffer to render to an actual output virtual framebuffer (which may have windows from multiple applications) which will likely be the head buffer in one of the VM's swap chains. Note, in the above discussion only two levels of rendering are discussed: an application and a compositor (window manager), however the present invention is not limited to a particular number of levels. A compositor may, for example, use EGL to render to a window of yet another compositor. [Paragraph 34], The use of swap chains as shown prevents display artifacts that would be visible due to (a) rendering to the same memory that the display hardware is outputting, i.e., the display may refresh in the middle of a render, leading to flickering or artifacts on the display; (b) non-synchronized switching of the active display window, causing "tearing" in which the display exhibits an effect where part of the display shows the old frame(s) and the other part shows the new frame.)
It would have been obvious to a person with ordinary skills in the art, before the effective filing date of the invention, to combine the teachings of Brockmann, Devereaux and Cheng wherein a notification of rendered frame(s) availability is/are provided by guest virtual machines that are ready/available to be displayed, rendered frame(s) is/are retrieved and composited for display by a host VM, into teachings of Khosa wherein swap chain(s) is/are used for displaying composited display, because this would enhance the teachings of Brockmann, Devereaux and Cheng wherein by utilizing swap-chain(s)/buffer swapping, it prevents various display issues such as non-synchronized switching, tearing, flickering, etc. [Khosa paragraph 34]
As per claim 19, this is a computing system claim corresponding to the method claim 9. Therefore, rejected based on similar rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DONG U KIM whose telephone number is (571)270-1313. The examiner can normally be reached 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bradley Teets can be reached at 5712723338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DONG U KIM/Primary Examiner, Art Unit 2197
Prosecution Insights