DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-15 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Lin et al. (U.S. Pub 2016/0044139) hereinafter Lin.
As per Claim 1, Lin teaches A system for providing a remote computing resource to a user equipment, the system comprising the user equipment and at least one server, wherein: (Fig. 2A, ¶5, ¶6, ¶88 wherein transmitting hardware data from drivers on a client to a server wherein receiving a rendering command, a parameter and/or a texture from the server, and transmitting the rendering command, the parameter and/or the texture to a driver on the client to render the graphic with a GPU on the client wherien interactive region 116 may be configured as a shutter button and the picture (the hardware data) generated when interactive region 116 is acted (shutter button pressed) may be transmitted to second computing device 200/server (or the virtual machine 212 or the second app 2).)
the user equipment is configured to: receive a user input from an input device of the user equipment; and (Fig. 2A, ¶39, ¶48, ¶52 , ¶88 wherien After configuration based on hardware setting is complete (i.e. after configuring first app 1 based on the received hardware setting), when a user presses button 116 shown on UI 118, first app 1 may initiate the camera module of first computing device 100 to take a picture wherien When a user interacts with UI 118 shown on the screen of first computing device 100 (e.g., by touching the screen of the smartphone), a coordinate on screen/UI 118 representing the location where the user touched may be transmitted to daemon 26 by first app 1 wherien the user may initiate a process via first app 1 on first computing device 100 to operate second app 2 on virtual machine 212. The user may touch the screen of first computing device 100. This generates an input event wherein the hardware setting relates to configuring first app 1 to take a picture. Accordingly, interactive region 116 may be configured as a shutter button and the picture (the hardware data) generated when interactive region 116 is acted (shutter button pressed) may be transmitted to second computing device 200/server (or the virtual machine 212 or the second app 2))
send data identifying the user input to the server, (Fig. 3A, ¶39, ¶48, ¶52, ¶88 when a user presses button 116 shown on UI 118, first app 1 may initiate the camera module of first computing device 100 to take a picture. Subsequently, the picture (in a file, binary code or other data format) may be transmitted to daemon 26 wherien when a user interacts with UI 118 shown on the screen of first computing device 100 (e.g., by touching the screen of the smartphone), a coordinate on screen/UI 118 representing the location where the user touched may be transmitted to daemon 26 by first app 1 wherein the touch event may be transmitted to second computing device 200 (via or by the help of daemon 26) as an input to second app 2 wherien the picture (the hardware data) generated when interactive region 116 is acted (shutter button pressed) may be transmitted to second computing device 200/ server)
wherein the data identifying the user input is based on data from a hardware driver of the input device; and (¶50, ¶89 wherien the function or method may include receiving hardware data from a Bluetooth, a Wi-Fi, an NFC, a camera, a GPS, a gyroscope, an e-compass and an accelerometer wherien solid lines represent hardware data transfer, e.g., picture or other kinds of hardware data (may be a file, binary code, and/or value/data in JSON/XML structure which comes from hardware device 12 or first app 1)
the server is configured to: receive the data identifying the user input; (¶39, ¶52 wherien Daemon 26 may be configured to receive the picture and store it in memory 20, the touch event may be transmitted to second computing device 200 (via or by the help of daemon 26) as an input to second app 2. In another example, first app 1 may only transmit the coordinate to daemon 26. Daemon 26 may then send the coordinate to generate an input event)
cause execution of the remote computing resource based on the data identifying the user input; and (¶39, ¶49 wherien second app 2 may go to memory 20 to retrieve the picture and display it on the screen output Next, second app 2 may receive an input event and respond to such input event accordingly (e.g., receiving notice that shutter button 216 is pressed based on the coordinate and using the camera API on virtual machine 212 to take a picture). Operating second app 2 on the virtual machine/server remotely via first app 1 on first computing device 100 is therefore achieved)
send an output of the executed remote computing resource to the user equipment. (Fig. 3C, ¶54 wherien Next, second app 2 may respond to the touch event as if the user is directly touching a point corresponding to (238, 458) on UI 218. [...] Hardware-setting module 214 may generate a hardware setting (e.g., a value or a set of values) that dispatches hardware-related module 17 to use a first camera API (not shown) on first computing device 100. The hardware setting may be transmitted to first app 1 by daemon 26 via internet)
As per Claim 2, Lin further teaches wherein one or more of the at least one servers
PNG
media_image1.png
87
6
media_image1.png
Greyscale
comprises a remote hardware abstraction layer for the input device of the user equipment, (Fig. 3A-3C, ¶59 wherein HAL 208 may receive hardware data (the picture) directly from the internet (i.e., by the help of daemon 26) and pass it to second app 2 or store it in memory 20 for second app 2 to access)
preferably wherein the user equipment comprises an interface that is configured to communicate the data identifying the user input to the remote hardware abstraction layer of the server. (Fig. 3A-3C, ¶39, ¶48, ¶66 wherien When a user interacts with UI 118 shown on the screen of first computing device 100 (e.g., by touching the screen of the smartphone), a coordinate on screen/UI 118 representing the location where the user touched may be transmitted to daemon 26 by first app 1. In one example, daemon 26 may generate a corresponding input event to virtual machine 212. In another example, daemon 26 may send the coordinate it received to virtual machine 212 and a virtual IO module/interface (e.g., a vr_io_int 210c in FIG. 3A, 3B, 4A or 4B) may generate a corresponding input event to second app 2 wherien first app 1 may further include a transmitter 19. Transmitter 19 may be configured to transmit hardware data from the driver (the driver 111) to virtual machine 212.)
As per Claim 3, the rejection of claim 2 is hereby incorporated by reference; Lin further teaches wherein the remote hardware abstraction layer of the server comprises or is configured to communicate using one or a plurality of application programming interfaces, APIs configured to cause the server to receive the data identifying the user input, preferably wherein the one or a plurality of APIs are configured to cause the server to interface with one or a plurality of respective hardware drivers of the user equipment to receive the data identifying the user input. (Fig. 3A-3C, ¶50, ¶54 wherien dash lines represent API calls/function calls/system calls, commands/instructions and/or parameters/values (e.g., in JSON or XML structure), while solid lines represent hardware data transfer wherien point (238, 458) may locate in a range of shutter button 216 on UI 218, and second app 2 may use a second camera API (not shown) after receiving the touch event. Since virtual machine 212 does not have a camera module (or relative hardware device), hardware-setting module 214 may be configured to intercept the call when/before it is sent to a camera driver (i.e. a driver 211 or a pseudo camera driver in this example) on virtual machine 212. Hardware-setting module 214 may generate a hardware setting (e.g., a value or a set of values) that dispatches hardware-related module 17 (shown in FIG. 3C) to use a first camera API (not shown) on first computing device 100. The hardware setting may be transmitted to first app 1 by daemon 26 via internet (e.g., via HTTP protocol). )
As per Claim 4, the rejection of claim 1 is hereby incorporated by reference; Lin further teaches wherein the remote computing resource comprises: a cloud-based operating system, OS, configured to be executed based on the data identifying the user input; and/or a remote application configured to be executed based on the data identifying the user input. (Fig. 2A-2B, ¶36 wherien first app 1 may be executed on first computing device 100 and second app 2 may be executed on virtual machine 212 that runs on second computing device 200. A user may use first app 1 to receive screen output (or UI 218) of second app 2 and display it as a screen output (or UI 118) of first app 1. Via the screen output/UI 218 of second app 2 shown on the screen output/UI 118 of first app 1, a user may further operate second app 2 by sending the coordinate of a point on UI 118 that was touched by user to a daemon 26, a software or a software server that runs on virtual machine 212 or couples with virtual machine 212 (please refer to FIG. 2C, 2D, 3A, 3B, 4A or 4B). If daemon 26 does not run on virtual machine 212 but instead couples with virtual machine 212, daemon 26 may be executed on second computing device 200 or on any computing device that couples with virtual machine 212 in the network. In another example, a remote desktop protocol may also be applied to generate a touch event to second app 2 when the user touches a point on UI 118.)
As per Claim 5, the rejection of claim 1 is hereby incorporated by reference; Lin further teaches wherein the user equipment comprises a local hardware abstraction layer for interfacing with one or a plurality of respective hardware drivers of the user equipment. (Fig. 3A, ¶56 wherien ) first app 1 may use a camera (hardware device 12) via the first camera API (e.g., initiating a first camera driver, that is, a driver 111 on first computing device 100) to take a picture or an image in binary format (i.e., the hardware data e.g., binary code, file value or data). The picture may be sent to memory 10 or other storage (not shown), and first app 1 may access memory 10 or the storage to get the picture. Next, first app 1 may transmit the hardware data (i.e. the picture/image) to daemon 26, and the hardware data may be kept in memory 20. Moreover, first app 1 may also notify virtual machine 212 (thru daemon 26) that the picture is successfully taken by hardware device 12)
As per Claim 6, the rejection of claim 1 is hereby incorporated by reference; Lin further teaches wherein the user equipment is configured to, in response to identifying that a connection between the user equipment does not satisfy one or more connection quality criteria, execute a local computing resource of the user equipment based on the data identifying the user input and using the local hardware abstraction layer, preferably wherein the local computing resource comprises a local operating system and/or a local application. (¶68, ¶99 wherien a buffering module (not shown) coupled with transmitter 19 may be configured to buffer hardware data in the storage (not shown, e.g., an SD card) or memory (e.g., memory 10 or other memory) of the client (first computing device 100) when there is no network service. In this example, transmitter 19 may be configured to transmit hardware data to daemon 26 (or virtual machine 212) after network resumes)
As per Claim 7, the rejection of claim 6 is hereby incorporated by reference; Lin further teaches wherein the one or more connection quality criteria comprise any one or more of: a measure of latency for the user equipment; a measure of latency for the user equipment and/or the server; a measure of latency between the user equipment and the server; a measure of signal strength at the user equipment; a measure of a download and/or upload bandwidth of the user equipment; a measure of a download and/or upload bandwidth of the server; a measure of a packet loss rate; an indication of whether the user equipment and/or the server is connected to a network; and/or a measure of jitter. (Fig. 2A-2D, Fig. 4A-4B, ¶41, ¶72 wherien the 3D graphics rendered by GPU 22 may be kept in memory 20 (wherein a part of memory 20 may be a framebuffer in this example), and the 3D graphics in the framebuffer may subsequently be accessed by daemon 26 and transmitted to first app 1 and displayed as app stream on UI 118 or the screen output of first app 1, as shown in FIG. 2C wherien a system for communications between first app 1 and the server or virtual machine 212 using hardware device on the client (i.e., hardware device 12 or GPU 12′) to render graphics of second app 2 on the server or virtual machine 212.)
As per Claim 8, the rejection of claim 6 is hereby incorporated by reference; Lin further teaches wherein the local computing resource has: lower computing resource requirements than the remote computing resource; or equal computing resource requirements to the remote computing resource. (Fig. 1A-1B, ¶34 wherien libraries or APIs related to hardware data are not coupled to real hardware (e.g., a GPU, a camera, sensors or chip, etc.) supported by mobile OS. Although mobile OS running on virtual machine can support various hardware resources (e.g., it may have APIs or drivers corresponding to the hardware), it cannot gather/receive hardware data, such as coordinates, tilts, rotation, shakings, proximity, lamination, etc., because it is not running on a device with real, physical hardware. In another example, although the PC/server on which the virtual machine is run may have hardware such as a GPU, the graphics API or driver in the mobile OS may still not be compatible with the hardware of the PC/server outside the virtual machine (e.g., GPUs for smartphones and servers may differ in both specifications, drivers, supported libraries and/or interfaces). The hardware resources on a PC or a server where the virtual machine is run may be different from those in a mobile device where mobile OS is designed to be run)
As per Claim 9, the rejection of claim 1 is hereby incorporated by reference; Lin further teaches further configured to send buffered content to the user equipment, preferably wherein the buffered content comprises any one or more of: screen animations; audio content; video content; image content, vibration, and/or haptic feedback. (Fig. 2A-2C, Fig. 3A-3B, Fig. 4A-4B, ¶41, ¶48 wherien the 3D graphics rendered by GPU 22 may be kept in memory 20 (wherein a part of memory 20 may be a framebuffer in this example), and the 3D graphics in the framebuffer may subsequently be accessed by daemon 26 and transmitted to first app 1 and displayed as app stream on UI 118 or the screen output of first app 1, as shown in FIG. 2C. wherein In one example, daemon 26 may generate a corresponding input event to virtual machine 212. In another example, daemon 26 may send the coordinate it received to virtual machine 212 and a virtual IO module/interface (e.g., a vr_io_int 210c in FIG. 3A, 3B, 4A or 4B) may generate a corresponding input event to second app 2. In one example, vr_io_int 210c may be configured to transmit/receive/buffer general input/output to/from the second app 2. Moreover, the general input/output may include an image, an audio or other hardware data.)
As per Claim 10, the rejection of claim 1 is hereby incorporated by reference; Lin further teaches wherein: the output of the executed remote computing resource comprises any one or more of: an image; an audio stream; a video stream; an instruction to provide haptic feedback; and/or an instruction to operate a sensor of the user equipment. (Fig. 3C, ¶55 wherien a pair of coordinates transmitted with hardware setting can be used to configure a rectangular area (i.e., shutter button 116) on screen output/UI 118 of first app 1 (i.e., the pair of the coordinates mean, e.g., a left-top and right-down point of the area to be configured) wherein First app 1 may be configured to couple the area with corresponding API (or hardware device), or with a camera (i.e., hardware device 12) of first computing device 100, based on the received hardware setting. )
As per Claim 11, the rejection of claim 1 is hereby incorporated by reference; Lin further teaches wherein the user input comprises any one or more of: a touch input; an audio input; a camera input; a physical button input; a fingerprint reader input; and/or a proximity sensor input. (¶54 wherien second app 2 may respond to the touch event as if the user is directly touching a point corresponding to (238, 458) on UI 218. In this example, point (238, 458) may locate in a range of shutter button 216 on UI 218, and second app 2 may use a second camera API (not shown) after receiving the touch event.)
As per Claim 12, the rejection of claim 1 is hereby incorporated by reference; Lin further teaches wherein the user equipment is further configured to: receive the output of the executed remote computing resource; and (Fig. 3C, ¶54 wherien Next, second app 2 may respond to the touch event as if the user is directly touching a point corresponding to (238, 458) on UI 218. In this example, point (238, 458) may locate in a range of shutter button 216 on UI 218, and second app 2 may use a second camera API (not shown) after receiving the touch event. Since virtual machine 212 does not have a camera module (or relative hardware device), hardware-setting module 214 may be configured to intercept the call when/before it is sent to a camera driver (i.e. a driver 211 or a pseudo camera driver in this example) on virtual machine 212. Hardware-setting module 214 may generate a hardware setting (e.g., a value or a set of values) that dispatches hardware-related module 17 (shown in FIG. 3C) to use a first camera API (not shown) on first computing device 100. The hardware setting may be transmitted to first app 1 by daemon 26 via internet (e.g., via HTTP protocol).)
provide the output of the executed remote computing resource on an output device of the user equipment. (Fig. 3C, ¶55 wherien a pair of coordinates transmitted with hardware setting can be used to configure a rectangular area (i.e., shutter button 116) on screen output/UI 118 of first app 1 (i.e., the pair of the coordinates mean, e.g., a left-top and right-down point of the area to be configured). First app 1 may be configured to couple the area with corresponding API (or hardware device), or with a camera (i.e., hardware device 12) of first computing device 100, based on the received hardware setting. In another example, the step of transmitting the pair of coordinates back to first app 1 may not be necessary since first app 1 may be configured to generate button 116 with predetermined size/area/shape (i.e., it may generate a predetermined sized/shaped button once it receives a hardware setting and a coordinate), and it may only configure the area in the predetermined size/shape at the coordinate where the user touched.)
As per Claim 13, the rejection of claim 1 is hereby incorporated by reference; Lin further teaches wherein the user equipment comprises any one or more of: a smartphone; a tablet; a personal computer; a television; a smart watch; a smart mirror; smart glasses; an augmented reality headset; a virtual reality headset; a smart windscreen; and/or a smart wearable device. (¶48, ¶53 wherien When a user interacts with UI 118 shown on the screen of first computing device 100 (e.g., by touching the screen of the smartphone), a coordinate on screen/UI 118 representing the location where the user touched may be transmitted to daemon 26 by first app 1 wherien first computing device 100 may be a tablet PC but virtual machine 212 may simulate a smartphone having different screen resolution)
As per Claim 14, Lin teaches A user equipment for interacting with a remote computing resource, configured to: receive a user input from an input device of the user equipment; (Fig. 2A, ¶39, ¶48, ¶52 , ¶88 wherien After configuration based on hardware setting is complete (i.e. after configuring first app 1 based on the received hardware setting), when a user presses button 116 shown on UI 118, first app 1 may initiate the camera module of first computing device 100 to take a picture wherien When a user interacts with UI 118 shown on the screen of first computing device 100 (e.g., by touching the screen of the smartphone), a coordinate on screen/UI 118 representing the location where the user touched may be transmitted to daemon 26 by first app 1 wherien the user may initiate a process via first app 1 on first computing device 100 to operate second app 2 on virtual machine 212. The user may touch the screen of first computing device 100. This generates an input event wherein the hardware setting relates to configuring first app 1 to take a picture. Accordingly, interactive region 116 may be configured as a shutter button and the picture (the hardware data) generated when interactive region 116 is acted (shutter button pressed) may be transmitted to second computing device 200/server (or the virtual machine 212 or the second app 2))
send data identifying the user input to a server configured to cause execution of the remote computing resource, (Fig. 3A, ¶39, ¶48, ¶52, ¶88 when a user presses button 116 shown on UI 118, first app 1 may initiate the camera module of first computing device 100 to take a picture. Subsequently, the picture (in a file, binary code or other data format) may be transmitted to daemon 26 wherien when a user interacts with UI 118 shown on the screen of first computing device 100 (e.g., by touching the screen of the smartphone), a coordinate on screen/UI 118 representing the location where the user touched may be transmitted to daemon 26 by first app 1 wherein the touch event may be transmitted to second computing device 200 (via or by the help of daemon 26) as an input to second app 2 wherien the picture (the hardware data) generated when interactive region 116 is acted (shutter button pressed) may be transmitted to second computing device 200/ server)
wherein the data identifying the user input is based on data from a hardware driver of the input device; and (¶50, ¶89 wherien the function or method may include receiving hardware data from a Bluetooth, a Wi-Fi, an NFC, a camera, a GPS, a gyroscope, an e-compass and an accelerometer wherien solid lines represent hardware data transfer, e.g., picture or other kinds of hardware data (may be a file, binary code, and/or value/data in JSON/XML structure which comes from hardware device 12 or first app 1)
receive an output of the executed remote computing resource from the server. (Fig. 3C, ¶54 wherien Next, second app 2 may respond to the touch event as if the user is directly touching a point corresponding to (238, 458) on UI 218. [...] Hardware-setting module 214 may generate a hardware setting (e.g., a value or a set of values) that dispatches hardware-related module 17 to use a first camera API (not shown) on first computing device 100. The hardware setting may be transmitted to first app 1 by daemon 26 via internet)
Claim 15 is similar in scope to Claim 1; therefore, Claim 15 is rejected under the same rationale as Claim 1.
Related Art
Related Art not relied upon KU et al. (U.S. Pub 2018/0026481) teaches bidirectional simultaneous steaming to multiple devices. Sanders (U.S. Pub 2017/0251235) for teaching synchronized media asset playback on multiple devices.
Inquiry
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANGIE BADAWI whose telephone number is (571)270-7590. The examiner can normally be reached Monday thru Wednesday 9:00am - 5:00pm EST with Thursdays and Fridays off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fred Ehichioya can be reached at (571) 272-4034. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANGIE BADAWI/ Primary Examiner, Art Unit 2179
Prosecution Insights