Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner Notes
Examiner cites particular columns and line numbers in the references as applied to the claims below for convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references cited in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Response to Amendment
The preliminary Amendment filed 10/17/2024 has been entered. Claims 6-8, 13-14, 21-23, 29-30, and 33-50 have been canceled. Claims 1-5, 9-12, 15-20, 24-28, and 31-32 remain pending in the present Office Action.
Claim Objections
Claims 15 and 31 are objected to because of the following informalities:
In claim 15, line 6, “includes” should be corrected to recite “, including[[es]]” or “, wherein completing the computing process includes”.
In claim 31, line 6, “includes” should be corrected to recite “, including[[es]]” or “, wherein completing the computing process includes”.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-5, 9-12, and 32 are rejected under 35 U.S.C. 102(a)(1) and (a)(2) as being anticipated by Desai et al. (U.S. Pub. No. 2020/0312005), hereinafter Desai.
Regarding claim 1, Desai teaches A method comprising:
initiating a computing process on a wearable device, the computing process including a plurality of tasks (Fig. 4A shows an example computing process; [0034] – “display device 100 may be a headset, helmet, goggles, glasses, a phone inserted into an enclosure, etc. worn by a user”; [0041] – “evaluate any of various tasks 154 for potential offloading. These tasks 154 may pertain to the rendering of content being displayed on display device 100 […] as will be discussed below with FIG. 4A, display device 100 may deliver an AR experience that uses an object classifier to identify a particular object captured in video frames collected by a camera sensor 110. Rather than implement the classifier fully at display device 100, distribution engine 150 may offload one or more tasks 154 pertaining the classifier to one or more compute nodes 140.”; [0059] – “a task graph 222 is a graph data structure having multiple nodes 400 corresponding to a set of tasks 154 being considered for offloading. In the illustrated embodiment, task graph 222A is an example of a task graph 222 for a set of tasks 154A-C performed to classify on object present in one or more video frames 402 from a camera sensor 110.”);
identifying a first companion device and determining that the first companion device is available to perform at least one task of the plurality of tasks (Fig. 4A; [0038] – “Compute nodes 140, in various embodiments, are nodes available to assist in producing content used by display device 100 such as facilitating the rendering of 3D view 102.”; [0040] – “Distribution engine 150, in various embodiments, is executable to discover compute nodes 140 and determine whether to offload tasks 154 to the discovered compute nodes 140”; [0043] – distribution engine 150 includes discovery engine 210”; [0044] – “Discovery engine 210, in various embodiments, handles discovery of available compute nodes 140”; [0068] – “task 154B has been assigned to watch 140A”. With respect to the example computing process of FIG. 4A, a “first companion device” determined to be available could correspond to watch 140A, as it would be discovered/identified as available before assignment of a task of the process.);
identifying a second companion device and determining that the second companion device is available to perform at least one task of the plurality of tasks (Fig. 4A; [0040] – “Distribution engine 150, in various embodiments, is executable to discover compute nodes 140 and determine whether to offload tasks 154 to the discovered compute nodes 140”; [0043]-[0044] – distribution engine 150 includes discovery engine 210 to determine available compute nodes 140; [0068] – “task 154C has been assigned to HPC 140E”. With respect to the example computing process of FIG. 4A, a “second companion device” determined to be available could correspond to HCE 140AE as it would be discovered/identified as available before assignment of a task of the process.);
causing the first companion device to perform a first task of the plurality of tasks including communicating data generated by the wearable device to the first companion device (Fig. 4A: “Perform at Watch 140A” for Image Crop 154B, receives Frames w/ Bounding Box 404 output from the object detection task 154A performed at display device 100; [0059] – “a set of tasks 154A-C performed to classify on object present in one or more video frames 402 from a camera sensor 110 […] an image-crop task 154B in which content external to the bounding box is removed from frames 402 to produce cropped frames 406”; [0043] – distribution engine 150 includes task issuer 250; [0048] – “issuer 250 may examine distribution plan 244 to determine that a particular task 154 has been assigned to a particular node 140 and contact that node 140 to request that it perform that assigned task 154. In some embodiments, issuer 250 also handles collecting the appropriate data to perform an assigned task 154 and conveying the data to the node 140. For example, if a given task 154 relies on information from a world sensor 110 and/or user sensor 120 (e.g., images collected by an externally facing camera sensor 110), issuer 250 may assemble this information from the sensor 110 or 120 and communicate this information over a network connection to the compute node 140 assigned the task 154.”);
causing the second companion device to perform a second task of the plurality of tasks (Fig. 4A: “Perform at HPC 140E” for Object Classification 154C; [0059] – “a set of tasks 154A-C performed to classify on object present in one or more video frames 402 from a camera sensor 110 […] an object-classification task 154C in which the cropped frames 406 are analyzed to identify the classification 408 of the object in the cropped frames 406”; [0043] – distribution engine 150 includes task issuer 250; [0048] – “issuer 250 may examine distribution plan 244 to determine that a particular task 154 has been assigned to a particular node 140 and contact that node 140 to request that it perform that assigned task 154.”); and
receiving, by the wearable device from the first companion device, a result associated with a completion of the first task and the second task (FIG. 4A: 408 Identified Classification “To Display system of Display Device”; [0041] – “Display device 100 may then indicate the results of the object classification in 3D view 102. […] distribution engine 150 may also handle collecting results from performance of tasks 154 by nodes 140 and routing the results to the appropriate consuming hardware and/or software in display device 100.”; [0042] – “Although depicted within display device 100, distribution engine 150 may reside elsewhere and, in some embodiments, in multiple locations. For example, a first instance of distribution engine 150 may reside at display device 100 and a second instance of distribution engine 150 may reside at laptop 140B. In such an embodiment, the distribution engine 150 at laptop 140B may collect instances of compute ability information 152 from one or more other compute nodes 140, such as tablet 140C as shown in FIG. 1, and provide a set of tasks 154 offloaded from display device 100 to the other compute nodes 140. In some embodiments, the distribution engine 150 at laptop 140B may forward the received compute ability information 152 ( or combine it with the compute ability information 152 sent by laptop 140B) on to the distribution engine 150 at display device 100, which may determine what to distribute to the other compute nodes 140.” An instance of distribution engine 150, which collects results of from performance of the tasks, may be implemented at a compute node 140 (e.g., at the first companion device), where it may collect data from other compute nodes 140 for communication/forwarding to the display device 100 (e.g., collects results from performance of the offloaded tasks and forwards to the display device 100 for display).).
Regarding claim 2, Desai teaches The method of claim 1, wherein
the wearable device is communicatively coupled to the second companion device ([0037] – “display device 100 includes one or network interfaces for establishing a network connection with compute nodes 140.”; [0073] – “display device 100 includes a display system 510, controller 520, memory 530, secure element 540, and a network interface 550”; [0079] – “Network interface 550, in various embodiments, includes one or more interfaces configured to communicate with external entities such as compute nodes 140”), and
the wearable device is configured to cause the second companion device to perform the second task ([0059] – “a set of tasks 154A-C performed to classify on object present in one or more video frames 402 from a camera sensor 110 […] an object-classification task 154C in which the cropped frames 406 are analyzed to identify the classification 408 of the object in the cropped frames 406”; [0033] – “a display device 100, which includes world sensors 110, user sensors 120, and a distribution engine 150”; [0040] – “Distribution engine 150, in various embodiments, is executable to discover compute nodes 140 and determine whether to offload tasks 154 to the discovered compute nodes 140”; [0043] – distribution engine 150 includes task issuer 250; [0048] – “issuer 250 may examine distribution plan 244 to determine that a particular task 154 has been assigned to a particular node 140 and contact that node 140 to request that it perform that assigned task 154.”).
Regarding claim 3, Desai teaches The method of claim 1, wherein
the first companion device is communicatively coupled to the second companion device ([0042] – the distribution engine 150 at one of the compute nodes 140 communicates with the other compute nodes 140; [0073] – “a given compute node 140 includes a controller 560, memory 570, and network interface 580”; [0082] – “Network interface 580, in various embodiments, includes one or more interfaces configured to communicate with external entities such as display device 100 as well as other compute nodes 140.”), and
the first companion device is configured to cause the second companion device to perform the second task ([0040] – “Distribution engine 150, in various embodiments, is executable to discover compute nodes 140 and determine whether to offload tasks 154 to the discovered compute nodes 140”; [0042] – “Although depicted within display device 100, distribution engine 150 may reside elsewhere and, in some embodiments, in multiple locations. For example, a first instance of distribution engine 150 may reside at display device 100 and a second instance of distribution engine 150 may reside at laptop 140B. In such an embodiment, the distribution engine 150 at laptop 140B may collect instances of compute ability information 152 from one or more other compute nodes 140, such as tablet 140C as shown in FIG. 1, and provide a set of tasks 154 offloaded from display device 100 to the other compute nodes 140. […] the distribution engine 150 at laptop 140B may, instead, make the determination locally as to what should be offloaded to the other nodes 140.”; [0043] – distribution engine 150 includes task issuer 250; [0048] – “issuer 250 may examine distribution plan 244 to determine that a particular task 154 has been assigned to a particular node 140 and contact that node 140 to request that it perform that assigned task 154.”).
Regarding claim 4, Desai teaches The method of claim 1, wherein
the first companion device is communicatively coupled to the second companion device ([0042] – the distribution engine 150 at one of the compute nodes 140 communicates with the other compute nodes 140; [0073] – “a given compute node 140 includes a controller 560, memory 570, and network interface 580”; [0082] – “Network interface 580, in various embodiments, includes one or more interfaces configured to communicate with external entities such as display device 100 as well as other compute nodes 140.”), and
the first companion device and the second companion device are configured to synchronize state information associated with the computing process (Fig. 4A – cropped frames 406, a result of executing the image crop task 154B, are provided from node 400B performed at watch 140A (a “first companion device”) to node 400C performed at HPC 140E (a “second companion device”); [0059] – “an image-crop task 154B in which content external to the bounding box is removed from frames 402 to produce cropped frames 406. Lastly, task graph 222A includes a graph node 400C for an object-classification task 154C in which the cropped frames 406 are analyzed to identify the classification 408 of the object in the cropped frames 406”; [0068] – “task 154B has been assigned to watch 140A, and indicate in node 400C that task 154C has been assigned to HPC 140E.”
For clarity of the record, the Examiner would like to point to paragraph [0059] of the Specification of the instant application, which recites “synchronize state information associated with a task(s) and/or process. State information can include information associated with and/or resulting from, for example, preceding processes, preceding events, preceding user interactions, preceding input and/or output data, logic values, data that can cause a change in state and/or the like.” Thus, providing a result of a preceding task of the computing process performed by the first companion device from the first companion device to the second companion device is “synchroniz[ing] state information associated with the computing process”.).
Regarding claim 5, Desai teaches The method of claim 1, further comprising:
receiving, by the first companion device from the second companion device, a result associated with a completion of the second task ([0041] – “distribution engine 150 may also handle collecting results from performance of tasks 154 by nodes 140 and routing the results to the appropriate consuming hardware and/or software in display device 100.”; [0042] – distribution engine 150 may be located at a compute node 140, i.e., the first companion device).
Regarding claim 9, Desai teaches The method of claim 1, wherein
the computing process is initiated by the first companion device ([0042] – “Although depicted within display device 100, distribution engine 150 may reside elsewhere and, in some embodiments, in multiple locations. For example, a first instance of distribution engine 150 may reside at display device 100 and a second instance of distribution engine 150 may reside at laptop 140B. In such an embodiment, the distribution engine 150 at laptop 140B may collect instances of compute ability information 152 from one or more other compute nodes 140, such as tablet 140C as shown in FIG. 1, and provide a set of tasks 154 offloaded from display device 100 to the other compute nodes 140.”; [0043] – distribution engine 150 includes constraint analyzer 240 and task issuer 250; [0048] – “Task issuer 250, in various embodiments, facilitates implementation of the distribution plan 244 selected by constraint analyzer 240. Accordingly, issuer 250 may examine distribution plan 244 to determine that a particular task 154 has been assigned to a particular node 140 and contact that node 140 to request that it perform that assigned task 154.”), and
initiating the computing process on the wearable device includes receiving a trigger from the first companion device, by the wearable device ([0042] – “In some embodiments, the distribution engine 150 at laptop 140B may forward the received compute ability information 152 (or combine it with the compute ability information 152 sent by laptop 140B) on to the distribution engine 150 at display device 100, which may determine what to distribute to the other compute nodes 140. In some embodiments, the distribution engine 150 at laptop 140B may, instead, make the determination locally as to what should be offloaded to the other nodes 140.”; [0068] – “analyzer 240 may indicate in node 400A that task 154A has been assigned to display device 100”; Fig 4A: the first task of the computing process is Object Detection task 154A assigned to/performed at display device 100. Thus, task issuer 250 (when distribution engine 150 implemented is at compute node 140, i.e., “first companion node”) as described in [0048], determines the Object Detection task 154A is assigned to the display device 100 and would contact and request display device 100 performs the task. Alternatively, the compute node 140 forwarding compute ability information 152 to the display device 100, as described in [0042], could be considered the “trigger”, as it causes determining which tasks of the process should be distributed and subsequent distribution according to that determination.).
Regarding claim 10, Desai teaches The method of claim 1, wherein
the wearable device includes a first socket ([0037] – “display device 100 includes one or network interfaces for establishing a network connection with compute nodes 140. The network connection may be established using any suitable network communication protocol including wireless protocols such as Wi-Fi®, Bluetooth®, Long-Term Evolution™, etc. or wired protocols such as Ethernet, Fibre Channel, Universal Serial Bus™ (USB), etc.”; [0073] – “display device 100 includes a display system 510, controller 520, memory 530, secure element 540, and a network interface 550”.
For clarity of the record, the Examiner would like to point to paragraph [0040] of the Specification of the instant application, which recites “A socket can be one endpoint of a two-way communication link between computer code (e.g., applications, programs, software systems, and/or the like) running on two computing devices.” Thus, a network interface at the wearable device, which is an endpoint of a two-way communication link with a particular compute node 140 (see at least claim 9), is analogous to a “first socket”. The Examiner has similarly interpreted the claimed second, third, and fourth sockets claimed below to be analogous to the network interfaces recited by Desai.),
the first companion device includes a second socket communicatively coupled to the first socket ([0055] – “the types of network interfaces supported by a given compute node 140 such as Wi-Fi®, Bluetooth®, etc. Capabilities 338 may also indicate the network bandwidth available via the network interfaces, which may be dynamic based on communication channel conditions.”; [0073] – “a given compute node 140 includes a controller 560, memory 570, and network interface 580”; [0082] – “Network interface 580, in various embodiments, includes one or more interfaces configured to communicate with external entities such as display device 100”; [0079] – “Network interface 550, in various embodiments, includes one or more interfaces configured to communicate with external entities such as compute nodes 140”. A network interface at a particular compute node 140, which is an endpoint of a two-way communication link with the display device 100 (see at least claim 9), is analogous to a “second socket”.),
the first companion device includes a third socket ([0073] – “a given compute node 140 includes a controller 560, memory 570, and network interface 580”; [0082] – “Network interface 580, in various embodiments, includes one or more interfaces configured to communicate with external entities such as display device 100 as well as other compute nodes 140.” A network interface at a particular compute node 140, which is an endpoint of a two-way communication link with another compute node 140 (see at least [0042]) is analogous to a “third socket”.),
the second companion device includes a fourth socket communicatively coupled to the third socket ([0073] – “a given compute node 140 includes a controller 560, memory 570, and network interface 580”; [0082] – “Network interface 580, in various embodiments, includes one or more interfaces configured to communicate with external entities such as display device 100 as well as other compute nodes 140.” A network interface at another compute node 140, which is an endpoint of a two-way communication link with a particular compute node 140 (see at least [0042]) is analogous to a “fourth socket”.),
causing the first companion device to perform the first task includes writing an instruction and data to the first socket ([0048] – “Task issuer 250, in various embodiments, facilitates implementation of the distribution plan 244 selected by constraint analyzer 240. Accordingly, issuer 250 may examine distribution plan 244 to determine that a particular task 154 has been assigned to a particular node 140 and contact that node 140 to request that it perform that assigned task 154. In some embodiments, issuer 250 also handles collecting the appropriate data to perform an assigned task 154 and conveying the data to the node 140. For example, if a given task 154 relies on information from a world sensor 110 and/or user sensor 120 ( e.g., images collected by an externally facing camera sensor 110), issuer 250 may assemble this information from the sensor 110 or 120 and communicate this information over a network connection to the compute node 140 assigned the task 154”; Claim 1: “distributing, via the network interface, the identified one or more tasks to the one or more compute nodes for processing” The connection between the display device 100 and particular compute node 140 has network interface (“first socket”) on the display device 100 that communicates with the network interface (“second socket”) on the compute node 140. Thus, sending data and the task to the compute node 140 from the display device 100, via the network interfaces, involves providing/writing data to be sent to the network interface on the display device 100.),
causing the first companion device to perform the first task includes reading data from the third socket ([0040] – “determine whether to offload tasks 154 to the discovered compute nodes 140. In the illustrated embodiment, distribution engine 150 make this determination based on compute ability information 152 and the particular tasks 154 being offloaded. Compute ability information 152 may refer generally to any suitable information usable by engine 150 to assess whether tasks 154 should ( or should not) be offloaded to particular compute nodes 140.”; [0042] – “Although depicted within display device 100, distribution engine 150 may reside elsewhere and, in some embodiments, in multiple locations. For example, a first instance of distribution engine 150 may reside at display device 100 and a second instance of distribution engine 150 may reside at laptop 140B. In such an embodiment, the distribution engine 150 at laptop 140B may collect instances of compute ability information 152 from one or more other compute nodes 140, such as tablet 140C as shown in FIG. 1, […] the distribution engine 150 at laptop 140B may forward the received compute ability information 152 (or combine it with the compute ability information 152 sent by laptop 140B) on to the distribution engine 150 at display device 100, which may determine what to distribute”. A distribution engine 150 may be implemented at a particular compute node 140 (e.g., the compute node corresponding to the “first companion device”) which collects/receives compute ability information 152 (“data”) by communicating with the other compute nodes 140, and forwards the compute ability information 152 to the display device 100 for use in determining (and thus in causing) the tasks to be offloaded to the compute nodes. The connection between the particular compute node 140 and the other compute node(s) 140 involves network interface (“third socket”) on the particular compute node 140 that communicates with the network interface (“fourth socket”) on the other compute node(s) 140. Thus, collecting/receiving, via the network interfaces, capability information 152 from another compute node 140 at a particular compute node 140 for forwarding to the display device 100 involves obtaining/reading data from the network interface on the particular compute node 140.), and
receiving the result associated with a completion of the first task includes reading the result from the second socket ([0041] – “distribution engine 150 may also handle collecting results from performance of tasks by nodes 140 and routing the results to the appropriate consumer hardware and/or software in display device 100”. The connection between the display device 100 and particular compute node 140 involves network interface (“first socket”) on the display device 100 that communicates with the network interface (“second socket”) on the compute node 140. Thus, receiving a result of a task from compute node 140 to the display device 100, via the network interfaces, involves obtaining/reading data from the network interface on the compute node 140.).
Regarding claim 11, Desai teaches The method of claim 1, wherein the result includes an image, the method further comprising displaying the image on a display of the wearable device ([0034] – “display device 100 is a head-mounted display (HMD) configured to be worn on the head and to display content to a user. For example, display device 100 may be a headset, helmet, goggles, glasses, a phone inserted into an enclosure, etc. worn by a user. As will be described below with respect to FIG. 5, display device 100 may include a near-eye display system that displays left and right images on screens in front of the user eyes to present 3D view 102 to a user.”; [0041] – “These tasks 154 may pertain to the rendering of content being displayed on display device 100 […] These tasks 154 may also pertain to the rendering in that they affect what is displayed. For example, as will be discussed below with FIG. 4A, display device 100 may deliver an AR experience that uses an object classifier to identify a particular object captured in video frames collected by a camera sensor 110. Rather than implement the classifier fully at display device 100, distribution engine 150 may offload one or more tasks 154 pertaining the classifier to one or more compute nodes 140. Display device 100 may then indicate the results of the object classification in 3D view 102.”).
Regarding claim 12, Desai teaches The method of claim 1, wherein
the wearable device is smart glasses ([0034] – “display device 100 may be a headset, helmet, goggles, glasses, a phone inserted into an enclosure, etc. worn by a user”),
the first companion device is at least one of another wearable device, a mobile device, a smart phone, a tablet, a server, and a device including a processor and an operating system ([0038] – “Compute nodes 140 may be or may include any type of computing system or computing device […] compute ability of his or her phone, watch 140A, laptop 140B, and/or tablet 140C”; [0068] – “watch 140A”), and
the second companion device is at least one of another wearable device, a mobile device, a smart phone, a tablet, a server, and a device including a processor and an operating system ([0038] – “Compute nodes 140 may be or may include any type of computing system or computing device […] high-performance compute (HPC) device 140E […] high-performance compute nodes 140 available to a user though cloud-based services. For example, server cluster 140F may be based at a server farm remote from display device 100 and may implement one or more services for display devices 100 such as rendering three-dimensional content, streaming media, storing rendered content”; [0068] – “HPC 140E”).
Regarding claim 32, Desai teaches A method comprising:
determining, by a first companion device, that the first companion device is available to perform at least one task of a plurality of tasks associated with a computing process (Fig. 4A shows an example computing process; [0038] – “Compute nodes 140, in various embodiments, are nodes available to assist in producing content used by display device 100 such as facilitating the rendering of 3D view 102.”; [0040] – “Distribution engine 150, in various embodiments, is executable to discover compute nodes 140 and determine whether to offload tasks 154 to the discovered compute nodes 140”; [0041] – “evaluate any of various tasks 154 for potential offloading. These tasks 154 may pertain to the rendering of content being displayed on display device 100 […] as will be discussed below with FIG. 4A, display device 100 may deliver an AR experience that uses an object classifier to identify a particular object captured in video frames collected by a camera sensor 110. Rather than implement the classifier fully at display device 100, distribution engine 150 may offload one or more tasks 154 pertaining the classifier to one or more compute nodes 140.”; [0042] – “Although depicted within display device 100, distribution engine 150 may reside elsewhere and, in some embodiments, in multiple locations. For example, a first instance of distribution engine 150 may reside at display device 100 and a second instance of distribution engine 150 may reside at laptop 140B. In such an embodiment, the distribution engine 150 at laptop 140B may collect instances of compute ability information 152 from one or more other compute nodes 140, such as tablet 140C as shown in FIG. 1, and provide a set of tasks 154 offloaded from display device 100 to the other compute nodes 140. […] the distribution engine 150 at laptop 140B may, instead, make the determination locally as to what should be offloaded to the other nodes 140.”; [0043] – distribution engine 150 includes discovery engine 210”; [0044] – “Discovery engine 210, in various embodiments, handles discovery of available compute nodes 140”; [0050] – “rather responding to any solicitation of recruiter 310, a compute node 140 may send a notification 304 indicating that it is available to assist any display device 100 that happens to need assistance.”; [0059] – “a task graph 222 is a graph data structure having multiple nodes 400 corresponding to a set of tasks 154 being considered for offloading. In the illustrated embodiment, task graph 222A is an example of a task graph 222 for a set of tasks 154A-C performed to classify on object present in one or more video frames 402 from a camera sensor 110.” [0068] – “task 154B has been assigned to watch 140A”. With respect to the example computing process of FIG. 4A, a “first companion device” determined to be available could correspond to watch 140A, as it would be discovered/identified as available before assignment of a task of the process. Distribution engine 150, which discovers compute nodes 140 and offloads tasks from display device 100 to compute nodes 140, may be located at one of the compute nodes (“a first companion device”) as described in [0042].);
receiving, by the first companion device from a wearable device, data associated with the computing process generated by the wearable device ([0034] – “display device 100 may be a headset, helmet, goggles, glasses, a phone inserted into an enclosure, etc. worn by a user”; [0043] – distribution engine 150 includes task issuer 250, where distribution engine 150 may reside on a compute node 140; [0048] – “issuer 250 may examine distribution plan 244 to determine that a particular task 154 has been assigned to a particular node 140 and contact that node 140 to request that it perform that assigned task 154. In some embodiments, issuer 250 also handles collecting the appropriate data to perform an assigned task 154 and conveying the data to the node 140. For example, if a given task 154 relies on information from a world sensor 110 and/or user sensor 120 (e.g., images collected by an externally facing camera sensor 110), issuer 250 may assemble this information from the sensor 110 or 120 and communicate this information over a network connection to the compute node 140 assigned the task 154.”);
determining, by the first companion device, that a second companion device is available to perform at least one task of the plurality of tasks associated with the computing process (Fig. 4A; [0040] – “Distribution engine 150, in various embodiments, is executable to discover compute nodes 140 and determine whether to offload tasks 154 to the discovered compute nodes 140”; [0042] – distribution engine 150 may be implemented on a compute node 140, e.g., the “first companion device”; [0043]-[0044] – distribution engine 150 includes discovery engine 210 to determine available compute nodes 140; [0068] – “task 154C has been assigned to HPC 140E”. With respect to the example computing process of FIG. 4A, a “second companion device” determined to be available could correspond to HCE 140AE as it would be discovered/identified as available before assignment of a task of the process.);
performing, by the first companion device, a first task of the plurality of tasks (Fig. 4A: “Perform at Watch 140A” for Image Crop 154B, receives Frames w/ Bounding Box 404 output from the object detection task 154A performed at display device 100; [0059] – “a set of tasks 154A-C performed to classify on object present in one or more video frames 402 from a camera sensor 110 […] an image-crop task 154B in which content external to the bounding box is removed from frames 402 to produce cropped frames 406”);
causing, by the first companion device, a second task of the plurality of tasks to be performed by the second companion device (Fig. 4A: “Perform at HPC 140E” for Object Classification 154C; [0059] – “a set of tasks 154A-C performed to classify on object present in one or more video frames 402 from a camera sensor 110 […] an object-classification task 154C in which the cropped frames 406 are analyzed to identify the classification 408 of the object in the cropped frames 406”; [0042] – distribution engine 150 may be located at a compute node 140, e.g., the “first companion device”; [0043] – distribution engine 150 includes task issuer 250; [0048] – “issuer 250 may examine distribution plan 244 to determine that a particular task 154 has been assigned to a particular node 140 and contact that node 140 to request that it perform that assigned task 154.”); and
communicating, by the first companion device to the wearable device, a result associated with a completion of the first task and associated with a completion of the second task (FIG. 4A: 408 Identified Classification “To Display system of Display Device”; [0041] – “Display device 100 may then indicate the results of the object classification in 3D view 102. […] distribution engine 150 may also handle collecting results from performance of tasks 154 by nodes 140 and routing the results to the appropriate consuming hardware and/or software in display device 100.”; [0042] – “Although depicted within display device 100, distribution engine 150 may reside elsewhere and, in some embodiments, in multiple locations. For example, a first instance of distribution engine 150 may reside at display device 100 and a second instance of distribution engine 150 may reside at laptop 140B. In such an embodiment, the distribution engine 150 at laptop 140B may collect instances of compute ability information 152 from one or more other compute nodes 140, such as tablet 140C as shown in FIG. 1, and provide a set of tasks 154 offloaded from display device 100 to the other compute nodes 140. In some embodiments, the distribution engine 150 at laptop 140B may forward the received compute ability information 152 ( or combine it with the compute ability information 152 sent by laptop 140B) on to the distribution engine 150 at display device 100, which may determine what to distribute to the other compute nodes 140.” An instance of distribution engine 150, which collects results of from performance of the tasks, may be implemented at a compute node 140 (e.g., at the “first companion device”), where it may collect data from other compute nodes 140 for communication/forwarding to the display device 100 (e.g., collects results from performance of the offloaded tasks and forwards to the display device 100 for display).).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Desai in view of O’Hare et al. (U.S. Patent No. 11,204,808), hereinafter O’Hare.
Regarding claim 15, Desai teaches The method of claim 1, wherein
the first companion device includes a […] runtime environment ([0032] – “a display device attempts to discover computing devices available to assist the display device and offloads tasks to these computing devices to expand the amount of available computing resources for delivering content.”; [0038] – “Compute nodes 140, in various embodiments, are nodes available to assist in producing content used by display device 100 such as facilitating the rendering of 3D view 102. Compute nodes 140 may be or may include any type of computing system or computing device.”; [0039] – “compute nodes 140 may vary substantially in their abilities to assist display device 100. […] For example, workstation 140D may execute specialized software such as a VR application capable of providing specialized content.”; [0073] – “a given compute node 140 includes a controller 560, memory 570, and network interface 580.”; [0080]-[0081] – “Controller 560, in various embodiments, includes circuitry configured to facilitate operation of display device 100. Controller 560 may implement any of the functionality described above with respect to controller 520. For example, controller 560 may include one or more processors configured to execute program instructions to cause compute node 140 to perform various operations described herein such as processing code 566 to process offloaded tasks 145. Memory 570, in various embodiments, is configured to store data and program instructions executed by processors in controller 560”),
causing the first companion device to perform the first task includes causing the […] runtime environment to perform the computing process ([0041] – “Rather than implement the classifier fully at display device 100, distribution engine 150 may offload one or more tasks 154 pertaining the classifier to one or more compute nodes 140”; [0080]-[0081] – “controller 560 may include one or more processors configured to execute program instructions to cause compute node 140 to perform various operations described herein such as processing code 566 to process offloaded tasks 145”), and
the result is the completion of the computing process, the method further comprising completing, by the wearable device, the computing process based on the result includes rendering an image on a display of the wearable device (FIG. 4A: 408 Identified Classification “To Display system of Display Device”; [0034] – “display device 100 is a head-mounted display (HMD) configured to be worn on the head and to display content to a user. For example, display device 100 may be a headset, helmet, goggles, glasses, a phone inserted into an enclosure, etc. worn by a user. As will be described below with respect to FIG. 5, display device 100 may include a near-eye display system that displays left and right images on screens in front of the user eyes to present 3D view 102 to a user.”; [0041] – “These tasks 154 may pertain to the rendering of content being displayed on display device 100 […] These tasks 154 may also pertain to the rendering in that they affect what is displayed. For example, as will be discussed below with FIG. 4A, display device 100 may deliver an AR experience that uses an object classifier to identify a particular object captured in video frames collected by a camera sensor 110. Rather than implement the classifier fully at display device 100, distribution engine 150 may offload one or more tasks 154 pertaining the classifier to one or more compute nodes 140. Display device 100 may then indicate the results of the object classification in 3D view 102. […] distribution engine 150 may also handle collecting results from performance of tasks 154 by nodes 140 and routing the results to the appropriate consuming hardware and/or software in display device 100.”).
Desai fails to expressly teach a virtual runtime environment.
However, O’Hare teaches a companion device includes a virtual runtime environment and causing the companion device to perform a task includes causing the virtual runtime environment to perform the computing process (Col. 2, lines 58-67 – “The OCP may outsource 110 the computing task 104 from more constrained devices such as wearables, smart phones, tablets, and others, to proximate devices, such as the IoT device 106, that are less constrained relative to the offloading device. The IoT device 106 may execute the computing task 104 and return 112 the results 114 to the mobile device 102. The IoT device 106 may include a Java virtual machine (JVM) 116 to execute the computing task 104 outsourced 110”; Col. 11, lines 37-44 – “The code in the OCP extensions 530 in an offloading OCP enabled device 500 may create an OCP bundle 534, which includes the information required by the processing device to execute the task. The information may include a return IP address, to which the IP task results should be sent. The code for performing the offload task, such as the Java code to be executed on the JVM in the receiving OCP enabled device 500.”).
Desai and O’Hare are considered to be analogous art to the claimed invention because they are in the same field of offloading tasks from a wearable device to a companion device. Therefore, it would have been obvious to one of ordinary skill in the art to have modified the first companion device taught by Desai to include a virtual runtime environment which performs the tasks of the computing process of the wearable device as taught by O’Hare in order to isolate and protect the offloaded computing workload (O’Hare: Col. 11, lines 31-36).
Claims 16-20 and 24-28 are rejected under 35 U.S.C. 103 as being unpatentable over Desai in view of Legge et al. (U.S. Pub. No. 2020/0110437), hereinafter Legge.
Regarding claim 16, Desai teaches A system ([0033] – “distribution system 10 includes”) comprising:
a wearable device ([0033] – “a display device 100”; [0034] – “display device 100 may be a headset, helmet, goggles, glasses, a phone inserted into an enclosure, etc. worn by a user”);
a first companion device ([0033] – “system 10 may further include one or more compute nodes 140A-F.”; [0038] – “Compute nodes 140, in various embodiments, are nodes available to assist in producing content used by display device 100 such as facilitating the rendering of 3D view 102. Compute nodes 140 may be or may include any type of computing system or computing device”; FIG 4A: watch 140A); and
a second companion device ([0033] – “system 10 may further include one or more compute nodes 140A-F.”; [0038] – “Compute nodes 140, in various embodiments, are nodes available to assist in producing content used by display device 100 such as facilitating the rendering of 3D view 102. Compute nodes 140 may be or may include any type of computing system or computing device”; FIG 4A: HPC 140E),
the wearable device including:
a device client ([0037] – display device 100 includes network interfaces to establish network connections/control communication with compute nodes 140; [0038] and [0050] – compute nodes 140 may provide services to display device 100 to perform offloaded tasks, thus the software on the display device 100 interacting with these services is a “client”), […], and
and at least one peripheral device driver ([0033] – “a display device 100, which includes world sensors 110”; [0035] – “world sensors 110 may include one or more visible-light cameras that capture video information of the user's environment […] world sensors 110 may include one or more world mapping sensors (e.g., infrared (IR) sensors with an IR illumination source, or Light Detection and Ranging (LIDAR) emitters and receivers/detectors) that, for example, capture depth or range information for objects and surfaces in the user's environment. […] world sensors 110 may include one or more light sensors ( e.g., on the front and top of display device 100) that capture lighting information”; [0048] – “information from a world sensor 110 and/or user sensor 120 (e.g., images collected by an externally facing camera sensor 110), issuer 250 may assemble this information from the sensor 110 or 120 and communicate this information over a network connection to the compute node 140 assigned the task 154.” Display device 100 contains world sensors 110 which sense and communicate peripheral data (e.g., captures an image by a camera sensor 110), and necessarily has the software/hardware to control them (i.e., a driver).),
the first companion device including a first runtime environment associated with the wearable device ([0032] – “a display device attempts to discover computing devices available to assist the display device and offloads tasks to these computing devices to expand the amount of available computing resources for delivering content.”; [0038] – “Compute nodes 140, in various embodiments, are nodes available to assist in producing content used by display device 100 such as facilitating the rendering of 3D view 102. Compute nodes 140 may be or may include any type of computing system or computing device.”; [0039] – “compute nodes 140 may vary substantially in their abilities to assist display device 100. […] For example, workstation 140D may execute specialized software such as a VR application capable of providing specialized content.”; [0073] – “a given compute node 140 includes a controller 560, memory 570, and network interface 580.”; [0080]-[0081] – “Controller 560, in various embodiments, includes circuitry configured to facilitate operation of display device 100. Controller 560 may implement any of the functionality described above with respect to controller 520. For example, controller 560 may include one or more processors configured to execute program instructions to cause compute node 140 to perform various operations described herein such as processing code 566 to process offloaded tasks 145. Memory 570, in various embodiments, is configured to store data and program instructions executed by processors in controller 560”),
the second companion device including a second runtime environment associated with the wearable device ([0032], [0038]-[0039], [0073], [0080]-[0081]), and
the system configured to: perform the method of claim 1.
Please see the rejection of claim 1 above regarding specific portions of Desai which teach the steps performed by the system.
Desai fails to expressly teach the wearable device including a hardware abstraction layer and an operating system abstraction layer.
However, Legge teaches a wearable device including a hardware abstraction layer and an operating system abstraction layer ([0130] – “Referring now to FIG. 6A, there is illustrated a schematic block diagram of an example platform architecture implemented by a wearable computing device, such as wearable computing device 110 of FIG. 1 and wearable computing device 210 of FIG. 2. Platform architecture 600 is represented by a “stack” in which successive layers represent increasing levels of abstraction from a bottom physical device layer.”; [0132] – “Platform architecture 600 further has a low-level layer 610, which represents the operating system kernel and device driver software. The kernel represents a lowest-level of abstraction and performs various functions, such as resource management, memory management, device management, and handling of system calls for other programs. For example, the kernel may be responsible for mediating accessing to the wearable computing device's physical resources found in the hardware layer 605.”; [0133] – “platform 600 has a hardware abstraction layer (HAL) 613. Hardware abstraction layer 613 generally presents a device-independent interface corresponding to each physical device of the hardware layer 605, making the device-independent interface available to higher-level elements of the platform in a generally consistent manner, and using the management and mediation of the kernel and drivers of low-level layer 610.”).
Desai and Legge are considered to be analogous art to the claimed invention because they are in the same field of offloading tasks from a wearable device to a companion device. Therefore, it would have been obvious to one of ordinary skill in the art to have modified the wearable device of Desai to incorporate the hardware abstraction layer and operating system abstraction layer taught by Legge, which provide simplified and physical-device independent means for interfacing with and performing operations on the hardware/physical resources of the wearable device (Legge: [0130]-[0134]).
Claim 17 only recites additional limitations which are also recited in claim 3. Thus, for the same reasons presented with respect to claim 3, claim 17 is rejected as being unpatentable over Desai in view of Legge.
Claim 18 recites additional limitations which are the same as those recited in claim 2. Thus, for the same reasons presented with respect to claim 2, claim 18 is rejected as being unpatentable over Desai in view of Legge.
Claim 19 recites additional limitations which are the same as those recited in claim 3. Thus, for the same reasons presented with respect to claim 3, claim 19 is rejected as being unpatentable over Desai in view of Legge.
Claim 20 recites additional limitations which are the same as those recited in claim 4. Thus, for the same reasons presented with respect to claim 4, claim 20 is rejected as being unpatentable over Desai in view of Legge.
Regarding claim 24, the combination of Desai in view of Legge teaches The system of claim 16. Desai further teaches wherein the computing process is initiated by the wearable device ([0033] – “a display device 100, which includes world sensors 110, user sensors 120, and a distribution engine 150”; [0043] – distribution engine 150 includes constraint analyzer 240 and task issuer 250; [0048] – “Task issuer 250, in various embodiments, facilitates implementation of the distribution plan 244 selected by constraint analyzer 240. Accordingly, issuer 250 may examine distribution plan 244 to determine that a particular task 154 has been assigned to a particular node 140 and contact that node 140 to request that it perform that assigned task 154.”).
Claim 25 recites additional limitations which are the same as those recited in claim 9. Thus, for the same reasons presented with respect to claim 9, claim 25 is rejected as being unpatentable over Desai in view of Legge.
Claim 26 recites additional limitations which are the same as those recited in claim 10. Thus, for the same reasons presented with respect to claim 10, claim 26 is rejected as being unpatentable over Desai in view of Legge.
Claim 27 recites additional limitations which are the same as those recited in claim 11. Thus, for the same reasons presented with respect to claim 11, claim 27 is rejected as being unpatentable over Desai in view of Legge.
Claim 28 recites additional limitations which are the same as those recited in claim 12. Thus, for the same reasons presented with respect to claim 12, claim 28 is rejected as being unpatentable over Desai in view of Legge.
Claim 31 is rejected under 35 U.S.C. 103 as being unpatentable over Desai in view of Legge as applied to claim 16 above, and further in view of O’Hare.
Claim 31 recites additional limitations which are the same as those recited in claim 15. Thus, for the same reasons presented with respect to claim 15, claim 31 is rejected as being unpatentable over Desai in view of Legge as applied to claim 16, and further in view of O’Hare.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Sarlandie de la Robertie et al. (U.S. Pub. No. 2017/0331901) teaches one or more companion application(s) in a mobile device cooperating with one or more integrated tracker application(s) in one or more wearable devices to provide extended capabilities for the integrated tracker app including adding connectivity to the Internet to access on-line services and gather data for servers on behalf of the integrated tracker application(s) (see Abstract, FIG. 5, [0099]-[0104]), where Websocket based communications may be used (see [0060]).
Jackson et al. (U.S. Pub. No. 2015/0277540) teaches a wearable device operating with other computing platform(s) to use the other computing platform’s processing, applications, connectivity, etc., where a connection between the wearable device and the computing platform may comprise a secure socket layer connection (see FIG. 1, [0032] and [0058]).
Lakshmikantha et al. (U.S. Patent No. 10,446,119) teaches a system for split rending of multiple graph layers including a wearable display device and a processing device, where computing tasks are performed by the processing device and the graphic layers are displayed at the wearable display device (see Abstract, FIG. 4).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JENNIFER MARIE GUTMAN whose telephone number is (703)756-1572. The examiner can normally be reached M-F: 8:00 am - 4:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kevin Young can be reached at 571-270-3180. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JENNIFER MARIE GUTMAN/Examiner, Art Unit 2194 /KEVIN L YOUNG/Supervisory Patent Examiner, Art Unit 2194
