DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 6/19/2026 has been entered.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 21, 41 and 49 are rejected under 35 U.S.C. 103 as being unpatentable over Kelly et al. (USPN 2020/0103963 A1), in view of Lee et al. (USPN 2015/0091793 A1), further in view of Leibel et al. (USPN 2017/0140570 A1).
As to claim 21, Kelly teaches a method comprising:
identifying two or more computing devices, the two or more computing devices including a first computing device and a second computing device (see at least fig. 8B-C, [0266] “the digital assistant may transmit instructions that causes an external device (e.g., a computer or phone) to transition from a first state to a second state when the external device changes a song, transitions from performing one task to another (e.g., playing music to providing the weather), or transitions from an inactive state to an active state, and vice versa.”; [0267] “when there are multiple external devices (e.g., table lamp 818, floor lamp 814) in the room (or accessible by electronic device 800), electronic device 800 is able to differentiate between the multiple external devices and only transmits instructions to the intended external device”; [0308] “third external device (e.g., 830) doesn't know which external device (e.g., 814, 818) the third external device (e.g., 840) is interacting with. The third external device (e.g., 830) only receives the input and transmits the user input to the electronic device (e.g., 800), which is the device that figures out which external device (e.g., 814, 818) to send the instruction to”);
receiving gaze input data corresponding to one or more users, the gaze input data being received from the first computing device (see at least fig. 6A, 8B-C, [0213] “The electronic device 600 obtains gaze information of the gaze of user 620 using one or more camera sensors optionally located on external devices 616A and/or integrated with electronic device 600, such as camera 616B”; [0261] “Electronic device 800 obtains information relating to the user's gaze through the one or more camera sensors that may be located on external devices 816A and/or is integrated with electronic device 800 at camera 816B. Cameras 816A are wirelessly connected to electronic device 600, though they could alternatively be wired”; [0261] “user 820 indicates which external device (e.g., light) he/she wants to act on (e.g., turn on) by looking at table lamp 818, … The electronic device receives and uses gaze information based on the user's gaze”);
determining, based at least on the gaze input data received from the first computing device, an action to perform on the second computing device (see at least fig. 8B-C, [0261] “The electronic device receives and uses gaze information based on the user's gaze, while the user 820 is speaking, to determine that table lamp 818 is the intended device, when the user's gaze satisfies a set of one or more gaze criteria.”; [0266] “the electronic device determines that the command is to perform a function “turn on” and that the function should be directed at a particular device—the table lamp 818. Accordingly, the electronic device transmits an instruction (e.g., to table lamp 818 or to another device that controls table lamp 818) to turn the table light from an off state to an on state. …, the digital assistant may transmit instructions that causes an external device (e.g., a computer or phone) to transition from a first state to a second state when the external device changes a song, transitions from performing one task to another (e.g., playing music to providing the weather), or transitions from an inactive state to an active state, and vice versa.”; [0288] In accordance with a determination .. using the first gaze information, that a set of one or more gaze criteria is met for the first external device ..: the electronic device .. transmits .., based on the first command, an instruction to transition the first external device .. from the first state to a second state”);
Kelly does not directly teach two or more computing devices associated with a shared computing engine, wherein the shared computing engine is configured to receive state information from each computing device of the two or more computing devices, an application that is currently executing on the second computing device; updating the state information received by the shared computing engine, based at least on the action; and adapting the application executing on the second computing device to perform the action, based at least on the updated state information.
Lee teaches an application that is currently executing on the second computing device (see at least [0077] “the device 2000 sets a control command. .., the device 2000 may match different control commands with each event according to a type of the device 2000, an application type, and a type of a user interface”; [0078] “when a multimedia application is being executed by the device 2000 .., the device 2000 may generate a control command to increase a volume”). Note Kelly’s gaze input identifies the second computing device on which the action is to be performed, while Lee’s application-aware command mapping is used to determine the particular action appropriate for the application currently executing on that second computing device.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Kelly so that the action determined for the selected external device is further based upon the application currently executing on that device, as taught by Lee, because doing so allows commands received through Kelly’s gaze-selection interface to be interpreted appropriately according to the operational context of the selected device, thereby improving command accuracy and user interaction.
Lee does not directly teach two or more computing devices associated with a shared computing engine, wherein the shared computing engine is configured to receive state information from each computing device of the two or more computing devices; updating the state information received by the shared computing engine, based at least on the action; and adapting the application executing on the second computing device to perform the action, based at least on the updated state information.
Leibel teaches two or more computing devices associated with a shared computing engine, wherein the shared computing engine is configured to receive state information from each computing device of the two or more computing devices (see at least [0141] “in case of gaming, computing device 1300 may be the same as the game server computer that arbitrates the game and stays in communication with one or more game client computers, such as computing devices 1440A-N.”; [0171] central renderer 1310 of computing device 1300 of FIG. 14A, detecting/receiving a request or an indication for processing a viewpoint-agnostic workload relating to a graphics scene from one or more satellite/client renderers at one or more satellite/client computers, such as satellite renderers 1450A-N of satellite computers 1440A-N of FIG. 14A” – note requests, indications, workload information, scene information, and client-specific viewpoint information received from the client computing devices constitute state information, because the information represents the current graphics scene, processing condition, viewpoint, and presentation requirements associated with the respective computing devices. Leibel’s centralized engine/central renderer uses the received information to process the common graphics scene and generate information for presentation at the respective client devices);
updating the state information received by the shared computing engine, based at least on the action (see at least [0152] “Once the viewpoint-agnostic tasks are performed by centralized engine 1405, the processing data may then be forwarded on to compilation/presentation logic 1407 to compile the processing data into viewpoint-agnostic data that is presentable or sharable with satellite renderer 1450A-N.”; [0171] “The viewpoint-agnostic workload is evaluated for further processing at block 1605.” – note the action determined using Kelly’s gaze information and Lee’s application-aware command selection modifies the condition of the shared scene or application represented by the state information maintained and processed by Leibel’s shared engine. The engine therefore updates the received state information to reflect the determined gaze-driven action before generating or distributing the resulting processed data); and
adapting the application executing on the second computing device to perform the action, based at least on the updated state information (see at least [0152] “Once the viewpoint-agnostic tasks are performed by centralized engine 1405, the processing data may then be forwarded on to compilation/presentation logic 1407 to compile the processing data into viewpoint-agnostic data that is presentable or sharable with satellite renderer 1450A-N.”; [0171] “the scene may be capable of being delivered at the one or more satellite computers .. with their own specific viewpoints corresponding to their users.” – also note Kelly [0266] “the digital assistant transmits instructions that causes an external device to change tasks as a result of transitioning from a first state to a second state. For example, the digital assistant may transmit instructions that causes an external device (e.g., a computer or phone) to transition from a first state to a second state when the external device changes a song, transitions from performing one task to another (e.g., playing music to providing the weather), or transitions from an inactive state to an active state, and vice versa.” and Lee [0077] “the device 2000 sets a control command. .., the device 2000 may match different control commands with each event according to a type of the device 2000, an application type, and a type of a user interface” - in the combination, Leibel’s shared computing engine updates the shared state to reflect the gaze-driven, application-appropriate action and distributes the corresponding processed or presentation data to the second computing device. The application executing on the second computing device is thereby adapted to perform the determined action based on the updated state information, such as changing a song, changing a task, changing a volume, or presenting an updated graphics scene.).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate Leibel’s shared computing engine into Kelly/Lee’s gaze control multi-device environment because centralized state management allows multiple devices to maintain synchronized operational information while coordinating commands across devices, improving consistency and reducing conflicting device states.
As to claim 41, Kelly teaches a system comprising:
a processor (see at least fig. 1A: processors 120, fig. 5B: processors 516, memory 518, [0200] Memory 518 of personal electronic device 500 can include one or more non-transitory computer-readable storage mediums, for storing computer-executable instructions, which, when executed by one or more computer processors 516, for example, can cause the computer processors to perform the techniques described below,); and
memory storing instructions that, when executed by the processor, cause the system to perform a set of operations (see at least fig. 1A: memory 102, processors 120, fig. 5B: processors 516, memory 518; [0055] “The one or more processors 120 run or execute various software programs and/or sets of instructions stored in memory 102 to perform various functions for device 100 and to process data.”; fig. 5B: processors 516, memory 518; and [0200] “Memory 518 of personal electronic device 500 can include one or more non-transitory computer-readable storage mediums, for storing computer-executable instructions, which, when executed by one or more computer processors 516, for example, can cause the computer processors to perform the techniques described below”), the set of operations comprising:
identifying two or more computing devices, the two or more computing devices including a first computing device and a second computing device (see at least fig. 8B-C, [0266] “the digital assistant may transmit instructions that causes an external device (e.g., a computer or phone) to transition from a first state to a second state when the external device changes a song, transitions from performing one task to another (e.g., playing music to providing the weather), or transitions from an inactive state to an active state, and vice versa.”; [0267] “when there are multiple external devices (e.g., table lamp 818, floor lamp 814) in the room (or accessible by electronic device 800), electronic device 800 is able to differentiate between the multiple external devices and only transmits instructions to the intended external device”; [0308] “third external device (e.g., 830) doesn't know which external device (e.g., 814, 818) the third external device (e.g., 840) is interacting with. The third external device (e.g., 830) only receives the input and transmits the user input to the electronic device (e.g., 800), which is the device that figures out which external device (e.g., 814, 818) to send the instruction to”);
receiving gaze input data corresponding to one or more users, the gaze input data being received from the first computing device (see at least fig. 6A, 8B-C, [0213] “The electronic device 600 obtains gaze information of the gaze of user 620 using one or more camera sensors optionally located on external devices 616A and/or integrated with electronic device 600, such as camera 616B”; [0261] “Electronic device 800 obtains information relating to the user's gaze through the one or more camera sensors that may be located on external devices 816A and/or is integrated with electronic device 800 at camera 816B. Cameras 816A are wirelessly connected to electronic device 600, though they could alternatively be wired”; [0261] “user 820 indicates which external device (e.g., light) he/she wants to act on (e.g., turn on) by looking at table lamp 818, … The electronic device receives and uses gaze information based on the user's gaze”);
determining, based at least on the gaze input data received from the first computing device, an action to perform on the second computing device (see at least fig. 8B-C, [0261] “The electronic device receives and uses gaze information based on the user's gaze, while the user 820 is speaking, to determine that table lamp 818 is the intended device, when the user's gaze satisfies a set of one or more gaze criteria.”; [0266] “the electronic device determines that the command is to perform a function “turn on” and that the function should be directed at a particular device—the table lamp 818. Accordingly, the electronic device transmits an instruction (e.g., to table lamp 818 or to another device that controls table lamp 818) to turn the table light from an off state to an on state. …, the digital assistant may transmit instructions that causes an external device (e.g., a computer or phone) to transition from a first state to a second state when the external device changes a song, transitions from performing one task to another (e.g., playing music to providing the weather), or transitions from an inactive state to an active state, and vice versa.”; [0288] In accordance with a determination .. using the first gaze information, that a set of one or more gaze criteria is met for the first external device ..: the electronic device .. transmits .., based on the first command, an instruction to transition the first external device .. from the first state to a second state”);
Kelly does not directly teach a shared computing engine, wherein the shared computing engine is configured to receive state information from each computing device of the two or more computing devices, an application that is currently executing on the second computing device; updating the state information received by the shared computing engine, based at least on the action; and causing the application executing on the second computing device to perform the action, based at least on the updated state information.
Lee teaches an application that is currently executing on the second computing device (see at least [0077] “the device 2000 sets a control command. .., the device 2000 may match different control commands with each event according to a type of the device 2000, an application type, and a type of a user interface”; [0078] “when a multimedia application is being executed by the device 2000 .., the device 2000 may generate a control command to increase a volume”). Note Kelly’s gaze input identifies the second computing device on which the action is to be performed, while Lee’s application-aware command mapping is used to determine the particular action appropriate for the application currently executing on that second computing device.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Kelly so that the action determined for the selected external device is further based upon the application currently executing on that device, as taught by Lee, because doing so allows commands received through Kelly’s gaze-selection interface to be interpreted appropriately according to the operational context of the selected device, thereby improving command accuracy and user interaction.
Lee does not directly teach a shared computing engine, wherein the shared computing engine is configured to receive state information from each computing device of the two or more computing devices; updating the state information received by the shared computing engine, based at least on the action; and causing the application executing on the second computing device to perform the action, based at least on the updated state information.
Leibel teaches a shared computing engine, wherein the shared computing engine is configured to receive state information from each computing device of the two or more computing devices (see at least [0141] “in case of gaming, computing device 1300 may be the same as the game server computer that arbitrates the game and stays in communication with one or more game client computers, such as computing devices 1440A-N.”; [0171] central renderer 1310 of computing device 1300 of FIG. 14A, detecting/receiving a request or an indication for processing a viewpoint-agnostic workload relating to a graphics scene from one or more satellite/client renderers at one or more satellite/client computers, such as satellite renderers 1450A-N of satellite computers 1440A-N of FIG. 14A” – note requests, indications, workload information, scene information, and client-specific viewpoint information received from the client computing devices constitute state information, because the information represents the current graphics scene, processing condition, viewpoint, and presentation requirements associated with the respective computing devices. Leibel’s centralized engine/central renderer uses the received information to process the common graphics scene and generate information for presentation at the respective client devices);
updating the state information received by the shared computing engine, based at least on the action (see at least [0152] “Once the viewpoint-agnostic tasks are performed by centralized engine 1405, the processing data may then be forwarded on to compilation/presentation logic 1407 to compile the processing data into viewpoint-agnostic data that is presentable or sharable with satellite renderer 1450A-N.”; [0171] “The viewpoint-agnostic workload is evaluated for further processing at block 1605.” – note the action determined using Kelly’s gaze information and Lee’s application-aware command selection modifies the condition of the shared scene or application represented by the state information maintained and processed by Leibel’s shared engine. The engine therefore updates the received state information to reflect the determined gaze-driven action before generating or distributing the resulting processed data); and
causing the application executing on the second computing device to perform the action, based at least on the updated state information (see at least [0152] “Once the viewpoint-agnostic tasks are performed by centralized engine 1405, the processing data may then be forwarded on to compilation/presentation logic 1407 to compile the processing data into viewpoint-agnostic data that is presentable or sharable with satellite renderer 1450A-N.”; [0171] “the scene may be capable of being delivered at the one or more satellite computers .. with their own specific viewpoints corresponding to their users.” – also note Kelly [0266] “the digital assistant transmits instructions that causes an external device to change tasks as a result of transitioning from a first state to a second state. For example, the digital assistant may transmit instructions that causes an external device (e.g., a computer or phone) to transition from a first state to a second state when the external device changes a song, transitions from performing one task to another (e.g., playing music to providing the weather), or transitions from an inactive state to an active state, and vice versa.” and Lee [0077] “the device 2000 sets a control command. .., the device 2000 may match different control commands with each event according to a type of the device 2000, an application type, and a type of a user interface” - in the combination, Leibel’s shared computing engine updates the shared state to reflect the gaze-driven, application-appropriate action and distributes the corresponding processed or presentation data to the second computing device. The application executing on the second computing device is thereby adapted to perform the determined action based on the updated state information, such as changing a song, changing a task, changing a volume, or presenting an updated graphics scene.).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate Leibel’s shared computing engine into Kelly/Lee’s gaze control multi-device environment because centralized state management allows multiple devices to maintain synchronized operational information while coordinating commands across devices, improving consistency and reducing conflicting device states.
As to claim 49, Kelly teaches a computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform a set of operations (see at least fig. 1A: memory 102, processors 120, fig. 5B: processors 516, memory 518; [0055] “The one or more processors 120 run or execute various software programs and/or sets of instructions stored in memory 102 to perform various functions for device 100 and to process data.”; fig. 5B: processors 516, memory 518; and [0200] “Memory 518 of personal electronic device 500 can include one or more non-transitory computer-readable storage mediums, for storing computer-executable instructions, which, when executed by one or more computer processors 516, for example, can cause the computer processors to perform the techniques described below”), the set of operations comprising:
identifying two or more computing devices, the two or more computing devices including a first computing device and a second computing device (see at least fig. 8B-C, [0266] “the digital assistant may transmit instructions that causes an external device (e.g., a computer or phone) to transition from a first state to a second state when the external device changes a song, transitions from performing one task to another (e.g., playing music to providing the weather), or transitions from an inactive state to an active state, and vice versa.”; [0267] “when there are multiple external devices (e.g., table lamp 818, floor lamp 814) in the room (or accessible by electronic device 800), electronic device 800 is able to differentiate between the multiple external devices and only transmits instructions to the intended external device”; [0308] “third external device (e.g., 830) doesn't know which external device (e.g., 814, 818) the third external device (e.g., 840) is interacting with. The third external device (e.g., 830) only receives the input and transmits the user input to the electronic device (e.g., 800), which is the device that figures out which external device (e.g., 814, 818) to send the instruction to”);
receiving gaze input data corresponding to one or more users, the gaze input data being received from the first computing device (see at least fig. 6A, 8B-C, [0213] “The electronic device 600 obtains gaze information of the gaze of user 620 using one or more camera sensors optionally located on external devices 616A and/or integrated with electronic device 600, such as camera 616B”; [0261] “Electronic device 800 obtains information relating to the user's gaze through the one or more camera sensors that may be located on external devices 816A and/or is integrated with electronic device 800 at camera 816B. Cameras 816A are wirelessly connected to electronic device 600, though they could alternatively be wired”; [0261] “user 820 indicates which external device (e.g., light) he/she wants to act on (e.g., turn on) by looking at table lamp 818, … The electronic device receives and uses gaze information based on the user's gaze”);
determining, based at least on the gaze input data received from the first computing device, a selected action to perform on the second computing device(see at least fig. 8B-C, [0261] “The electronic device receives and uses gaze information based on the user's gaze, while the user 820 is speaking, to determine that table lamp 818 is the intended device, when the user's gaze satisfies a set of one or more gaze criteria.”; [0266] “the electronic device determines that the command is to perform a function “turn on” and that the function should be directed at a particular device—the table lamp 818. Accordingly, the electronic device transmits an instruction (e.g., to table lamp 818 or to another device that controls table lamp 818) to turn the table light from an off state to an on state. …, the digital assistant may transmit instructions that causes an external device (e.g., a computer or phone) to transition from a first state to a second state when the external device changes a song, transitions from performing one task to another (e.g., playing music to providing the weather), or transitions from an inactive state to an active state, and vice versa.”; [0288] In accordance with a determination .. using the first gaze information, that a set of one or more gaze criteria is met for the first external device ..: the electronic device .. transmits .., based on the first command, an instruction to transition the first external device .. from the first state to a second state”);
Kelly does not directly teach a shared computing engine, wherein the shared computing engine is configured to receive state information from each computing device of the two or more computing devices, an application that is currently executing on the second computing device; updating the state information received by the shared computing engine, based at least on the selected action; and causing the application executing on the second computing device to perform the selected action, based at least on the updated state information.
Lee teaches an application that is currently executing on the second computing device (see at least [0077] “the device 2000 sets a control command. .., the device 2000 may match different control commands with each event according to a type of the device 2000, an application type, and a type of a user interface”; [0078] “when a multimedia application is being executed by the device 2000 .., the device 2000 may generate a control command to increase a volume”). Note Kelly’s gaze input identifies the second computing device on which the action is to be performed, while Lee’s application-aware command mapping is used to determine the particular action appropriate for the application currently executing on that second computing device.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Kelly so that the action determined for the selected external device is further based upon the application currently executing on that device, as taught by Lee, because doing so allows commands received through Kelly’s gaze-selection interface to be interpreted appropriately according to the operational context of the selected device, thereby improving command accuracy and user interaction.
Lee does not directly teach a shared computing engine, wherein the shared computing engine is configured to receive state information from each computing device of the two or more computing devices; updating the state information received by the shared computing engine, based at least on the selected action; and causing the application executing on the second computing device to perform the selected action, based at least on the updated state information.
Leibel teaches a shared computing engine, wherein the shared computing engine is configured to receive state information from each computing device of the two or more computing devices (see at least [0141] “in case of gaming, computing device 1300 may be the same as the game server computer that arbitrates the game and stays in communication with one or more game client computers, such as computing devices 1440A-N.”; [0171] central renderer 1310 of computing device 1300 of FIG. 14A, detecting/receiving a request or an indication for processing a viewpoint-agnostic workload relating to a graphics scene from one or more satellite/client renderers at one or more satellite/client computers, such as satellite renderers 1450A-N of satellite computers 1440A-N of FIG. 14A” – note requests, indications, workload information, scene information, and client-specific viewpoint information received from the client computing devices constitute state information, because the information represents the current graphics scene, processing condition, viewpoint, and presentation requirements associated with the respective computing devices. Leibel’s centralized engine/central renderer uses the received information to process the common graphics scene and generate information for presentation at the respective client devices);
updating the state information received by the shared computing engine, based at least on the action (see at least [0152] “Once the viewpoint-agnostic tasks are performed by centralized engine 1405, the processing data may then be forwarded on to compilation/presentation logic 1407 to compile the processing data into viewpoint-agnostic data that is presentable or sharable with satellite renderer 1450A-N.”; [0171] “The viewpoint-agnostic workload is evaluated for further processing at block 1605.” – note the action determined using Kelly’s gaze information and Lee’s application-aware command selection modifies the condition of the shared scene or application represented by the state information maintained and processed by Leibel’s shared engine. The engine therefore updates the received state information to reflect the determined gaze-driven action before generating or distributing the resulting processed data); and
causing the application executing on the second computing device to perform the selected action, based at least on the updated state information (see at least [0152] “Once the viewpoint-agnostic tasks are performed by centralized engine 1405, the processing data may then be forwarded on to compilation/presentation logic 1407 to compile the processing data into viewpoint-agnostic data that is presentable or sharable with satellite renderer 1450A-N.”; [0171] “the scene may be capable of being delivered at the one or more satellite computers .. with their own specific viewpoints corresponding to their users.” – also note Kelly [0266] “the digital assistant transmits instructions that causes an external device to change tasks as a result of transitioning from a first state to a second state. For example, the digital assistant may transmit instructions that causes an external device (e.g., a computer or phone) to transition from a first state to a second state when the external device changes a song, transitions from performing one task to another (e.g., playing music to providing the weather), or transitions from an inactive state to an active state, and vice versa.” and Lee [0077] “the device 2000 sets a control command. .., the device 2000 may match different control commands with each event according to a type of the device 2000, an application type, and a type of a user interface” - in the combination, Leibel’s shared computing engine updates the shared state to reflect the gaze-driven, application-appropriate action and distributes the corresponding processed or presentation data to the second computing device. The application executing on the second computing device is thereby adapted to perform the determined action based on the updated state information, such as changing a song, changing a task, changing a volume, or presenting an updated graphics scene.).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate Leibel’s shared computing engine into Kelly/Lee’s gaze control multi-device environment because centralized state management allows multiple devices to maintain synchronized operational information while coordinating commands across devices, improving consistency and reducing conflicting device states.
Claims 22, 25-28, 42, 45-48 and 50-51 are rejected under 35 U.S.C. 103 as being unpatentable over Kelly et al. (USPN 2020/0103963 A1), in view of Lee et al. (USPN 2015/0091793 A1), further in view of Leibel et al. (USPN 2017/0140570 A1), and further in view of Chu et al. (USPN 11,449,149 B2).
As to claim 22, the combination of Kelly, Lee and Leibel teach the method of claim 21 (see above rejection and Kelly at least [0266] “an external device (e.g., a computer or phone) to transition from a first state to a second state when the external device changes a song, transitions from performing one task to another (e.g., playing music to providing the weather)”; [0267] “when there are multiple external devices (e.g., table lamp 818, floor lamp 814) in the room (or accessible by electronic device 800), electronic device 800 is able to differentiate between the multiple external devices and only transmits instructions to the intended external device”; and Lee at least [0077] “a type of the device 2000, an application type, and a type of a user interface. In addition, an area inside and an area outside the device 2000 may be divided into a plurality of areas as previously described”; [0078] “when a multimedia application is being executed by the device 2000 and the user blinks once while gazing at a central upper area of the screen of the device 2000, the device 2000 may generate a control command to increase a volume of the device 2000”).
Kelly, Lee and Leibel do not directly teach wherein the action involves presenting the application on the second device in response to the gaze input data received from the first computing device.
Chu teaches wherein the action involves presenting the application on the second device in response to the gaze input data received from the first computing device (see at least col. 7 lines 38-58 “the music application icon can be rendered in the lenses 206 above the second computing device.”, and col. 7 line 59 – col. 8 line 20 “the user 202 can adjust their gaze and/or the direction of the computerized glasses 204 more towards the first computing device or the second computing device. In response, the automated assistant can detect the adjustment of the gaze and/or facing direction of the user and cause the first instance or the second instance of the music application icon to provide feedback that one has been selected.”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the gaze-controlled multi-device system of Kelly, as supplemented by the application-aware command selection of Lee and the centralized state coordination of Leibel, to use Chu’s gaze-based selection of a device-associated application instance and to present the selected application on the selected second computing device. Presenting the selected application on the selected device would have been the predictable implementation of Chu’s application-instance selection and would have provided immediate confirmation that the intended application and intended target device were selected. The combination would have amounted to applying Chu’s known gaze-based application-selection technique to Kelly’s known gaze-based device-control system, using Lee’s known application-aware action mapping and Leibel’s known centralized state coordination, with no change in the respective principles of operation.
As to claim 25, the combination of Kelly, Lee and Leibel teach the method of claim 21 (see above rejection).
Kelly, Lee and Leibel do not directly teach wherein the action comprises presenting a particular tab on the second computing device when the gaze input data indicates the one or more users are gazing at the particular tab on the first computing device.
Chu teaches wherein the action comprises presenting a particular tab on the second computing device when the gaze input data indicates the one or more users are gazing at the particular tab on the first computing device (see at least col. 7 lines 38-58 “the music application icon can be rendered in the lenses 206 above the second computing device.”, and col. 7 line 59 – col. 8 line 20 “the user 202 can adjust their gaze and/or the direction of the computerized glasses 204 more towards the first computing device or the second computing device. In response, the automated assistant can detect the adjustment of the gaze and/or facing direction of the user and cause the first instance or the second instance of the music application icon to provide feedback that one has been selected.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to replace Chu’s application icon with another well-known selectable graphical user interface element (i.e. a tab) because both are well-known selectable graphical user interface elements used to identify and access particular application content. A tab merely represents a different graphical interface control for selecting application content than an application icon, while performing the same well-understood function of allowing a user to select a particular application view or content. Further, because Kelly teaches selecting a target computing device using gaze input, Lee teaches selecting an action according to the application executing on the target computing device, and Leibel teaches propagating updated shared state information between computing devices, one of ordinary skill in the art would have found it obvious to present the corresponding selected tab on the second computing device when the user gazes at that tab on the first computing device. This modification merely substitutes one known graphical user interface selection element for another while preserving the same gaze-based selection mechanism and shared multi-device operation, yielding the predictable result of presenting the selected application content on the target computing device.
As to claim 26, the combination of Kelly, Lee, Leibel and Chu teach the method of claim 22 (see above rejection), further comprising: receiving context data from the first computing device; and selecting the action to perform on the second computing device based at least on the context data received from the first computing device (see Kelly at least [0042] “FIGS. 8A-8L illustrate exemplary user interfaces for providing context to commands using gaze information.”; [0258] “FIGS. 8A-8L illustrate exemplary user interfaces for the electronic device to use gaze information to determine the context (e.g., an appliance that the command is directed at) for performing a command once the digital assistant has been activated”; [0261] “user 820 indicates which external device (e.g., light) he/she wants to act on (e.g., turn on) by looking at table lamp 818”); [0266] “the electronic device determines that the command is to perform a function “turn on” and that the function should be directed at a particular device—the table lamp 818.”).
As to claim 27, the combination of Kelly, Lee, Leibel and Chu teach the method of claim 26 (see above rejection), wherein the context data identifies a particular hardware device that is coupled to the first computing device (see Kelly at least [0258] “FIGS. 8A-8L illustrate exemplary user interfaces for the electronic device to use gaze information to determine the context (e.g., an appliance that the command is directed at) for performing a command once the digital assistant has been activated”; [0261] “user 820 indicates which external device (e.g., light) he/she wants to act on (e.g., turn on) by looking at table lamp 818”); [0266] “the electronic device determines that the command is to perform a function “turn on” and that the function should be directed at a particular device—the table lamp 818. Accordingly, the electronic device transmits an instruction (e.g., to table lamp 818 or to another device that controls table lamp 818) to turn the table light from an off state to an on state”; [0267] “when there are multiple external devices (e.g., table lamp 818, floor lamp 814) in the room (or accessible by electronic device 800), electronic device 800 is able to differentiate between the multiple external devices and only transmits instructions to the intended external device” – note table lamp is a hardware device coupled to electronic device).
As to claim 28, the combination of Kelly, Lee and Leibel teach the method of claim 21 (see above rejection).
Kelly, Lee and Leibel do not directly teach wherein the action is performed in a first instance when the application is associated with the gaze input data and a default action is performed in a second instance when the application is not associated with the gaze input data.
Chu teaches wherein the action is performed in a first instance when the application is associated with the gaze input data and a default action is performed in a second instance when the application is not associated with the gaze input data (see at least col. 7 line 59 – col. 8 line 20 “each instance of the music application icon can be rendered in a way that indicates to the user that a particular device has not been selected …the music application icon can be “grayed out,” blurry, blinking, and/or otherwise have one or more features that indicate that one of the devices should be selected by the user. …, when the user 202 directs their gaze .., the first instance of the music application can icon blink, shake, become idle, no long be grayed out, no longer be blurry, ... In this way, the user 202 can receive feedback that they have selected a particular device” – note Chu teaches two operating instances based upon whether an application/application instance is associated with the user’s gaze. In the first instance, when the user’s gaze is directed to and thereby associated with the particular application instance/device, the system performs the gaze-responsive action by changing the state or appearance of the selected application icon to indicate selection. In the second instance, when an application instance is not associated with the user’s gaze and therefore has not been selected, the system performs the default action of maintaining/rendering that application instance in its default unselected state, such as grayed out or blurry).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate Chu’s gaze-dependent selected and unselected application behavior into the gaze-controlled multi-device system of Kelly, as modified by Lee and Leibel, because doing so would provide the user with feedback distinguishing an application/device associated with the user’s gaze from applications/devices that are not associated with the user’s gaze, thereby facilitating identification of the target of the gaze-based command and reducing unintended device or application selection.
As to claim 42, the combination of Kelly, Lee and Leibel teach the system of claim 41 (see above rejection and Kelly at least [0266] “an external device (e.g., a computer or phone) to transition from a first state to a second state when the external device changes a song, transitions from performing one task to another (e.g., playing music to providing the weather)”; [0267] “when there are multiple external devices (e.g., table lamp 818, floor lamp 814) in the room (or accessible by electronic device 800), electronic device 800 is able to differentiate between the multiple external devices and only transmits instructions to the intended external device”; and Lee at least [0077] “a type of the device 2000, an application type, and a type of a user interface. In addition, an area inside and an area outside the device 2000 may be divided into a plurality of areas as previously described”; [0078] “when a multimedia application is being executed by the device 2000 and the user blinks once while gazing at a central upper area of the screen of the device 2000, the device 2000 may generate a control command to increase a volume of the device 2000”).
Kelly, Lee and Leibel do not directly teach wherein the action involves presenting the application on the second device in response to the gaze input data received from the first computing device.
Chu teaches wherein the action involves presenting the application on the second device in response to the gaze input data received from the first computing device (see at least col. 7 lines 38-58 “the music application icon can be rendered in the lenses 206 above the second computing device.”, and col. 7 line 59 – col. 8 line 20 “the user 202 can adjust their gaze and/or the direction of the computerized glasses 204 more towards the first computing device or the second computing device. In response, the automated assistant can detect the adjustment of the gaze and/or facing direction of the user and cause the first instance or the second instance of the music application icon to provide feedback that one has been selected.”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the gaze-controlled multi-device system of Kelly, as supplemented by the application-aware command selection of Lee and the centralized state coordination of Leibel, to use Chu’s gaze-based selection of a device-associated application instance and to present the selected application on the selected second computing device. Presenting the selected application on the selected device would have been the predictable implementation of Chu’s application-instance selection and would have provided immediate confirmation that the intended application and intended target device were selected. The combination would have amounted to applying Chu’s known gaze-based application-selection technique to Kelly’s known gaze-based device-control system, using Lee’s known application-aware action mapping and Leibel’s known centralized state coordination, with no change in the respective principles of operation.
As to claim 45, the combination of Kelly, Lee and Leibel teach the system of claim 41 (see above rejection).
Kelly, Lee and Leibel do not directly teach wherein the action comprises presenting a particular tab on the second computing device when the gaze input data indicates the one or more users are gazing at the particular tab on the first computing device.
Chu teaches wherein the action comprises presenting a particular tab on the second computing device when the gaze input data indicates the one or more users are gazing at the particular tab on the first computing device (see at least col. 7 lines 38-58 “the music application icon can be rendered in the lenses 206 above the second computing device.”, and col. 7 line 59 – col. 8 line 20 “the user 202 can adjust their gaze and/or the direction of the computerized glasses 204 more towards the first computing device or the second computing device. In response, the automated assistant can detect the adjustment of the gaze and/or facing direction of the user and cause the first instance or the second instance of the music application icon to provide feedback that one has been selected.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to replace Chu’s application icon with another well-known selectable graphical user interface element (i.e. a tab) because both are well-known selectable graphical user interface elements used to identify and access particular application content. A tab merely represents a different graphical interface control for selecting application content than an application icon, while performing the same well-understood function of allowing a user to select a particular application view or content. Further, because Kelly teaches selecting a target computing device using gaze input, Lee teaches selecting an action according to the application executing on the target computing device, and Leibel teaches propagating updated shared state information between computing devices, one of ordinary skill in the art would have found it obvious to present the corresponding selected tab on the second computing device when the user gazes at that tab on the first computing device. This modification merely substitutes one known graphical user interface selection element for another while preserving the same gaze-based selection mechanism and shared multi-device operation, yielding the predictable result of presenting the selected application content on the target computing device.
As to claim 46, the combination of Kelly, Lee, Leibel and Chu teach the system of claim 45 (see above rejection), further comprising: receiving context data from the first computing device; and selecting the action to perform on the second computing device based at least on the context data received from the first computing device (see Kelly at least [0042] “FIGS. 8A-8L illustrate exemplary user interfaces for providing context to commands using gaze information.”; [0258] “FIGS. 8A-8L illustrate exemplary user interfaces for the electronic device to use gaze information to determine the context (e.g., an appliance that the command is directed at) for performing a command once the digital assistant has been activated”; [0261] “user 820 indicates which external device (e.g., light) he/she wants to act on (e.g., turn on) by looking at table lamp 818”); [0266] “the electronic device determines that the command is to perform a function “turn on” and that the function should be directed at a particular device—the table lamp 818.”).
As to claim 47, the combination of Kelly, Lee, Leibel and Chu teach the system of claim 46 (see above rejection), wherein the context data identifies a particular hardware device that is coupled to the first computing device (see Kelly at least [0258] “FIGS. 8A-8L illustrate exemplary user interfaces for the electronic device to use gaze information to determine the context (e.g., an appliance that the command is directed at) for performing a command once the digital assistant has been activated”; [0261] “user 820 indicates which external device (e.g., light) he/she wants to act on (e.g., turn on) by looking at table lamp 818”); [0266] “the electronic device determines that the command is to perform a function “turn on” and that the function should be directed at a particular device—the table lamp 818. Accordingly, the electronic device transmits an instruction (e.g., to table lamp 818 or to another device that controls table lamp 818) to turn the table light from an off state to an on state”; [0267] “when there are multiple external devices (e.g., table lamp 818, floor lamp 814) in the room (or accessible by electronic device 800), electronic device 800 is able to differentiate between the multiple external devices and only transmits instructions to the intended external device” – note table lamp is a hardware device coupled to electronic device).
As to claim 48, the combination of Kelly, Lee and Leibel teach the system of claim 41 (see above rejection).
Kelly, Lee and Leibel do not directly teach wherein the action is performed in a first instance when the application is associated with the gaze input data and a default action is performed in a second instance when the application is not associated with the gaze input data.
Chu teaches wherein the action is performed in a first instance when the application is associated with the gaze input data and a default action is performed in a second instance when the application is not associated with the gaze input data (see at least col. 7 line 59 – col. 8 line 20 “each instance of the music application icon can be rendered in a way that indicates to the user that a particular device has not been selected …the music application icon can be “grayed out,” blurry, blinking, and/or otherwise have one or more features that indicate that one of the devices should be selected by the user. …, when the user 202 directs their gaze .., the first instance of the music application can icon blink, shake, become idle, no long be grayed out, no longer be blurry, ... In this way, the user 202 can receive feedback that they have selected a particular device” – note Chu teaches two operating instances based upon whether an application/application instance is associated with the user’s gaze. In the first instance, when the user’s gaze is directed to and thereby associated with the particular application instance/device, the system performs the gaze-responsive action by changing the state or appearance of the selected application icon to indicate selection. In the second instance, when an application instance is not associated with the user’s gaze and therefore has not been selected, the system performs the default action of maintaining/rendering that application instance in its default unselected state, such as grayed out or blurry).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate Chu’s gaze-dependent selected and unselected application behavior into the gaze-controlled multi-device system of Kelly, as modified by Lee and Leibel, because doing so would provide the user with feedback distinguishing an application/device associated with the user’s gaze from applications/devices that are not associated with the user’s gaze, thereby facilitating identification of the target of the gaze-based command and reducing unintended device or application selection.
As to claim 50, the combination of Kelly, Lee and Leibel teach the computer-readable storage medium of claim 49 (see above rejection and Kelly at least [0266] “an external device (e.g., a computer or phone) to transition from a first state to a second state when the external device changes a song, transitions from performing one task to another”; Lee at least [0077] “a type of the device 2000, an application type, and a type of a user interface. In addition, an area inside and an area outside the device 2000 may be divided into a plurality of areas as previously described”; [0078] “when a multimedia application is being executed by the device 2000 and the user blinks once while gazing at a central upper area of the screen of the device 2000, the device 2000 may generate a control command to increase a volume of the device 2000”).
Kelly, Lee and Leibel do not directly teach the set of operations further comprising: detecting that the gaze input data indicates that a particular user has gazed at a second application on the first computing device; and in response to detecting that the gaze input data indicates that the particular user has gazed at the second application on the first computing device, causing the second application to perform a second action on the second computing device.
Chu teaches the set of operations further comprising: detecting that the gaze input data indicates that a particular user has gazed at a second application on the first computing device; and in response to detecting that the gaze input data indicates that the particular user has gazed at the second application on the first computing device, causing the second application to perform a second action on the second computing device (see at least col. 7 lines 38-58 “the music application icon can be rendered in the lenses 206 above the second computing device.”, and col. 7 line 59 – col. 8 line 20 “the user 202 can adjust their gaze and/or the direction of the computerized glasses 204 more towards the first computing device or the second computing device. In response, the automated assistant can detect the adjustment of the gaze and/or facing direction of the user and cause the first instance or the second instance of the music application icon to provide feedback that one has been selected.”). Note, in the combination, Chu’s gaze-based selection of an application instance would identify the second application selected by the particular user’s gaze; Lee’s application-aware command mapping would determine the second action appropriate for that second application; and Kelly’s multi-device control would cause the corresponding application on the second computing device to perform the second action. The shared-state processing taught by Leibel, would provide the corresponding action within the multi-device environment.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the Kelly/Lee/Leibel system to employ Chu’s gaze-based selection of a device-associated application instance because Chu expressly teaches using a user’s gaze to select between application instances associated with different computing devices. Applying Chu’s gaze-based application selection to Kelly’s gaze-controlled multi-device environment would have predictably allowed a user to select an application for control using the same gaze input already used by Kelly to select a computing device, while Lee’s application-aware command mapping would have provided an action appropriate to the selected application. It further would have been obvious to apply this known gaze-based, application-aware control technique to a second application and a second action, as recited, because the combined system is not technically limited to a single application or action. Lee teaches matching control commands according to “an application type” ([0077]), Kelly teaches different application-related actions including changing a song and transitioning from one task to another ([0266]), and Chu teaches gaze-based selection of an application instance. Applying the same known selection-and-control process to another application would have been a predictable use of the combined teachings to permit control of multiple applications in an ordinary multi-application computing environment.
As to claim 51, the combination of Kelly, Lee and Leibel teach the computer-readable storage medium of claim 49 (see above rejection).
Kelly, Lee and Leibel do not directly teach the set of operations further comprising: determining whether to perform the selected action or a default action based at least on whether the application is associated with the gaze input data; and in at least one instance when the application is not associated with the gaze input data, performing the default action instead of the selected action.
Chu teaches the set of operations further comprising: determining whether to perform the selected action or a default action based at least on whether the application is associated with the gaze input data; and in at least one instance when the application is not associated with the gaze input data, performing the default action instead of the selected action (see at least col. 7 line 59 – col. 8 line 20 “each instance of the music application icon can be rendered in a way that indicates to the user that a particular device has not been selected …the music application icon can be “grayed out,” blurry, blinking, and/or otherwise have one or more features that indicate that one of the devices should be selected by the user. …, when the user 202 directs their gaze .., the first instance of the music application can icon blink, shake, become idle, no long be grayed out, no longer be blurry, ... In this way, the user 202 can receive feedback that they have selected a particular device” – note Chu teaches two operating instances based upon whether an application/application instance is associated with the user’s gaze. In the first instance, when the user’s gaze is directed to and thereby associated with the particular application instance/device, the system performs the gaze-responsive action by changing the state or appearance of the selected application icon to indicate selection. In the second instance, when an application instance is not associated with the user’s gaze and therefore has not been selected, the system performs the default action of maintaining/rendering that application instance in its default unselected state, such as grayed out or blurry).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate Chu’s gaze-dependent selected and unselected application behavior into the gaze-controlled multi-device system of Kelly, as modified by Lee and Leibel, because doing so would provide the user with feedback distinguishing an application/device associated with the user’s gaze from applications/devices that are not associated with the user’s gaze, thereby facilitating identification of the target of the gaze-based command and reducing unintended device or application selection.
Claims 23 and 43 are rejected under 35 U.S.C. 103 as being unpatentable over Kelly et al. (USPN 2020/0103963 A1), in view of Lee et al. (USPN 2015/0091793 A1), in view of Leibel et al. (USPN 2017/0140570 A1), further in view of Chu et al. (USPN 11,449,149 B2) and Faulkner (USPN 2020/0371673 A1).
As to claim 23, the combination of Kelly, Lee, Leibel and Chu teach the method of claim 22 (see above rejection and Chu at least col. 7 lines 38-58 “the music application icon can be rendered in the lenses 206 above the second computing device.”, and col. 7 line 59 – col. 8 line 20 “the user 202 can adjust their gaze and/or the direction of the computerized glasses 204 more towards the first computing device or the second computing device.).
Kelly, Lee, Leibel and Chu do not directly teach wherein the gaze input data indicates that a particular user gazed at the application on the first computing device.
Faulkner teaches wherein the gaze input data indicates that a particular user gazed at the application on the first computing device (see at least [0088] “the system can select an object in response to determining that a gaze target meets one or more criteria with respect to that object.”; [0105] “the user selects the virtual object 110B using a gaze gesture that causes the system 10 define the gaze target 112.”; and [0106] “In response the selection, the system can change the mode of an application.”).
It would have been obvious to a person having ordinary skill in the art before the effective date of the claimed invention to use Faulkner’s object-level gaze-target determination to determine which of Chu’s device-associated application objects or application instances the particular user was viewing. Applying Faulkner’s gaze-target determination to Chu’s application object would have predictably allowed the system to determine that the particular user gazed at the application associated with the first computing device before causing that application to be presented on the second computing device. The modification would have improved selection accuracy by distinguishing gaze directed to the relevant application from gaze merely directed generally toward the computing device. It would also have reduced unintended activation when multiple application objects or device-associated application instances were visible.
As to claim 43, the combination of Kelly, Lee, Leibel and Chu teach the system of claim 42 (see above rejection and Chu at least col. 7 lines 38-58 “the music application icon can be rendered in the lenses 206 above the second computing device.”, and col. 7 line 59 – col. 8 line 20 “the user 202 can adjust their gaze and/or the direction of the computerized glasses 204 more towards the first computing device or the second computing device.).
Kelly, Lee, Leibel and Chu do not directly teach wherein the gaze input data indicates that a particular user gazed at the application on the first computing device.
Faulkner teaches wherein the gaze input data indicates that a particular user gazed at the application on the first computing device (see at least [0088] “the system can select an object in response to determining that a gaze target meets one or more criteria with respect to that object.”; [0105] “the user selects the virtual object 110B using a gaze gesture that causes the system 10 define the gaze target 112.”; and [0106] “In response the selection, the system can change the mode of an application.”).
It would have been obvious to a person having ordinary skill in the art before the effective date of the claimed invention to use Faulkner’s object-level gaze-target determination to determine which of Chu’s device-associated application objects or application instances the particular user was viewing. Applying Faulkner’s gaze-target determination to Chu’s application object would have predictably allowed the system to determine that the particular user gazed at the application associated with the first computing device before causing that application to be presented on the second computing device. The modification would have improved selection accuracy by distinguishing gaze directed to the relevant application from gaze merely directed generally toward the computing device. It would also have reduced unintended activation when multiple application objects or device-associated application instances were visible.
Claims 24 and 44 are rejected under 35 U.S.C. 103 as being unpatentable over Kelly et al. (USPN 2020/0103963 A1), in view of Lee et al. (USPN 2015/0091793 A1), in view of Leibel et al. (USPN 2017/0140570 A1), further in view of Chu et al. (USPN 11,449,149 B2) and Dow et al. (US 2018/0081431 A1).
As to claim 24, the combination of Kelly, Lee, Leibel and Chu teach the method of claim 22 (see above rejection)
Kelly, Lee, Leibel and Chu do not directly teach wherein the gaze input data indicates that more than half of multiple users gazed at the application on the first computing device.
Dow teaches wherein the gaze input data indicates that more than half of multiple users gazed at the application on the first computing device (see at least [0022] “the gaze tracker 105 is monitoring for one or more users”; Claim 1 “monitoring, by the gaze tracker, one or more of a plurality of users approaching the display device; capturing, by the gaze tracker, gaze direction data from the plurality of users viewing the initial visualization, wherein the gaze direction data comprises an eye motion, a head position, and a head direction of each of the plurality of users, .. processing, by a processor of the computing device, the gaze direction data to determine multiple points of interest for the plurality of users; determining, by the processor, a common region of the multiple points of interest to acquire supplementary information for all of the plurality of users”). Dow does not directly teach determining whether “more than half” of the users gaze at a particular region. However, once the individual points of interest for the plurality of users are determined as taught by Dow, determining whether a particular region is viewed by more than half of those users would have been a straightforward and predictable threshold technique for identifying the predominant region of collective interest.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the combined system to count the individual gaze determinations supplied by Dow and select or present Chu’s application when the number of users gazing at that application exceeds one-half of the participating users. A majority threshold would have predictably identified the application receiving the predominant attention of the group while preventing a minority of users from controlling the shared presentation.
As to claim 44, the combination of Kelly, Lee, Leibel and Chu teach the system of claim 42 (see above rejection)
Kelly, Lee, Leibel and Chu do not directly teach wherein the gaze input data indicates that more than half of multiple users gazed at the application on the first computing device.
Dow teaches wherein the gaze input data indicates that more than half of multiple users gazed at the application on the first computing device (see at least [0022] “the gaze tracker 105 is monitoring for one or more users”; Claim 1 “monitoring, by the gaze tracker, one or more of a plurality of users approaching the display device; capturing, by the gaze tracker, gaze direction data from the plurality of users viewing the initial visualization, wherein the gaze direction data comprises an eye motion, a head position, and a head direction of each of the plurality of users, .. processing, by a processor of the computing device, the gaze direction data to determine multiple points of interest for the plurality of users; determining, by the processor, a common region of the multiple points of interest to acquire supplementary information for all of the plurality of users”). Dow does not directly teach determining whether “more than half” of the users gaze at a particular region. However, once the individual points of interest for the plurality of users are determined as taught by Dow, determining whether a particular region is viewed by more than half of those users would have been a straightforward and predictable threshold technique for identifying the predominant region of collective interest.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the combined system to count the individual gaze determinations supplied by Dow and select or present Chu’s application when the number of users gazing at that application exceeds one-half of the participating users. A majority threshold would have predictably identified the application receiving the predominant attention of the group while preventing a minority of users from controlling the shared presentation.
Claim 52 is rejected under 35 U.S.C. 103 as being unpatentable over Kelly et al. (USPN 2020/0103963 A1), in view of Lee et al. (USPN 2015/0091793 A1), in view of Leibel et al. (USPN 2017/0140570 A1) and further in view of Faulkner (USPN 2020/0371673 A1).
As to claim 52, the combination of Kelly, Lee and Leibel teach the computer-readable storage medium of claim 49 (see above rejection).
Kelly, Lee and Leibel do not directly teach the selected action involving enlarging the application on the second computing device.
Faulkner teaches the selected action involving enlarging the application on the second computing device (see at least [0077] “enlarging the display area 121 of the selected object.”; [0078] “the display area 121B of the selected object is enlarged.”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the gaze-based, application-aware multi-device system of Kelly, as modified by Lee and Leibel, so that the selected action includes enlarging the application on the second computing device, as taught by Faulkner, because enlarging a gaze-selected display area provides increased visual emphasis and brings focus to the content selected by the user’s gaze, thereby facilitating the user’s interaction with the selected content. Such a modification would have predictably applied Faulkner’s known gaze-responsive enlargement technique to the application selected for action in the Kelly/Lee/Leibel system to provide enhanced visibility and focus of the selected application on the second computing device.
Response to Arguments
Applicant’s arguments filed 6/19/2026 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JENNIFER L ZUBAJLO whose telephone number is (571)270-1551. The examiner can normally be reached Monday - Thursday 10 am - 8 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KE XIAO can be reached at 571-272-7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JENNIFER L ZUBAJLO/Examiner, Art Unit 2627 8/2/2026
/KE XIAO/Supervisory Patent Examiner, Art Unit 2627
Prosecution Insights