DETAILED ACTION
In response to communication filed on 01 July 2026, claim 8 has been canceled. Claim 21 is the newly added claim. Claims 1-7 and 10-20 are amended. Claims 1-7 and 9-21 are pending.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see “Remarks Concerning Objections”, filed 01 July 2026, have been carefully considered. Based on the claim amendments the claim objections have been withdrawn.
Applicant’s arguments, see “Remarks Concerning Rejections Under 35 U.S.C. § 101”, filed 01 July 2026, have been carefully considered. Based on the claim amendments and the persuasive remarks filed on pages 11-13, the claim rejections have been withdrawn.
Applicant’s arguments, see “Remarks Concerning Rejections Under 35 U.S.C. § 103”, filed 01 July 2026, have been carefully considered but the arguments are not considered to be persuasive since the arguments are related to newly added limitations and those are addressed in the rejection below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 9-14 and 16-21 are rejected under 35 U.S.C. 103 as being unpatentable over Carbune et al. (US 2022/0272055 A1, hereinafter “Carbune”) in view of Cella et al. (US 2024/0144141 A1, hereinafter “Cella”) further in view of Huang (US 2022/0337780 A1, hereinafter “Huang”).
Regarding claim 1, Carbune teaches
A non-transitory, computer-readable storage medium including executable instructions that, when executed by one or more processors, cause the one or more processors to perform: (see Carbune, [0141] “Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform”).
… providing an Al agent sensor data… (see Carbune, [0021] “The sensor data can include any data generated by an assistant input device of the user…the instance of the sensor data can include data generated by multiple assistant input devices and/or assistant non-input devices”; [0036] “a given one of the assistant devices 106, 185 may be equipped with a presence sensor 105 that detects various types of wireless signals (e.g., waves such as radio, ultrasonic, electromagnetic, etc.) emitted by, for instance, other assistant devices carried/operated by a particular user (e.g., a mobile device, a wearable computing device, etc.) and/or other assistant devices in the ecosystem”; [0087] “the system obtains, via one or more sensors of an assistant device of a user, an instance of sensor data. The instance of the sensor data can include, for example… any other sensor data generated by various sensors of the assistant device of the user and/or one or more additional assistant devices of the user” – Fig. 4 block 452; [0056] “the training engine 140 can utilize one or more of the training instances to train the ambient sensing ML model… ambient sensing ML model can be a neural network, for example, a convolutional model, long short-term memory (LSTM) model… that can process ambient states and/or instances of sensor data to generate one or more suggested actions that are suggested for performance”).
determining, by the Al agent, a context-based activity based on the sensor data… (see Carbune, [0022] “The ambient state can be one of a plurality of disparate ambient states (e.g., classes, categories, etc.) that may be defined with varying degrees of granularity… an ambient state may be a general cooking ambient state… a general workout ambient state, or, more particularly, a weight lifting ambient state, a running ambient state, a jogging ambient state, a walking ambient state, and/or other ambient states associated with the general workout ambient state… a vacation ambient state, and/or or other ambient states associated with the general away ambient state; and/or other ambient states defined with varying degrees of granularity”; [0088] “the system determines, based on the instance of the sensor data, an ambient state. The ambient state reflects an ambient state of the user of the assistant device and/or an environment of the user of the assistant device. The ambient state can be determined based on the instance of the sensor data” – Fig. 4 block 454; [0056] “the training engine 140 can utilize one or more of the training instances to train the ambient sensing ML model… ambient sensing ML model can be a neural network, for example, a convolutional model, long short-term memory (LSTM) model… that can process ambient states and/or instances of sensor data to generate one or more suggested actions that are suggested for performance”).
generating, by the Al agent, orchestrated guidance based on the context-based activity, wherein the orchestrated guidance includes a recommended action for performing the context- based activity; and (see Carbune, [0060] “the automated assistant 120 can cause the action suggestion engine 170 to utilize the ambient sensing ML model in generating one or more suggested actions”; [0090] “the system processes, using a trained ambient sensing ML model, the ambient state to generate one or more suggested actions that are suggested to be performed on behalf of the user” – Fig. 4 block 458; [0056] “the training engine 140 can utilize one or more of the training instances to train the ambient sensing ML model… ambient sensing ML model can be a neural network, for example, a convolutional model, long short-term memory (LSTM) model… that can process ambient states and/or instances of sensor data to generate one or more suggested actions that are suggested for performance”).
presenting the orchestrated guidance, at the assistant device (see Carbune, [0092] “the system causes a corresponding representation of one or more of the suggested actions to be provided for presentation to the user via the assistant device and/or an additional assistant device of the user” – Fig. 4 block 462) wherein presenting the orchestrated guidance includes causing presentation of a user interface element associated with the orchestrated guidance at a display of a device… (see Carbune, [0061] “can cause an indication of one or more of the suggested actions to be provided for presentation to a user. For example, a corresponding suggestion chip or selectable graphical element can be visually rendered for presentation to the user via a display of one or more of the assistant devices 106, 185. The corresponding suggestion chips or selectable graphical elements can be associated with a disparate one of the suggested actions (e.g., as disparate actions that can be performed by the automated assistant 120 as shown in FIGS. 5A and 5B) and/or associated with each of the one or more suggested actions (e.g., as a routine to be performed by the automated assistant 120)”).
Carbune does not explicitly teach in response to an indication received at a head-wearable device that an artificial intelligence (Al) agent trigger condition is present, data obtained by the head-wearable device, including image data obtained by one or more imaging devices of the head-wearable device; data obtained by the head-wearable device; display of a wrist-wearable device communicatively coupled to the head-wearable device.
However, Cella discloses AI-based learning models and teaches
in response to an indication received at a non-adaptive security module that an artificial intelligence (AI) agent trigger condition is present, (see Cella, [2378] “data collection may occur in response to a triggering condition. These triggering conditions may include, for example, expiration of a static or a dynamic predetermined interval, obtaining a value short of or in excess of a static or dynamic value, receiving an automatically generated request or instruction from the digital twin module 13420 or components thereof, interaction of an element with the respective sensor or sensors (e.g., in response to an object coming within a predetermined distance from the proximity sensor), interaction of a user with a digital twin (e.g., selection of a smart container digital twin, a sensor array digital twin, or a sensor digital twin)”; [1787] “a non-adaptive security module 12284 may be configured to, in response to detecting a specific set of conditions, trigger actions”; [0843] “an expert agent may be configured with artificial intelligence rules that determine actions”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of trigger condition, in response to determination, following and exclusion, as being disclosed and taught by Cella, in the system taught by Carbune to yield the predictable results of effectively training AI-based learning model based on tasks (see Cella, [0045] “The information may be provided to a set of Artificial Intelligence (AI)-based learning models, wherein at least one member of the set of AI-based learning models may be trained on a training data… at least one member of the set of AI-based learning models is trained on the training data set to determine, upon receiving the classification, a task to be completed for the value chain network. A robotic process automation system may be configured to execute the task to facilitate an improvement in the value chain network”).
The proposed combination of Carbune and Cella does not explicitly teach indication received at a head-wearable device; data obtained by the head-wearable device, including image data obtained by one or more imaging devices of the head-wearable device; data obtained by the head-wearable device; display of a wrist-wearable device communicatively coupled to the head-wearable device.
However, Huang discloses wrist and head wearable devices and teaches
receive instructions at a head-wearable device (see Huang, [0299] “receiving (1410) an instruction to use a camera of a head-worn wearable device to capture video data for a video stream”).
data obtained by the head-wearable device, including image data obtained by one or more imaging devices of the head-wearable device; (see Huang, [0290] “responsive to receiving the user input 1304 (FIG. 13A) or 1305 (FIG. 13B), the smart glasses 150 capture, via the imaging device 169, video data. In some embodiments, the imaging device 169 is configured to capture a field of view of the imaging device of the smart glasses 150 (e.g., imaging device 169)”; [0099] “a head-worn wearable device (in this example, a pair of smart glasses)”).
data obtained by the head-wearable device; (see Huang, [0290] “responsive to receiving the user input 1304 (FIG. 13A) or 1305 (FIG. 13B), the smart glasses 150 capture, via the imaging device 169, video data. In some embodiments, the imaging device 169 is configured to capture a field of view of the imaging device of the smart glasses 150 (e.g., imaging device 169)”; [0099] “a head-worn wearable device (in this example, a pair of smart glasses)”).
display of a wrist-wearable device communicatively coupled to the head-wearable device (see Huang, [0034] “a determination that the wrist-wearable device is communicatively coupled with smart glasses”; [0122] “indicators 127 and/or 125 can presented on the display 115 of the wrist-wearable device to provide the user with a visual indication”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of connection between head wearable device and wrist wearable device, audible data and speaker as being disclosed and taught by Huang, in the system taught by the proposed combination of Carbune and Cella to yield the predictable results of providing an improved user experience (see Cella, [0134] “in accordance with a determination that the wrist-wearable device 102 is in communication with the smart glasses 150… the wrist-wearable device 102 automatically selects the speaker to provide a user with an improved experience”).
Claims 10 and 16 incorporate substantively all the limitations of claim 1 in a method (see Carbune, [0066] “the system of the method 200 can be implemented by one or more of the assistant devices”) and device form (see Carbune, [0018] “There is a proliferation of smart, multi-sensing network connected devices (also referred to herein as assistant devices) such smart phones, tablet computers, vehicle computing systems, wearable computing devices”; [0030] “The assistant input devices 106 may include… a wearable apparatus of the user that includes a computing device ( e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device)”; [0061] “a display of one or more of the assistant devices”; [0020] “an instance of sensor data obtained via sensor(s) of one or more of the assistant devices”; [0040] “The automated assistant 120 may be implemented as, for example, computer programs running on one or more computers in one or more locations that are coupled to each other through a network”) and are rejected under the same rationale.
Regarding claim 2, the proposed combination of Carbune, Cella and Huang teaches
wherein: the context-based activity is a first context-based activity; (see Carbune, [0022] “The ambient state can be one of a plurality of disparate ambient states ( e.g., classes, categories, etc.) that may be defined with varying degrees of granularity”; [0023] “the audio data is processed, using a classifier (or an ambient noise detection ML model), to generate output indicating the audio data captures ambient noise of food sizzling, an appliance dinging or buzzing, and/or cutlery clinking on a dish. In this example, the determined ambient state may correspond to a cooking ambient state, or, more particularly, a breakfast ambient state”).
the sensor data is first sensor data; (see Carbune, [0087] “the system obtains, via one or more sensors of an assistant device of a user, an instance of sensor data. The instance of the sensor data can include, for example, audio data (e.g., audio data capturing spoken utterances, ambient noise, etc.),… any other sensor data generated by various sensors of the assistant device of the user and/or one or more additional assistant devices of the user” – audio data has been interpreted as first sensor data).
the orchestrated guidance is first orchestrated guidance; the recommended action is a first recommended action; and (see Carbune, [0060] “the automated assistant 120 can cause the action suggestion engine 170 to utilize the ambient sensing ML model in generating one or more suggested actions” – there are plurality of suggested actions since the orchestrated guidance includes a recommended action; [0106] “the automated assistant may generate an indication of a weather action, a traffic action, and a car start action for a first user associated with the assistant devices based on a cooking or breakfast ambient state”).
the instructions, when executed by one or more processors, cause the one or more processors to perform: (see Carbune, [0141] “Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform”).
in accordance with a determination of recognized patterns, providing a specific output (see Cella, [1067] “such as providing an output governing autonomous control of a system in response to the recognized condition or pattern”) that the first recommended action for performing the first context-based activity was performed, (see Carbune, [0060] “the automated assistant 120 can cause the action suggestion engine 170 to utilize the ambient sensing ML model in generating one or more suggested actions” – there are plurality of suggested actions since the orchestrated guidance includes a recommended action; [0091] “the system determines whether to automatically perform one or more of the suggested actions on behalf of the user. The system can perform one or more of the suggested actions via the assistant device of the user and/or one or more additional assistant devices of the user”) providing the Al agent second sensor data (see Carbune, [0087] “the system obtains, via one or more sensors of an assistant device of a user, an instance of sensor data. The instance of the sensor data can include, for example,… motion data (e.g., GPS signal(s), accelerometer data, etc.),… any other sensor data generated by various sensors of the assistant device of the user and/or one or more additional assistant devices of the user” – motion data has been interpreted as second sensor data; [0056] “the training engine 140 can utilize one or more of the training instances to train the ambient sensing ML model… ambient sensing ML model can be a neural network, for example, a convolutional model, long short-term memory (LSTM) model… that can process ambient states and/or instances of sensor data to generate one or more suggested actions that are suggested for performance”) obtained by the head-wearable device, including image data obtained by the one or more imaging devices of the head-wearable device, (see Huang, [0290] “responsive to receiving the user input 1304 (FIG. 13A) or 1305 (FIG. 13B), the smart glasses 150 capture, via the imaging device 169, video data. In some embodiments, the imaging device 169 is configured to capture a field of view of the imaging device of the smart glasses 150 (e.g., imaging device 169)”; [0099] “a head-worn wearable device (in this example, a pair of smart glasses)”).
determining, by the Al agent, a second context-based activity (see Carbune, [0078] “may generate sensor data… generates motion data… the motion data may correspond to an ambient sensing event, and the resulting ambient state determined based on the motion data generated by the mobile device may correspond to a workout ambient state, running ambient state, jogging ambient state, waking ambient state, and/or another ambient state determined based on the motion data”; [0056] “the training engine 140 can utilize one or more of the training instances to train the ambient sensing ML model… ambient sensing ML model can be a neural network, for example, a convolutional model, long short-term memory (LSTM) model… that can process ambient states and/or instances of sensor data to generate one or more suggested actions that are suggested for performance”) based on the second sensor data (see Carbune, [0087] “the system obtains, via one or more sensors of an assistant device of a user, an instance of sensor data. The instance of the sensor data can include, for example,… motion data (e.g., GPS signal(s), accelerometer data, etc.),… any other sensor data generated by various sensors of the assistant device of the user and/or one or more additional assistant devices of the user” – motion data has been interpreted as second sensor data) obtained by the head-wearable device, (see Huang, [0290] “responsive to receiving the user input 1304 (FIG. 13A) or 1305 (FIG. 13B), the smart glasses 150 capture, via the imaging device 169, video data. In some embodiments, the imaging device 169 is configured to capture a field of view of the imaging device of the smart glasses 150 (e.g., imaging device 169)”; [0099] “a head-worn wearable device (in this example, a pair of smart glasses)”).
generating, by the Al agent, second orchestrated guidance based on the second context-based activity, wherein the second orchestrated guidance includes a second recommended action for performing the second context-based activity, and (see Carbune, [0101] “generates motion data that captures the user walking… the automated assistant can receive one or more suggested actions as indicated by 552B1 based on processing at least the pairing data and motion data, and cause a corresponding representation of the one or more suggested actions to be presented to the user… the user can select one or more of the suggested actions for performance by the automated assistant”; [0056] “the training engine 140 can utilize one or more of the training instances to train the ambient sensing ML model… ambient sensing ML model can be a neural network, for example, a convolutional model, long short-term memory (LSTM) model… that can process ambient states and/or instances of sensor data to generate one or more suggested actions that are suggested for performance”).
presenting the second orchestrated guidance, at the automated assistant wherein presenting the second orchestrated guidance includes causing (see Carbune, Fig. 5B; [0101] “cause a corresponding representation of the one or more suggested actions to be presented to the user… the user can select one or more of the suggested actions for performance by the automated assistant”) presentation of a second user interface element associated with (see Huang, [0169] “user interfaces used in conjunction with presenting video-mode data or data for other calling modes based on whether one or more video-viewing preconditions are satisfied at a wrist-wearable device” – there are plurality of user interfaces) the second orchestrated guidance (see Carbune, Fig. 5B; [0101] “cause a corresponding representation of the one or more suggested actions to be presented to the user… the user can select one or more of the suggested actions for performance by the automated assistant”) at the display of the wrist-wearable device communicatively coupled to the head-wearable device (see Huang, [0034] “a determination that the wrist-wearable device is communicatively coupled with smart glasses”; [0122] “indicators 127 and/or 125 can presented on the display 115 of the wrist-wearable device to provide the user with a visual indication”). The motivation for the proposed combination is maintained.
Claims 11 and 17 incorporate substantively all the limitations of claim 2 in a method and device form and are rejected under the same rationale.
Regarding claim 3, the proposed combination of Carbune, Cella and Huang teaches
wherein: the context-based activity is a first context-based activity of a plurality of context-based activities determined by the Al agent (see Carbune, [0022] “The ambient state can be one of a plurality of disparate ambient states ( e.g., classes, categories, etc.) that may be defined with varying degrees of granularity”; [0023] “the audio data is processed, using a classifier (or an ambient noise detection ML model), to generate output indicating the audio data captures ambient noise of food sizzling, an appliance dinging or buzzing, and/or cutlery clinking on a dish. In this example, the determined ambient state may correspond to a cooking ambient state, or, more particularly, a breakfast ambient state” – there are plurality of ambient states; [0056] “the training engine 140 can utilize one or more of the training instances to train the ambient sensing ML model… ambient sensing ML model can be a neural network, for example, a convolutional model, long short-term memory (LSTM) model… that can process ambient states and/or instances of sensor data to generate one or more suggested actions that are suggested for performance”) based on the sensor data; (see Carbune, [0088] “the system determines, based on the instance of the sensor data, an ambient state. The ambient state reflects an ambient state of the user of the assistant device and/or an environment of the user of the assistant device. The ambient state can be determined based on the instance of the sensor data” – Fig. 4 block 454).
the orchestrated guidance includes a plurality of recommended actions for performing the plurality of context-based activities; and (see Carbune, [0060] “the automated assistant 120 can cause the action suggestion engine 170 to utilize the ambient sensing ML model in generating one or more suggested actions” – there are plurality of suggested actions since the orchestrated guidance includes recommended action; [0061] “can initiate performance of one or more of the suggested actions based on a user selection of one or more of the corresponding suggestion”; [0106] “the automated assistant may generate an indication of a weather action, a traffic action, and a car start action for a first user associated with the assistant devices based on a cooking or breakfast ambient state”).
the recommended action is a first recommended action of the plurality of recommended actions, the first recommended action being configured to perform the first context-based activity; and (see Carbune, [0060] “the automated assistant 120 can cause the action suggestion engine 170 to utilize the ambient sensing ML model in generating one or more suggested actions” – there are plurality of suggested actions since the orchestrated guidance includes recommended action; [0061] “can initiate performance of one or more of the suggested actions based on a user selection of one or more of the corresponding suggestion”; [0106] “the automated assistant may generate an indication of a weather action, a traffic action, and a car start action for a first user associated with the assistant devices based on a cooking or breakfast ambient state” – car start action has been interpreted as first recommended action).
presenting the orchestrated guidance at the assistant device (see Carbune, [0092] “the system causes a corresponding representation of one or more of the suggested actions to be provided for presentation to the user via the assistant device and/or an additional assistant device of the user” – Fig. 4 block 462) includes presenting at least the first recommended action of the plurality of recommended actions (see Carbune, [0097] “the automated assistant can receive one or more suggested actions as indicated by 552A1 based on processing at least the audio data, and cause a corresponding representation of the one or more suggested actions to be presented to the user”; [0099] “the automated assistant can automatically perform a weather action and a traffic action based on the user making and/or eating breakfast”; Fig. 5A – 552A2, 552A3, 552A4 ad 552A5). The motivation for the proposed combination is maintained.
Claims 12 and 18 incorporate substantively all the limitations of claim 3 in a method and device form and are rejected under the same rationale.
Regarding claim 4, the proposed combination of Carbune, Cella and Huang teaches
wherein: generating the orchestrated guidance includes (see Carbune, [0060] “the automated assistant 120 can cause the action suggestion engine 170 to utilize the ambient sensing ML model in generating one or more suggested actions” – there are plurality of suggested actions since the orchestrated guidance includes recommended action; [0106] “the automated assistant may generate an indication of a weather action, a traffic action, and a car start action for a first user associated with the assistant devices based on a cooking or breakfast ambient state”) determining a subset of the plurality of recommended actions (see Carbune, [0100] “a first selectable element 560A1 of "Yes" indicates the user likes the routine… selection of a second selectable element 560A2 of "No" indicates the user does not like the routine… selection of a third selectable element 560A2A of "No weather" indicates the user does not like the weather action of the routine… selection of a fourth selectable element 560A2B of "No traffic" indicates the user does not like the traffic action of the routine… selection of a fifth selectable element 560A2C of "No car start" indicates the user does not like the car start action of the routine”) for performing the first context-based activity; and (see Carbune, [0060] “the automated assistant 120 can cause the action suggestion engine 170 to utilize the ambient sensing ML model in generating one or more suggested actions” – there are plurality of suggested actions since the orchestrated guidance includes recommended action; [0106] “the automated assistant may generate an indication of a weather action, a traffic action, and a car start action for a first user associated with the assistant devices based on a cooking or breakfast ambient state”).
presenting the orchestrated guidance at the assistant device (see Carbune, [0092] “the system causes a corresponding representation of one or more of the suggested actions to be provided for presentation to the user via the assistant device and/or an additional assistant device of the user” – Fig. 4 block 462) includes presenting at least the first recommended action of the plurality of recommended actions and (see Carbune, [0097] “the automated assistant can receive one or more suggested actions as indicated by 552A1 based on processing at least the audio data, and cause a corresponding representation of the one or more suggested actions to be presented to the user”; [0099] “the automated assistant can automatically perform a weather action and a traffic action based on the user making and/or eating breakfast”; Fig. 5A – 552A2, 552A3, 552A4 ad 552A5) the subset of the plurality of recommended actions (see Carbune, [0100] “a first selectable element 560A1 of "Yes" indicates the user likes the routine… selection of a second selectable element 560A2 of "No" indicates the user does not like the routine… selection of a third selectable element 560A2A of "No weather" indicates the user does not like the weather action of the routine… selection of a fourth selectable element 560A2B of "No traffic" indicates the user does not like the traffic action of the routine… selection of a fifth selectable element 560A2C of "No car start" indicates the user does not like the car start action of the routine”) for performing the first context-based activity (see Carbune, [0060] “the automated assistant 120 can cause the action suggestion engine 170 to utilize the ambient sensing ML model in generating one or more suggested actions” – there are plurality of suggested actions since the orchestrated guidance includes recommended action; [0106] “the automated assistant may generate an indication of a weather action, a traffic action, and a car start action for a first user associated with the assistant devices based on a cooking or breakfast ambient state”).
Claims 13 and 19 incorporate substantively all the limitations of claim 4 in a method and device form and are rejected under the same rationale.
Regarding claim 5, the proposed combination of Carbune, Cella and Huang teaches
wherein: generating the orchestrated guidance includes (see Carbune, [0060] “the automated assistant 120 can cause the action suggestion engine 170 to utilize the ambient sensing ML model in generating one or more suggested actions” – there are plurality of suggested actions since the orchestrated guidance includes recommended action; [0106] “the automated assistant may generate an indication of a weather action, a traffic action, and a car start action for a first user associated with the assistant devices based on a cooking or breakfast ambient state”) determining a sequence of tasks (see Cella, [3035] “the directed action request includes an executable sequence of tasks for the secondary system 21200 to perform”) context-based activities of the plurality of context-based activities to be performed, (see Carbune, [0022] “The ambient state can be one of a plurality of disparate ambient states (e.g., classes, categories, etc.) that may be defined with varying degrees of granularity… an ambient state may be a general cooking ambient state… a general workout ambient state, or, more particularly, a weight lifting ambient state, a running ambient state, a jogging ambient state, a walking ambient state, and/or other ambient states associated with the general workout ambient state… a vacation ambient state, and/or or other ambient states associated with the general away ambient state; and/or other ambient states defined with varying degrees of granularity” – there are plurality of ambient states) including a second context- based activity (see Carbune, [0078] “may generate sensor data… generates motion data… the motion data may correspond to an ambient sensing event, and the resulting ambient state determined based on the motion data generated by the mobile device may correspond to a workout ambient state, running ambient state, jogging ambient state, waking ambient state, and/or another ambient state determined based on the motion data”) one task to follow another task (see Cella, [2255] “may include a cargo unloading task… followed by a self-stacking storage task”) the first context-based activity; and (see Carbune, [0022] “The ambient state can be one of a plurality of disparate ambient states ( e.g., classes, categories, etc.) that may be defined with varying degrees of granularity”; [0023] “the audio data is processed, using a classifier (or an ambient noise detection ML model), to generate output indicating the audio data captures ambient noise of food sizzling, an appliance dinging or buzzing, and/or cutlery clinking on a dish. In this example, the determined ambient state may correspond to a cooking ambient state, or, more particularly, a breakfast ambient state” – there are plurality of ambient states).
presenting the orchestrated guidance the assistant device (see Carbune, [0092] “the system causes a corresponding representation of one or more of the suggested actions to be provided for presentation to the user via the assistant device and/or an additional assistant device of the user” – Fig. 4 block 462) includes presenting at least the first recommended action (see Carbune, [0097] “the automated assistant can receive one or more suggested actions as indicated by 552A1 based on processing at least the audio data, and cause a corresponding representation of the one or more suggested actions to be presented to the user”; [0099] “the automated assistant can automatically perform a weather action and a traffic action based on the user making and/or eating breakfast”; Fig. 5A – 552A2, 552A3, 552A4 ad 552A5) and the second recommended action of the plurality of recommended actions at the automated assistant (see Carbune, Fig. 5B; [0101] “cause a corresponding representation of the one or more suggested actions to be presented to the user… the user can select one or more of the suggested actions for performance by the automated assistant”) for performing the plurality of context-based activities (see Carbune, [0060] “the automated assistant 120 can cause the action suggestion engine 170 to utilize the ambient sensing ML model in generating one or more suggested actions” – there are plurality of suggested actions since the orchestrated guidance includes recommended action; [0061] “can initiate performance of one or more of the suggested actions based on a user selection of one or more of the corresponding suggestion”; [0106] “the automated assistant may generate an indication of a weather action, a traffic action, and a car start action for a first user associated with the assistant devices based on a cooking or breakfast ambient state”; [0101] “generates motion data that captures the user walking… the automated assistant can receive one or more suggested actions as indicated by 552B1 based on processing at least the pairing data and motion data, and cause a corresponding representation of the one or more suggested actions to be presented to the user… the user can select one or more of the suggested actions for performance by the automated assistant”). The motivation for the proposed combination is maintained.
Claims 14 and 20 incorporate substantively all the limitations of claim 5 in a method and device form and are rejected under the same rationale.
Regarding claim 9, the proposed combination of Carbune, Cella and Huang teaches
wherein the context-based activity is to be performed at a physical activity (see Carbune, [0022] “The ambient state can be one of a plurality of disparate ambient states (e.g., classes, categories, etc.) that may be defined with varying degrees of granularity… an ambient state may be a general cooking ambient state… a general workout ambient state, or, more particularly, a weight lifting ambient state, a running ambient state, a jogging ambient state, a walking ambient state, and/or other ambient states associated with the general workout ambient state… a vacation ambient state, and/or or other ambient states associated with the general away ambient state; and/or other ambient states defined with varying degrees of granularity”; [0088] “the system determines, based on the instance of the sensor data, an ambient state. The ambient state reflects an ambient state of the user of the assistant device and/or an environment of the user of the assistant device. The ambient state can be determined based on the instance of the sensor data” – running, jogging, walking are physical activities).
Regarding claim 21, the proposed combination of Carbune, Cella and Huang teaches
wherein: the head-wearable device (see Huang, [0177] “the smart glasses 150 (or other head-worn wearable device) is provided”) does not comprise (see Cella, [2438] “does not include”) a display; and (see Huang, [0135] “an integrated display 155, ceasing to cause presentation of the video data via the display 115 of the wrist-wearable device”).
presenting the orchestrated guidance further includes at the assistant device (see Carbune, [0092] “the system causes a corresponding representation of one or more of the suggested actions to be provided for presentation to the user via the assistant device and/or an additional assistant device of the user” – Fig. 4 block 462) causing presentation of audible guidance associated with the data (see Huang, [0120] “a head-worn wearable device that includes a speaker, the wrist-wearable device can instead user the speaker of the head-worn wearable device for presentation of the audio data”) the orchestrated guidance device (see Carbune, [0092] “the system causes a corresponding representation of one or more of the suggested actions to be provided for presentation to the user via the assistant device and/or an additional assistant device of the user” – Fig. 4 block 462) at a speaker of the head-wearable device (see Huang, [0120] “a head-worn wearable device that includes a speaker, the wrist-wearable device can instead user the speaker of the head-worn wearable device for presentation of the audio data”). The motivation for the proposed combination is maintained.
Claims 6 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Carbune, Cella and Huang in view of Ziemianska et al. (US 2014/0171132 A1, hereinafter “Ziemianska”).
Regarding claim 6, the proposed combination of Carbune, Cella and Huang teaches
wherein the instructions, when executed by one or more processors, cause the one or more processors to perform: (see Carbune, [0141] “Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform”).
in response to a user input selecting the recommended action for performing the context-based activity, (see Carbune, [0097] “the user can select one or more of the suggested actions for performance by the automated assistant”) causing the head-wearable device to… (see Huang, [0177] “the smart glasses 150 (or other head-worn wearable device) is provided”) the head-wearable device… (see Huang, [0177] “the smart glasses 150 (or other head-worn wearable device) is provided”).
in response to an indication that participation in the context-based activity ceased: (see Carbune, [0050] “the one or more temporally corresponding actions can be considered to temporally correspond to the ambient state if they are detected within a threshold duration of time of the instance of the sensor data being captured by one or more of the assistant devices”).
… the head-wearable device… (see Huang, [0177] “the smart glasses 150 (or other head-worn wearable device) is provided”).
… by the AI agent,… (see Carbune, [0025] “The example environment includes a plurality of assistant input devices… one or more cloud-based automated assistant components… one or more assistant non-input devices… The assistant input devices 106 and the assistant non-input device 185 of FIG. 1 may also be referred to collectively herein as "assistant devices”; [0056] “the training engine 140 can utilize one or more of the training instances to train the ambient sensing ML model… ambient sensing ML model can be a neural network, for example, a convolutional model, long short-term memory (LSTM) model… that can process ambient states and/or instances of sensor data to generate one or more suggested actions that are suggested for performance”) the head-wearable device… (see Huang, [0177] “the smart glasses 150 (or other head-worn wearable device) is provided”) the head-wearable device… (see Huang, [0177] “the smart glasses 150 (or other head-worn wearable device) is provided”).
The proposed combination of Carbune, Cella and Huang does not explicitly teach initiate a do-not-disturb mode, wherein, while in the do-not-disturb mode, the head-wearable device suppresses, at least, received notifications; and causing the head-wearable device to cease the do-not-disturb mode, generating, a notification summary based on the notifications received while the head-wearable device was in the do-not-disturb mode, and presenting the notification summary at the head-wearable device.
However, Ziemianska discloses sensed changes in user activity and teaches
initiate a do-not-disturb mode, wherein, while in the do-not-disturb mode, (see Ziemianska, [0021] “device 100 (which may be a smartphone) activates the Do Not Disturb function (or state) of the device”) the device suppresses, at least, received notifications; and (see Ziemianska, [0022] “The device enters the DND state by suppressing alarms and notifications which would otherwise activate”)
causing… the device to cease the do-not-disturb mode, (see Ziemianska, [0022] “the DND state is deselected and, any notifications received during the DND period are resent (at 230) to alert the user of the suppressed notifications”; [0027] “to detect active use of the device, and, in response, automatically deselect the DND state”).
generating,… a notification summary based on the notifications received while the device was in the do-not-disturb mode, and presenting the notification summary at the smartphone (see Ziemianska, [0024] “the process continues to box 228 (see FIG. 2B) at which point the DND state is deselected and, any notifications received during the DND period are resent (at 230) to alert the user of the suppressed notifications”; [0026] “permits a user to briefly pick up his or her smartphone to check the time of day (or other indications) without triggering notifications …the device automatically returns it to an active state (DND off) and triggers notifications” – since it is an alert to the user it is presented to the user).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of do not disturb functionality as being disclosed and taught by Ziemianska, in the system taught by the proposed combination of Carbune, Cella and Huang to yield the predictable results of effectively configuring several options of Do Not Disturb (see Ziemianska, [0009]-[0011] “The user may configure the "Do not disturb" feature to function on a predefined schedule, or may simply turn it on and off as needed. The user may also specify certain contacts sometime designated as "VIPs"-who are allowed to get through to the user even if the phone is in "do not disturb" mode… Various options may allow the Do Not Disturb settings on a smartphone to be further customized. For example, an option for "Repeated Calls" may allow activation of a mode wherein whenever someone calls back a second time from the same number within a certain time interval, the second call will not be silenced”).
Claim 15 incorporates substantively all the limitations of claim 6 in a method form and is rejected under the same rationale.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Carbune, Cella and Huang further in view of Krishna et al. (US 2025/0103642 A1, hereinafter “Krishna”).
Regarding claim 7, the proposed combination of Carbune, Cella and Huang teaches
wherein the instructions, when executed by one or more processors, cause the one or more processors to perform: (see Carbune, [0141] “Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform”).
in response to a user input selecting the recommended action for performing the context- based activity, performing, by the AI agent,… (see Carbune, [0092] “the system causes a corresponding representation of one or more of the suggested actions to be provided for presentation to the user via the assistant device and/or an additional assistant device of the user… the system determines whether a user selection of the corresponding representation of one or more of the suggested actions is received from the user. The user selection can be, for example, touch input directed to a display of the assistant device, spoken input received by microphone(s) of the assistant device, etc.”; [0056] “the training engine 140 can utilize one or more of the training instances to train the ambient sensing ML model… ambient sensing ML model can be a neural network, for example, a convolutional model, long short-term memory (LSTM) model… that can process ambient states and/or instances of sensor data to generate one or more suggested actions that are suggested for performance”) based on the recommended action; (see Carbune, [0092] “the system causes a corresponding representation of one or more of the suggested actions”).
… the head-wearable device (see Huang, [0177] “the smart glasses 150 (or other head-worn wearable device) is provided”).
The proposed combination of Carbune, Cella and Huang does not explicitly teach a search based on the recommended action; determining a task to perform based on the search; and presenting the task at the wearable device.
However, Krishna discloses visual search interface and teaches
a search to determine predicted actions (see Krishna, [0101] “The plurality of application suggestions 520 may be determined based on processing the web page 518, the selected text 510, and/or the contents of the search results interface to determine predicted actions associated with the processed data”).
determining a task to perform based on the search; and (see Krishna, [0101] “The plurality of application suggestions 520 may be determined based on processing the web page 518, the selected text 510, and/or the contents of the search results interface to determine predicted actions associated with the processed data. For example, a topic, entity, and/or task may be determined to be associated with the processed data”).
presenting the task at display on the computing device (see Krishna, [0063] “the display data and/or the visual search data is associated with a particular topic, a particular entity, and/or a particular task”; [0033] “The visual search interface can generate display data descriptive of content provided for display on the computing device”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of searching, determining tasks and presenting the task as being disclosed and taught by Krishna, in the system taught by the proposed combination of Carbune, Cella and Huang to yield the predictable results of providing improved computational efficiency and improvements in the functioning of a computer system (see Krishna, [0046] “Another example of technical effect and benefit relates to improved computational efficiency and improvements in the functioning of a computing system. For example, the systems and methods disclosed herein can leverage a visual search interface in the operating system to reduce the inputs and operations necessitated for performing particular data processing tasks across different applications”).
Citation Of Relevant Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US Publication No. US 2018/0150131 A1 (Ranieri et al.) teaches deploying at least two measurement devices that are linked, preferably in a wireless manner, through a communication channel and integrated into a wearable tracking system such that one measurement device provides a reference frame in which the other device is positioned and localized through said communication link, a more effective system may be realized.
US Publication No. US 2019/0000374 A1 (Johnson et al.) teaches a patient monitor configured to wirelessly communicate with at least one of the head mounted device and the wrist-worn device.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAISHALI SHAH whose telephone number is (571)272-8532. The examiner can normally be reached Monday - Friday (7:30 AM to 4:00 PM).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, AJAY BHATIA can be reached at (571)272-3906. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VAISHALI SHAH/Primary Examiner, Art Unit 2156
Prosecution Insights