DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed on 12/10/2025, has been received and made of record. In response to the Non-Final Office Action, dated on 07/10/2025. Claims 1-6 and 8-20 are pending in the current application. Claim 7 has been cancelled. Claims 1-6 and 8-20 have been amended.
Terminal Disclaimer
The terminal disclaimer filed on 12/10/2025 disclaiming the terminal portion of any patent granted on this application which would extend beyond the expiration date of the full statutory term of any patent granted on pending reference application numbers 18/425,527 and 18/425,547 has been reviewed and is accepted. The Terminal Disclaimer has been recorded.
Response to Arguments
Applicant’s arguments filed on 12/10/2025 have been fully considered.
In the Arguments/Remarks:
Re: Rejection of the Claims Under Double Patenting
Rejection of claims 1-20 under Double Patenting has been withdrawn in view of the Terminal Disclaimer filed 12/10/2025.
Re: Rejection of the Claims Under 35 U.S.C. 101
Rejection of claims 1-20 under 35 U.S.C. 101 has been withdrawn in view of applicant’s amendments.
Re: Rejection of the Claims Under 35 U.S.C. 112(b)
Rejection of claims 7-10 under 35 U.S.C. 112(b) has been withdrawn in view of applicant’s amendments.
Re: Rejection of the Claims Under 35 U.S.C. 103
Applicant argues beginning on page 7 of the Remarks, that Cupersmith fails to teach “after the initiating of the task, detecting by the robot, the human is waiting for the robot to complete the task”. Examiner respectfully disagrees. Examiner notes that Cupersmith states in paragraph 209 “For example, the patron may have submitted an order for a drink and the robot management system 400 may provide status information on the order (e.g., via the player app). The status information may include current task status (e.g., awaiting drink to be made, carrying drink to patron), or current location information of any robot 300 currently assigned to that task, thereby allowing the patron to see where their robot 300 is currently located, watch the robot movement (e.g., on a floor map), or the like.” Examiner states that the robot management system sends status information of the robot to the user while the user waits. Examiner submits that the status updates regarding the robot executing the task of making/bringing a drink is the robot system acknowledging and detecting that the human is waiting for the robot to complete the task. Furthermore, examiner submits that Cupersmith discloses in paragraph 253 “the method 1100 can include a robot escort service. With robot escorting, the kiosk robot may be configured to escort the user to a particular destination. For example, the user may request an escort to the nearest restroom, to a particular type of EGM 104 or gaming table, to a sports book or poker room, to their car or valet attendant (e.g., as a security feature after a large win), or the like. The kiosk robot may, accordingly, identify an escort destination location for this request and plot a course between the current location of the kiosk robot and the escort destination location (e.g., based upon map analysis, avoiding static obstacles). The kiosk robot may then navigate to the escort destination location, allowing the user to follow the robot to their desired destination. Upon arrival at the destination location, the kiosk robot may terminate the kiosk session. In some embodiments, the kiosk robot may be configured to request, from the user, whether to await the user for a return trip (e.g., through audible interaction, kiosk GUI, or the like). For example, the user may wish to return to their original location after using the bathroom. Accordingly, the kiosk robot may be configured to wait a predetermined amount of time and detect the return of the user to the robot (e.g., via video analysis).” Examiner submits that the robot is to wait and escort the patron and can detect the return/presence of the patron using video analysis. Examiner notes that robot can detect the return of the patron and once confirmed that the patron has returned and is waiting for a return trip to the patron’s previous location the robot can then execute the task of leading the patron to their previous location.
Examiner submits under the broadest reasonable interpretation (BRI) of the argued limitation that Cupersmith teaches or suggest the argued limitation by the sections provided here and shown below in the newest rejection, therefore applicant’s arguments are unpersuasive. The same reasoning as applied to the independent claim also apply to its corresponding dependent claims.
Applicant argues, beginning on page 8 of the Remarks, that neither Cupersmith nor Hausman teaches an “interim interaction”. Applicant has amended the argued limitation found in independent claim 1 to further include “wherein the interim interaction does not contribute to the completing of the task.” However, Examiner respectfully disagrees that neither Cupersmith nor Hausman teaches an “interim interaction”. Examiner notes that Hausman discloses in paragraph 63 “The explanation 204A can be an explanation generated based on processing a prior LLM prompt, also based on the FF NL input, in a prior pass and utilizing the LLM 150. For example, the prior LLM prompt could be “explain how you would bring me a snack from the table”, and the explanation 204A can be generated based on the highest probability decoding from the prior LLM output. For instance, the explanation 204A can be “I would find the table, then find a snack on the table, then bring it to you”. The explanation 204A can be prepended to the LLM prompt 205A, replace term(s) of the FF NL input 105 in the LLM prompt 205A, or otherwise incorporated into the LLM prompt 205A.”. Examiner submits that this is being seen as an interim interaction which does not contribute to the completing of the task. The robot is responding to the dialogue of the user of how it would grab a snack for the user by answering it with the steps the robot would take to complete the task. The process of responding to the user of how it would complete the task does not affect the completion of the task. Examiner submits under the broadest reasonable interpretation (BRI) that the cited section of Hausman teaches or suggest “initiating, by the robot, an interim interaction with the human, wherein the interim interaction is based at least in part on the response from the LLM and wherein the interaction does not contribute to the completing of the task”.
Examiner submits under the broadest reasonable interpretation (BRI) of the argued limitation that the combination of Cupersmith and Hausman teaches or suggest the argued limitation by the sections provided here and shown below in the newest rejection, therefore applicant’s arguments are unpersuasive. The same reasoning as applied to the independent claim also apply to its corresponding dependent claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-6 and 8-20 are rejected under 35 U.S.C. 103 as being unpatentable over Cupersmith (US 2021/0304559 A1) in view of Hausman (US 2023/0311335 A1).
Regarding claim 1, Cupersmith teaches a non-transitory processor-readable storage medium, storing data and processor-executable instructions that, when executed by one or more processors communicatively coupled to the storage media, cause the one or more processors to perform a method of operation of a robotic system including a robot and an interface, the method comprising: operating, by the robotic system, the robot in an environment, the environment comprising a human [(see at least paragraph 38) “robotic service systems and methods, including various robotic devices (“robots”) configured for use in various service capacities. In example embodiments, a robot management system server is configured to manage a fleet of service robots that are deployed within a casino property, also referred to herein as an “operations venue” or just “venue” (e.g., gaming floor, hotel, lobbies, or such). The robot management system includes a robot management system server that wirelessly connects with each of the service robots and provides centralized task scheduling and assignment.”]
Cupersmith teaches initiating, by the robot, a task; after the initiating of the task, detecting, by the robot, the human is waiting for the robot to complete the task; [(see at least paragraph 209) “the patron may be able to view a current status of a robot 300 currently assigned to a task for that patron. For example, the patron may have submitted an order for a drink and the robot management system 400 may provide status information on the order (e.g., via the player app). The status information may include current task status (e.g., awaiting drink to be made, carrying drink to patron), or current location information of any robot 300 currently assigned to that task, thereby allowing the patron to see where their robot 300 is currently located, watch the robot movement (e.g., on a floor map), or the like. In some embodiments, the patron or an administrator may be able to view current camera data from the robot 300 or take control of the robot 300.”]
Cupersmith does not explicitly teach a large language model (LLM); in response to detecting the human is waiting for the robot to complete the task, sending, by the interface, a query to the LLM; receiving, by the interface, a response from the LLM, the response in reply to the query; and initiating, by the robot, an interim interaction with the human, wherein the interim interaction is based at least in part on the response from the LLM.
However, Hausman teaches a large language model (LLM) [(see at least paragraph 3) “For example, large language models (LLMs) have been developed that are trained on massive amounts of data and are able to be utilized to robustly process a wide range of NL inputs and generate corresponding LM output that reflects corresponding NL content that is accurate and responsive to the NL input. An LLM can include at least hundreds of millions of parameters and can often include at least billions of parameters, such as one hundred billion or more parameters. An LLM can, for example, be a sequence-to-sequence model, Transformer-based, and/or include an encoder and/or a decoder. One non-limiting example of an LLM is GOOGLE'S Pathways Language Model (PaLM). Another non-limiting example of an LLM is GOOGLE'S Language Model for Dialogue Applications (LaMDA).”]
Hausman teaches in response to detecting the human is waiting for the robot to complete the task, sending, by the interface, a query to the LLM; [(see at least paragraph 60) “the LLM engine 130 generates an LLM prompt 205A based on the FF NL input 105 (“bring me a snack from the table”). The LLM engine 130 can generate the LLM prompt 205A such that it conforms strictly to the FF NL input 105 or can generate the LLM prompt 205A such that it is based on, but does not strictly conform to, the FF NL input 105. For example, as illustrated by LLM prompt 205A1, a non-limiting example of LLM prompt 205A, the LLM prompt can be “How would you bring me a snack from the table?”] receiving, by the interface, a response from the LLM, the response in reply to the query; [(see at least paragraph 8) “The generated LLM prompt can be processed, using the LLM, to generate LLM output that models a probability distribution, over candidate word compositions, that is dependent on the instruction. Continuing with the working example, the highest probability decoding of the LLM output can be, for example, “use a vacuum”. However, implementations disclosed herein do not simply blindly utilize the probability distribution of the LLM output in determining how to control a robot. Rather, implementations leverage the probability distribution of the LLM output, while also considering robotic skills that are actually performable by the robot, such as tens of, hundreds of, or thousands of pre-trained robotic skills”] and initiating, by the robot, an interim interaction with the human, wherein the interim interaction is based at least in part on the response from the LLM and wherein the interim interaction does not contribute to the completing of the task. [(see at least paragraph 63) “The explanation 204A can be an explanation generated based on processing a prior LLM prompt, also based on the FF NL input, in a prior pass and utilizing the LLM 150. For example, the prior LLM prompt could be “explain how you would bring me a snack from the table”, and the explanation 204A can be generated based on the highest probability decoding from the prior LLM output. For instance, the explanation 204A can be “I would find the table, then find a snack on the table, then bring it to you”. The explanation 204A can be prepended to the LLM prompt 205A, replace term(s) of the FF NL input 105 in the LLM prompt 205A, or otherwise incorporated into the LLM prompt 205A.”] Examiner notes that this is being interpreted as an interim interaction which does not contribute to the completing of the task. The robot is responding to the dialogue of the user of how it would grab a snack for the user by answering it with the steps the robot would take to complete the task. The process of responding to the user of how it would complete the task does not affect the completion of the task.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Cupersmith to incorporate the teachings of Hausman of a large language model (LLM) and in response to detecting the human is waiting for the robot to complete the task, sending, by the interface, a query to the LLM, receiving, by the interface, a response from the LLM, the response in reply to the query and and initiating, by the robot, an interim interaction with the human, wherein the interim interaction is based at least in part on the response from the LLM in order to perform certain tasks in response to various free-form natural language inputs of a user attempting to control the robot. [(Hausman 2)]
Regarding claim 2, In view of the above combination of references, Cupersmith further teaches wherein the initiating, by the robot, a task includes initiating at least one of an action, a generation of an action plan, a motion, a generation of a motion plan, a simulation, a calculation, a loading of a capability, a reception and/or a response to instructions, and an identification of an object in the environment. [(see at least paragraph 120) “the robot 300 may include a robotic arm (not shown) that can be configured to interact with objects in the nearby environment. The robotic arm can include several arm segments, joints, and motors that are configured to enable the robot arm to articulate in various degrees of freedom. The robotic arm includes at least one end effector (or “manipulator”) configured to be used to perform various tasks, such as pressing buttons, grabbing and releasing objects, housing a sensor (e.g., RFID reader, camera), or the like. In some embodiments, the robot 300 may be configured to use the robot arm during maintenance operations, for example, to depress mechanical buttons 236 or touch touchscreen components of a gaming device 104, insert or collect tickets to test bill validators 234, ticket printers 222, or ticket readers 224. In some embodiments, the robot 300 may be configured to use the robot arm to load or unload delivery items from onboard storage (e.g., food and beverage, coat check). In some embodiments, the robot 300 may be configured to use the robotic arm to collect and deposit trash items into trash receptacles, empty ash trays, or collect dirty dishes”]
Regarding claim 3, In view of the above combination of references, Cupersmith further teaches wherein the detecting, by the robot, the human is waiting for the robot to complete the task includes detecting autonomously, by the robot, the human is waiting for the robot to complete the task. [(see at least paragraph 209) “For example, the patron may have submitted an order for a drink and the robot management system 400 may provide status information on the order (e.g., via the player app). The status information may include current task status (e.g., awaiting drink to be made, carrying drink to patron), or current location information of any robot 300 currently assigned to that task, thereby allowing the patron to see where their robot 300 is currently located, watch the robot movement (e.g., on a floor map), or the like”]
Regarding claim 4, In view of the above combination of references, Cupersmith further teaches wherein the detecting, by the robot, the human is waiting for the robot to complete the task includes detecting a timer has expired. [(see at least paragraph 280) “In some embodiments, RMS server 106 or robot 900 may automatically initiate a delivery request (e.g., periodically, according to a particular schedule, at the start of an operational shift, according to a predefined schedule (e.g., as related to a loyalty tier of a player), based upon one or more timers, and the like).”]
Regarding claim 5, In view of the above combination of references, Cupersmith further teaches wherein the detecting, by the robot, the human is waiting for the robot to complete the task includes detecting a duration of the task exceeds an expected duration. [(see at least paragraph 270) “In some embodiments, the operator assistance experience may be automatically initiated when a patron interaction session with a robot 300 has exceeded a predetermined amount of time. For example, some patrons may have difficulties interacting with the robot 300, and thus may become frustrated and take up an extensive amount of interaction time”]
Regarding claim 6, In view of the above combination of references, Cupersmith further teaches wherein the detecting a duration of the task exceeds an expected duration includes determining an expected duration based at least in part on historical data. [(see at least paragraph 270) “In another example, the RMS 400 may automatically trigger a cordiality visit of a robot 300 to a highly regarded patron upon detection of that patron at the venue 600, and that cordiality visit may include an operator assistance experience. In some embodiments, the operator assistance experience may be automatically initiated when a patron interaction session with a robot 300 has exceeded a predetermined amount of time. For example, some patrons may have difficulties interacting with the robot 300, and thus may become frustrated and take up an extensive amount of interaction time. As such, automatic initiation of the operator assist experience may allow the operator 420 a chance to guide the patron through their interaction with the robot 300 or otherwise be able to assist in the patron's underlying issues or requests.”]
Regarding claim 8, In view of the above combination of references, Cupersmith further teaches the robot comprising a visual sensor, wherein detecting the human is waiting for the robot to complete the task includes detecting the human is waiting for the robot to complete the task based at least in part on data from the visual sensor. [(see at least paragraphs 99, 209) As in 99 “The head unit 302 also includes a camera device 360 configured to capture digital images or video near the robot 300, which may be used for navigation functions, human detection, patron recognition, or other various use cases described herein. The body module 310, in this example, includes a bill/ticket/card reader 362 and a ticket/card printer 364, which may be similar to the bill validator 234, ticket reader 224, and ticket printer 222 shown in FIG. 2A.” As in 209 “The status information may include current task status (e.g., awaiting drink to be made, carrying drink to patron)”]
Regarding claim 9, In view of the above combination of references, Cupersmith further teaches wherein detecting the human is waiting for the robot to complete the task based at least in part on data from the visual sensor includes: scanning the environment, by the visual sensors, to generate sensor data; and analyzing, by the sensor data processor, the sensor data to detect the human is waiting for the robot to complete the task based at least in part on the sensor data. [(see at least paragraphs 99, 209) As in 99 “The head unit 302 also includes a camera device 360 configured to capture digital images or video near the robot 300, which may be used for navigation functions, human detection, patron recognition, or other various use cases described herein. The body module 310, in this example, includes a bill/ticket/card reader 362 and a ticket/card printer 364, which may be similar to the bill validator 234, ticket reader 224, and ticket printer 222 shown in FIG. 2A.” As in 209 “The status information may include current task status (e.g., awaiting drink to be made, carrying drink to patron)”]
Regarding claim 10, In view of the above combination of references, Cupersmith further teaches the robot comprising a sound sensor, wherein the detecting the human is waiting for the robot to complete the task includes detecting the human is waiting for the robot to complete the task based at least in part on data from the sound sensor. [(see at least paragraph 239) “Upon activation of the robot 300 (e.g., by receiving a remote task, via physical interaction with the device, by keyword voice activation from a nearby player), the robot 300 may display the face waking up and looking alert, or otherwise changing away from the sleeping face. The robot 300 may display a moving face or animation while the robot 300 is moving to a destination (e.g., bobbing back and forth, sweating, or the like). The robot 300 may display facial expressions and lip movements while interacting with a player”]
Regarding claim 11, Modified Cupersmith has all of the elements of claim 1 as discussed above.
Cupersmith does not explicitly teach wherein the sending, by the interface, a query to the LLM includes sending a query to a chatbot.
However, Hausman teaches wherein the sending, by the interface, a query to the LLM includes sending a query to a chatbot. [(see at least paragraphs 27-31) As in 27 “Accordingly, implementations leverage an LLM to provide task-grounding to determine useful actions for a high-level goal and leverage affordance functions (e.g., learned function(s)) to provide world-grounding to determine what action(s) are actually possible to execute in achieving the high-level goal. Some of those implementations utilize reinforcement learning (RL) as a way to learn language-conditioned value functions that provide affordances of what is possible in the real world.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Cupersmith to incorporate the teachings of Hausman of wherein the sending, by the interface, a query to the LLM includes sending a query to a chatbot in order to learn language-conditioned value functions that provide affordances of what is possible in the real world. [(Hausman 27)]
Regarding claim 12, Modified Cupersmith has all of the elements of claim 1 as discussed above.
Cupersmith does not explicitly teach wherein the sending, by the interface, a query to the LLM includes sending a query to an LLM in the robotic system.
However, Hausman teaches wherein the sending, by the interface, a query to the LLM includes sending a query to an LLM in the robotic system. [(see at least paragraphs 33-38) As in 38 “the corresponding policy is executed by the agent and the LLM query is amended to include .sub.π (the language description of the selected skill), and the process is run again until a termination token (e.g., “done”) is chosen. This process is described in Algorithm 1, provided below. These two mirrored processes together lead to a probabilistic interpretation, where the LLM provides probabilities of a skill being useful for the high-level instruction and the affordances provide probabilities of successfully executing each skill. Combining these two probabilities together provides a probability that this skill furthers the execution of the high-level instruction commanded by the user.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Cupersmith to incorporate the teachings of Hausman of wherein the sending, by the interface, a query to the LLM includes sending a query to an LLM in the robotic system in order to inform LMs that the high-level instruction should be broken down into sequences of available low-level skills. [(Hausman 33)]
Regarding claim 13, Modified Cupersmith has all of the elements of claim 12 as discussed above.
Cupersmith does not explicitly teach wherein the sending a query to an LLM in the robotic system includes sending a query to an LLM onboard the robot.
However, Hausman teaches wherein the sending a query to an LLM in the robotic system includes sending a query to an LLM onboard the robot. [(see at least Fig.3, paragraphs 83-90, 103) As in 85 “the system processes, using an LLM, an LLM prompt that is based on the FF NL instruction, to generate LLM output. Block 354 optionally includes sub-block 354A and/or sub-block 354B. At sub-block 354A, the system includes scene descriptor(s in the LLM prompt. At sub-block 354B, the system generates an explanation and includes the explanation in the LLM prompt.” As in 86 “the system generates, based on the LLM output of block 354 and a corresponding NL skill description for each of multiple candidate robotic skills, a corresponding task-grounding measure. Block 356 optionally includes sub-block 356A, in which the system generates the task-grounding measure, of a candidate robotic skill, based on a probability of the NL skill description as reflected in a probability distribution of the LLM output.” As in 103 “The robot control system 560 may be implemented in one or more processors, such as a CPU, GPU, and/or other controller(s) of the robot 520. In some implementations, the robot 520 may comprise a “brain box” that may include all or aspects of the control system 560. For example, the brain box may provide real time bursts of data to the operational components 540a-n, with each of the real time bursts comprising a set of one or more control commands that dictate, inter alia, the parameters of motion (if any) for each of one or more of the operational components 540a-n. In some implementations, the robot control system 560 may perform one or more aspects of method(s) described herein, such as method 300 of FIG. 3”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Cupersmith to incorporate the teachings of Hausman of wherein the sending a query to an LLM in the robotic system includes sending a query to an LLM onboard the robot in order to respond to various free-form natural language inputs of a user attempting to control the robot. [(Hausman 2)]
Regarding claim 14, Modified Cupersmith has all of the elements of claim 1 as discussed above.
Cupersmith does not explicitly teach wherein the sending, by the interface, a query to the LLM includes formulating a natural language statement.
However, Hausman teaches wherein the sending, by the interface, a query to the LLM includes formulating a natural language statement. [(see at least paragraph 36) “Namely, while the decoding of the instruction obtained in this way always consists of skills that are available to the robot, these skills may not necessarily be appropriate for executing the desired high-level task in the specific situation that the robot is currently in. For example, if an FF NL prompt is “bring me an apple”, the optimal set of skills changes if there is no apple in view of the robot or if the robot already has one in its gripper.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Cupersmith to incorporate the teachings of Hausman of wherein the sending, by the interface, a query to the LLM includes formulating a natural language statement in order for the robot system to be able to receive a user-provided natural language instruction that describes a task that the robot should execute. [(Hausman 31)]
Regarding claim 15, Modified Cupersmith has all of the elements of claim 14 as discussed above.
Cupersmith does not explicitly teach wherein the formulating a natural language statement includes describing a context in the natural language.
However, Hausman teaches wherein the formulating a natural language statement includes describing a context in the natural language. [(see at least paragraph 5) “For example, prompting an LLM with “I spilled my drink on the table, can you help” can result in generation of LLM output that reflects NL content that is descriptive of reasonable step(s) for cleaning a spill. For instance, the highest probability decoding of the LLM output can reflect NL content of “you could try using a vacuum”. However, “you could try using a vacuum” may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. For example, the robot may not have an integrated vacuum and may not have access to any separate vacuum in the particular environment, or may be incapable of controlling a separate vacuum in the particular environment.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Cupersmith to incorporate the teachings of Hausman of wherein the formulating a natural language statement includes describing a context in the natural language to control a robot to perform a task in response to a free-form natural language input. [(Hausman 6)]
Regarding claim 16, Modified Cupersmith has all of the elements of claim 15 as discussed above.
Cupersmith does not explicitly teach wherein the describing a context in the natural language includes describing, in the natural language, at least one of the robot, the environment, the human, and the task.
However, Hausman teaches wherein the describing a context in the natural language includes describing, in the natural language, at least one of the robot, the environment, the human, and the task. [(see at least paragraph 7, 12) “an explanation generated in a prior pass utilizing the LLM (e.g., based on a prior LLM prompt of “explain how you would help when a user says ‘I spilled my drink on the table, can you help’”); and/or term(s) to encourage prediction of step(s) by the LLM (e.g., including, at the end of the prompt “I would 1.”).” As in 12 “assume that current environmental state data includes an image, captured by a camera of the robot, and that the image captures a nearby sponge and a nearby banana, but does not capture any squeegee. In such a situation, the world-grounding measure for the first robotic skill having an NL skill description of “pick up a sponge” can reflect a high probability, the world-grounding measure for the second robotic skill having an NL skill description of “pick up a banana” can also reflect a high probability, and the world-grounding measure for the third robotic skill having an NL description of “pick up a squeegee” can reflect a low probability.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Cupersmith to incorporate the teachings of Hausman of wherein the describing a context in the natural language includes describing, in the natural language, at least one of the robot, the environment, the human, and the task in order for the robot system to be able to receive a user-provided natural language instruction that describes a task that the robot should execute. [(Hausman 31)]
Regarding claim 17, Modified Cupersmith has all of the elements of claim 1 as discussed above.
Cupersmith does not explicitly teach wherein the receiving, by the interface, a response from the LLM includes receiving a natural language statement from the LLM, and parsing the natural language statement.
However, Hausman teaches wherein the receiving, by the interface, a response from the LLM includes receiving a natural language statement from the LLM, and parsing the natural language statement. [(see at least paragraph 7) “For instance, the prompt can include some or all of the terms of the FF NL instruction, but can additionally include: scene descriptor(s) of the current environment (e.g., NL descriptor(s) of object(s) detected in the environment); an explanation generated in a prior pass utilizing the LLM (e.g., based on a prior LLM prompt of “explain how you would help when a user says ‘I spilled my drink on the table, can you help’”); and/or term(s) to encourage prediction of step(s) by the LLM (e.g., including, at the end of the prompt “I would 1.”).”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Cupersmith to incorporate the teachings of Hausman of wherein the receiving, by the interface, a response from the LLM includes receiving a natural language statement from the LLM, and parsing the natural language statement in order to respond to various free-form natural language inputs of a user attempting to control the robot. [(Hausman 2)]
Regarding claim 18, In view of the above combination of references, Cupersmith further teaches wherein the initiating, by the robot, an interim interaction with the human includes initiating autonomously, by the robot, an interim interaction with the human. [(see at least paragraph 355) “robots 300 may display certain persona interactions based on a current celebration activity or task currently being performed by robot 300. For example, upon activation (e.g., by receiving a remote celebration task), robot 300 may display a moving face, facial expressions, or animations while robot 300 is moving to a destination (e.g., bobbing back and forth, searching eye movements, perspiring, or the like). Robot 300 may display facial expressions and lip movements while interacting with other robots and/or players at a celebration scene (e.g., emulating articulation of audible interactions such as lip-syncing to songs). Such persona animations allow nearby players to understand what robot 300 are currently doing, as well as comforting and easing human interaction with robot 300.”]
Regarding claim 19, In view of the above combination of references, Cupersmith further teaches wherein the initiating, by the robot, an interim interaction with the human includes initiating, by the robot, a diversion. [(see at least paragraph 239) “As such, the robots 300 may be configured to emulate aspects of human interactions and social conventions through various persona animations (e.g., via graphical displays and audio outputs). For example, the robots 300 may be configured to display facial features such as, for example, eyes, nose, mouth, lips, eye brows, and the like. The robots 300 may use and alter the display of these facial features during interactions with users, thereby making the interactions seem more like human to human interactions and easing discomfort of the user (e.g., smiling, mouth and lip movements while talking, or the like). In some embodiments, the robots 300 may display certain persona interactions based on a current activity or task currently being performed by the robot 300.”]
Regarding claim 20, In view of the above combination of references, Cupersmith further teaches wherein the initiating, by the robot, an interim interaction with the human includes at least one of telling a joke, sharing a fact, posing a brainteaser, and initiating a conversation. [(see at least paragraph 107) “In some embodiments, the robot 300 may be configured to provide entertainment functions through use of the displays 350 and speakers, including playing songs or videos, telling jokes, performing animated movements or dances (e.g., alone or with other robots 300), or any combination thereof.”]
The Examiner has cited particular paragraphs or columns and line numbers in the references applied to the claims above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested of the Applicant in preparing responses, to fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner. See MPEP 2141.02 [R-07.2015] VI. A prior art reference must be considered in its entirety, i.e., as a whole, including portions that would lead away from the claimed Invention. W.L. Gore & Associates, Inc. v. Garlock, Inc., 721 F.2d 1540, 220 USPQ 303 (Fed. Cir. 1983), cert, denied, 469 U.S. 851 (1984). See also MPEP §2123.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMMED YOUSEF ABUELHAWA whose telephone number is (571)272-3219. The examiner can normally be reached Monday-Friday 8:30-5:00 with flex.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wade Miles can be reached at 571-270-7777. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMMED YOUSEF ABUELHAWA/Examiner, Art Unit 3656
/WADE MILES/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights