DETAILED ACTION
This Final Office Action is responsive to the claims filed on April 27, 2026. Claims 1, 6-12, and 15-23 are presented for examination.
Claims 1, 6-10, 12, 15-16, 18, and 20-23 are rejected for nonstatutory double patenting.
Claims 1, 6-12, and 15-23 are rejected under 35 USC 101.
Claims --1, 7-10, 12, 15-16, 18, and 20-23 are rejected under 35 USC 103 over Bigdeli in view of Reddy.
Claim 6 is are rejected under 35 USC 103 over Bigdeli in view of Reddy and Rosenfeld.
Claims 11, 17, and 19 are rejected under 35 USC 103 over Bigdeli in view of Reddy and Caren.
Response To Arguments/Amendments
Double Patenting: The Applicant attempted to file a terminal disclaimer. However, the terminal disclaimer was not accepted for what appears to be formal reasons. The Applicant is invited to remedy this without charge in order to advance prosecution. However, currently the rejection must be maintained.
35 USC 112(b): The Applicant’s arguments and amendments have been considered and are persuasive. The rejections are withdrawn.
35 USC 101: Regarding the software per se rejections, the Applicant’s arguments and amendments have been considered, and are persuasive. Those rejections have been withdrawn. Regarding the subject matter eligibility rejections, the Applicant’s arguments and amendments have been considered but are not persuasive. The Applicant’s arguments will be addressed in the order presented in the latest response.
With regard to the software per se rejection, the Applicant’s arguments and amendments have been considered and are persuasive. With regard to the subject matter eligibility rejection, the Applicant’s amendments and arguments has been considered but are not persuasive. The Applicant’s arguments will be treated in the order presented in the most recent response.
Mimicking A Real-World Environment: The Applicant asserts that the operations of the claims mimic the real-world activity of robots. However, the claim language itself only ties the method to the real-world in a superficial way. That is, there is no interaction between the real-world and the features of the claims. MPEP 2106.05(f)(1) states,
The recitation of claim limitations that attempt to cover any solution to an identified problem with no restriction on how the result is accomplished and no description of the mechanism for accomplishing the result, does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it". See Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44 (Fed. Cir. 2016); Intellectual Ventures I v. Symantec, 838 F.3d 1307, 1327, 120 USPQ2d 1353, 1366 (Fed. Cir. 2016); Internet Patents Corp. v. Active Network, Inc., 790 F.3d 1343, 1348, 115 USPQ2d 1414, 1417 (Fed. Cir. 2015). In contrast, claiming a particular solution to a problem or a particular way to achieve a desired outcome may integrate the judicial exception into a practical application or provide significantly more. See Electric Power, 830 F.3d at 1356, 119 USPQ2d at 1743.
The claims, as recited, fail to interact with the real world using a mechanism that qualifies under 35 USC 101. The claim also does not represent any improvement to computing, because the computer is merely being used as a tool to carry out operations entirely contained within the abstract idea. This mechanism must include additional limitations that confer eligibility by either integrating the abstract idea into a practical application or by combining with other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept. With respect to the possibility of an inventive concept, the references of record to demonstrate that the steps are not conventional. The Applicant must identify the additional limitations that combine with the other elements of the claim to provide significantly more than the abstract idea that would confer the inventive concept. As it stands, the steps of the claims, as shown by the art of record, are well-understood, routine, and conventional (WURC) activity. Accordingly, the rejections are maintained.
Art Rejections: The Applicant’s arguments and amendments have been considered an are persuasive. The existing rejections have been withdrawn based on the amendments made. However, new art rejections, necessitated by the Applicant’s amendments, are presented herein.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 6-10, 12, 15-16, 18, and 20-23 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims of 1, 5-10, 12, 15-17, and 20-23 (See mapping below) of copending Application No. 17/846,262 (‘262) in view of NPL: “” by Rosenfeld et al. (Rosenfeld).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claims to modify the robot and tele-operation systems of ‘262 by the GUI of Rosenfeld because a person of ordinary skill in the art would be motivated based on the exchange of data and update of the simulation environment in the robot-human collaborative environment of the independent claims of ‘262 to look to Rosenfeld for human-robot team collaboration that significantly improve performance. (‘262 Claim 1 “1. A method of updating, by a robot system including a robot body, a simulation of an external environment of the robot body, the method comprising: loading a simulation of an external environment of the robot body; providing data collected by at least one sensor on-board the robot body to a tele- operation system that is physically remote from the robot body; receiving simulation instructions from the tele-operation system; and updating the simulation of the external environment based on the simulation instructions.; Rosenfeld Abstract “In this article we propose a novel approach for utilizing automated advising agents in assisting an operator to better manage a team of multiple robots in complex environments. We introduce an advice provision methodology and exemplify its implementation using automated advising agents in two real-world human–multi-robot team collaboration tasks: the Search And Rescue (SAR) and the warehouse operation tasks. Our intelligent advising agents were evaluated through extensive field trials, with over 150 human operators using both simulated and physical mobile robots, and showed a significant improvement in the team’s performance.”)
This is a provisional nonstatutory double patenting rejection.
Claim
17/846,243
17/846,262
1
1. A method of updating a simulation of an external environment of an agent by a tele-operation system, the method comprising:
1. A method of updating, by a robot system including a robot body, a simulation of an external environment of the robot body, the method comprising:
displaying a simulation of the external environment of the agent to at least one user of the tele-operation system;
Rosenfeld Page 220, 4.1.4. Results "We further analyze the subject’s acceptance rate of the agent’s advice. Advice is considered acceptedif it is performed by the subject while it is displayed on the GUI." See Rosenfeld Figs. 1-2 on Page 217
receiving data collected by at least one sensor of the agent;
1. […] providing data collected by at least one sensor on-board the robot body to a tele- operation system that is physically remote from the robot body; […]
providing the data collected by at least one sensor of the agent to the at least one user of the tele-operation system;
Rosenfeld Page 217, Second Paragraph "A is the action set that the operator can take. Recall that A is also the advice set from which the agent can propose advice. We define Aas the instantiations of the following 7 action schemes: “Robot iis waiting for your command”, “Unload item xat station y.”, “Complete the packing of order z.”, “Clear an obstacle from the floor.”, “Obstacle was detected – restrict its cell.”,“Clear a critical obstacle from the floor.”, and “Critical obstacle was detected – restrict its cell.”" - The user is informed if an obstacle is detected/sensed.)
receiving simulation instructions from the at least one user of the tele-operation system, the simulation instructions based at least in part on the data collected by at least one sensor of the agent, wherein the simulation instructions describe a modification to at least one object representation in the simulation of the external environment;
1. receiving simulation instructions from the tele-operation system, wherein the simulation instructions describe a modification to at least one object representation in the simulation of the external environment; and
updating the simulation of the external environment based on the simulation instructions, wherein updating the simulation of the external environment based on the simulation instructions includes applying the modification to at least one object representation in the simulation of the external environment to cause the at least one object representation to more closely resemble a corresponding real-world counterpart object in the real-time external environment; and
1. updating the simulation of the external environment based on the simulation instructions, wherein updating the simulation of the external environment based on the simulation instructions includes applying the modification to at least one object representation in the simulation of the external environment to cause the at least one object representation to more closely resemble a corresponding real-world counterpart object in the real-time external environment.
displaying the updated simulation of the external environment to the at least one user of the tele-operation system.
Rosenfeld Page 220, 4.1.4. Results "We further analyze the subject’s acceptance rate of the agent’s advice. Advice is considered acceptedif it is performed by the subject while it is displayed on the GUI." See Rosenfeld Figs. 1-2 on Page 217
6. The method of claim 1 wherein receiving simulation instructions from the tele-operation system includes receiving instructions that describe a modification to the simulation of the external environment.
7. The method of claim 6 wherein updating the simulation of the external environment based on the simulation instructions includes applying the modification to the simulation of the external environment to cause the simulation of the external environment to more closely match a reality of the external environment.
8. The method of claim 6 wherein receiving instructions that describe a modification to the simulation of the external environment includes receiving instructions that describe a modification to at least one object representation in the simulation of the external environment.
9. The method of claim 8 wherein updating the simulation of the external environment based on the simulation instructions includes applying the modification to at least one object representation in the simulation of the external environment to cause the at least one object representation to more closely resemble a corresponding real-world counterpart object in the external environment.
6
6. The method of claim 1 wherein receiving simulation instructions from the at least one user of the tele-operation system includes receiving instructions that describe a new object representation for the simulation of the external environment, and wherein updating the simulation of the external environment based on the simulation instructions includes applying the simulation instructions to add the new object representation to the simulation of the external environment, the new object representation corresponding to a real-world counterpart in the external environment characterized, at least in part, by the data collected by at least one sensor of the agent.
10. The method of claim 1 wherein receiving simulation instructions from the tele-operation system includes receiving instructions that describe a new object representation for the simulation of the external environment, and wherein updating the simulation of the external environment based on the simulation instructions includes applying the simulation instructions to add the new object representation to the simulation of the external environment, the new object representation corresponding to a real-world counterpart in the external environment characterized, at least in part, by the data collected by at least one sensor on- board the robot body.
7
7. The method of claim 1, further comprising:providing additional data collected by at least one sensor of the agent to the at least one user of the tele-operation system; receiving additional simulation instructions from the at least one user of the tele- operation system; re-updating the simulation of the external environment based on the additional simulation instructions; and displaying the re-updated simulation of the external environment to the at least one user of the tele-operation system.
This is just a repetition of steps of claim 1 with new data, which is rejected for the same reasons as claim 1.
8
8. The method of claim 1 wherein the agent includes a robot system including a robot body and the at least one sensor of the agent includes at least one image sensor on-board the robot body, and wherein: displaying a simulation of the external environment of the agent to at least one user of the tele-operation system includes displaying a simulation of the external environment of the robot body to at least one user of the tele-operation system; receiving data collected by at least one sensor of the agent includes receiving data collected by at least one image sensor of the robot body; and providing the data collected by at least one sensor of the agent to the at least one user of the tele-operation system includes providing the data collected by at least one image sensor of the robot body to the at least one user of the tele-operation system.
1. […] providing data collected by at least one sensor on-board the robot body to a tele- operation system that is physically remote from the robot body; Rosenfeld See Fig. 9 on Page 223; Page 229, Fifth Paragraph "The GUI (see Fig.2) provides the operator with on-line feedback from the cameras mounted on the robots (Thumbnail area), the 2D map of the terrain including the robots’ reported positions and their footprints (area 1), an enlarged camera view of the robot of interest (area 2), an action control bar, and a joystick widget. The action bar’s commands and joystick functions are also available using keyboard and mouse shortcuts inspired by strategic computer games. For example, in order to set interest on a specific robot, the operator could click its thumbnail camera or location on the map or could click on its number on the keyboard. Double clicking will center the map on the robot’s location."
9
9. The method of claim 8, further comprising:training the robot system to autonomously update the simulation of the external environment based on multiple iterations of: [The method of claim 1]
5. The method of claim 1, further comprising: training the robot system to autonomously update the simulation of the external environment based on multiple iterations of: [The rejection of the steps is the same as in claim 1, including Rosenfeld]
10
10. The method of claim 8 wherein the at least one user of the tele-operation system includes at least one tele-operator of the robot system.
1. […] providing data collected by at least one sensor on-board the robot body to a tele- operation system that is physically remote from the robot body; Rosenfeld See Fig. 9 on Page 223; Page 229, Fifth Paragraph "The GUI (see Fig.2) provides the operator with on-line feedback from the cameras mounted on the robots (Thumbnail area), the 2D map of the terrain including the robots’ reported positions and their footprints (area 1), an enlarged camera view of the robot of interest (area 2), an action control bar, and a joystick widget. The action bar’s commands and joystick functions are also available using keyboard and mouse shortcuts inspired by strategic computer games. For example, in order to set interest on a specific robot, the operator could click its thumbnail camera or location on the map or could click on its number on the keyboard. Double clicking will center the map on the robot’s location."
11
Not Rejected
12
12. A tele-operation system comprising: at least one processor; and at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable storage medium storing data and/or processor-executable instructions that, when executed by the at least one processor, cause the tele-operation system to: [Perform operations of claim 1 under limitations of claim 1]
12. A robot system comprising: […] at least one processor; and […] at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable storage medium storing data and/or processor-executable instructions that, when executed by the at least one processor, cause the robot system to: [Perform operations rejected under claim 1 under limitations of claim 1 using Rosenfeld]
16. The robot system of claim 12 wherein the simulation instructions received from the tele-operation system describe a modification to the simulation of the external environment, and wherein the data and/or processor-executable instructions that, when executed by the at least one processor, cause the robot system to update the simulation of the external environment based on the simulation instructions, cause the robot system to apply the modification to the simulation of the external environment to cause the simulation of the external environment to more closely match a reality of the external environment.
17. The robot system of claim 12 wherein the simulation instructions received from the tele-operation system describe a modification to at least one object representation in the simulation of the external environment, and wherein the data and/or processor-executable instructions that, when executed by the at least one processor, cause the robot system to update the simulation of the external environment based on the simulation instructions, cause the robot system to apply the modification to at least one object representation in the simulation of the external environment to cause the at least one object representation to more closely resemble a corresponding real-world counterpart object in the external environment.
15
15. The tele-operation system of claim 12 wherein the agent includes a robot system including a robot body, and the at least one sensor of the agent includes at least one image sensor on-board the robot body.
1. […] providing data collected by at least one sensor on-board the robot body to a tele- operation system that is physically remote from the robot body; Rosenfeld See Fig. 9 on Page 223; Page 229, Fifth Paragraph "The GUI (see Fig.2) provides the operator with on-line feedback from the cameras mounted on the robots (Thumbnail area), the 2D map of the terrain including the robots’ reported positions and their footprints (area 1), an enlarged camera view of the robot of interest (area 2), an action control bar, and a joystick widget. The action bar’s commands and joystick functions are also available using keyboard and mouse shortcuts inspired by strategic computer games. For example, in order to set interest on a specific robot, the operator could click its thumbnail camera or location on the map or could click on its number on the keyboard. Double clicking will center the map on the robot’s location."
16
16. The tele-operation system of claim 15, further comprising: data and/or processor-executable instructions stored in the non-transitory processor-readable storage medium that, when executed by the at least one processor, cause the tele-operation system to: train the robot system to autonomously update to the simulation of the external environment based on multiple iterations of:receiving simulation instructions from the at least one user of the tele-operation system based at least in part on the data collected by at least one image sensor of the robot body; and updating the simulation of the external environment based on the simulation instructions.
15. The robot system of claim 12, further comprising: data and/or processor-executable instructions stored in the at least one non- transitory processor-readable storage medium that, when executed by the at least one processor, cause the robot system to: train the robot system to autonomously update the simulation of the external environment based on multiple iterations of: receiving simulation instructions from the tele-operation system; and updating the simulation of the external environment based on the simulation instructions.
17
Not Rejected
18
18. A computer program product comprising data and/or processor- executable instructions stored in a non-transitory processor-readable storage medium, the data and/or processor-executable instructions which, when the non-transitory processor-readable storage medium is communicatively coupled to at least one processor of a tele-operation system and the at least one processor executes the data and/or processor-executable instructions, cause the tele-operation system to: [Perform operations under limitations of claim 1]
20. A computer program product comprising data and/or processor- executable instructions stored in a non-transitory processor-readable storage medium, the data and/or processor-executable instructions which, when the non-transitory processor-readable storage medium is communicatively coupled to at least one processor of a robot system and the at least one processor executes the data and/or processor-executable instructions, cause the robot system to: [See rejection of claim 1 operations and features with Rosenfeld]
19
Not Rejected
20
20. The computer program product of claim 18, wherein the agent includes a robot system including a robot body and the at least one sensor of the agent includes at least one image sensor on-board the robot body, and further comprising: data and/or processor-executable instructions which, when the non-transitory processor-readable storage medium is communicatively coupled to at least one processor of a tele-operation system and the at least one processor executes the data and/or processor- executable instructions, cause the tele-operation system to: train the robot system to autonomously update the simulation of the external environment based on multiple iterations of:
20. A computer program product comprising data and/or processor- executable instructions stored in a non-transitory processor-readable storage medium, the data and/or processor-executable instructions which, when the non-transitory processor-readable storage medium is communicatively coupled to at least one processor of a robot system and the at least one processor executes the data and/or processor-executable instructions, cause the robot system to: [Execute operations of claim 1]15. The robot system of claim 12, further comprising: data and/or processor-executable instructions stored in the at least one non- transitory processor-readable storage medium that, when executed by the at least one processor, cause the robot system to: train the robot system to autonomously update the simulation of the external environment based on multiple iterations of: providing data collected by at least one sensor of the robot body to a tele- operation system that is physically remote from the robot body; receiving simulation instructions from the tele-operation system; and updating the simulation of the external environment based on the simulation instructions.
21-23
The non-transitory processor-readable storage medium of claim 18, further comprising:/ The tele-operation system of claim 12, further comprising: The method of claim 1, further comprising:
data and/or processor-executable instructions stored in the non-transitory processor-readable storage medium that, when executed by the at least one processor, cause the tele-operation system to:
load a stored instance of the simulation of the external environment of the agent,
wherein the modification to at least one object representation corrects a discrepancy between stored instance of the simulation and the actual external environment of the agent
See the respective rejections of independent claims that teach these features.
See the respective independent claims that teach these features.
12. at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable storage medium storing data and/or processor-executable instructions that, when executed by the at least one processor, cause the robot system to:
21. wherein the modification to at least one object representation corrects a discrepancy between the simulation and the actual external environment of the robot body.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Subject Matter Eligibility
Claims 1-20 are rejected under 35 U.S.C. 101 as ineligible subject matter.
Independent Claims
Claim 12 (Statutory Category – Machine)
Step 2A – Prong 1: Judicial Exception Recited?
Yes, the claims recite a mental process.
Claim 12 recites:
update the simulation of the external environment based on the simulation instructions, wherein updating the simulation of the external environment based on the simulation instructions includes applying the modification to at least one object representation in the simulation of the external environment to cause the at least one object representation to more closely resemble a corresponding real-world counterpart object in the real-time external environment; and (Also Amendment to Claim 1 Preamble With Same Subject Matter) (Evaluation/Mental Process – Updating a simulation (e.g., modifying an image) can practically be performed in the mind or with aid of pen, paper, and/or a calculator.)
Claim 12 recites a mental process, which is an abstract idea.
Claim 12 recites an abstract idea.
Step 2A – Prong 2: Integrated into a Practical Solution?
No.
The additional limitations:
at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable storage medium storing data and/or processor-executable instructions that, when executed by the at least one processor, cause the tele-operation system to:
[…] teleoperation system […]
[…] sensor […] […] agent […]
These are generic computing elements recited at a high level of generality, and, under MPEP 2106.05(f), fail to integrate the abstract idea into a practical application.
[…]
receive data collected by at least one sensor of the agent;
provide the data collected by at least one sensor of the agent to the at least one user of the tele-operation system;
receive simulation instructions from the at least one user of the tele-operation system based at least in part on the data collected by at least one sensor of the agent, wherein the simulation instructions describe a modification to at least one object representation in the simulation of the external environment;
[…]
These steps are mere data gathering similar to the MPEP 2106.05(g) examples: “e.g., a step of obtaining information about credit card transactions, which is recited as part of a claimed process of analyzing and manipulating the gathered information by a series of steps in order to detect whether the transactions were fraudulent” “iv. Obtaining information about transactions using the Internet to verify credit card transactions,” “iii. Selecting information, based on types of information and availability of information in a power-grid environment, for collection, analysis and display,” “v. Consulting and updating an activity log, Ultramercial.” Mere data gather is insignificant extra-solution activity and, under MPEP 2106.05(g), fails to integrate the abstract idea into a practical application.
[…]
display a simulation of an external environment of an agent to at least one user of the tele-operation system;
[…]
display the updated simulation of the external environment to the at least one user of the tele-operation system.
Display of data is insignificant extra-solution activity similar to the MPEP 2106.05(g) example: “iii. Selecting information, based on types of information and availability of information in a power-grid environment, for collection, analysis and display,” “ii. Printing or downloading generated menus.” These steps are insignificant extra-solution activity and, under MPEP 2106.05(g), fail to integrate the abstract idea into a practical application.
Any recited data merely limits the abstract idea to a particular field of technology and, under MPEP 2106.05(h), fails to integrate the abstract idea into a practical application.
None of the additional limitations of claim 12, whether in isolation or combination, integrate the abstract idea into a practical application.
Accordingly, claim 12 is directed to the abstract idea.
Step 2B: Claim provides an Inventive Concept?
No.
The additional limitations:
at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable storage medium storing data and/or processor-executable instructions that, when executed by the at least one processor, cause the tele-operation system to:
[…] teleoperation system […]
[…] sensor […] […] agent […]
These are generic computing elements recited at a high level of generality, and, under MPEP 2106.05(f), fail to combine with the other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept.
[…]
receive data collected by at least one sensor of the agent;
provide the data collected by at least one sensor of the agent to the at least one user of the tele-operation system;
receive simulation instructions from the at least one user of the tele-operation system based at least in part on the data collected by at least one sensor of the agent, wherein the simulation instructions describe a modification to at least one object representation in the simulation of the external environment;
[…]
These steps are well-understood, routine, conventional (WURC) activity similar to the MPEP 2106.05(d) examples: “i. Receiving or transmitting data over a network” “iii. Electronic recordkeeping” “iv. Storing and retrieving information in memory” “i. Determining the level of a biomarker in blood by any means” “vi. Arranging a hierarchy of groups, sorting information, eliminating less restrictive pricing information and determining the price.” Because these steps are WURC and, as previously demonstrated, insignificant extra-solution activity, under MPEP 2106.05(d) and 2106.05(g), the steps to fail to combine with the other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept.
[…]
display a simulation of an external environment of an agent to at least one user of the tele-operation system, the at least one user being physically remote from the agent;
[…]
display the updated simulation of the external environment to the at least one user of the tele-operation system.
Display of data is WURC similar to the MPEP 2106.05(d) examples: “iv. Storing and retrieving information in memory “ “iv. Presenting offers and gathering statistics.” Because these steps are WURC and, as previously demonstrated, insignificant extra-solution activity, under MPEP 2106.05(d) and 2106.05(g), the steps to fail to combine with the other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept.
Any recited data merely limits the abstract idea to a particular field of technology and, under MPEP 2106.05(h), fails to combine with the other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept.
None of the additional limitations of claim 12, whether in isolation or combination, combine with the other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept.
Claim 12 is ineligible.
Claim 1 (Statutory Category – Process)
Regarding claim 1, claim 1 recites the method executed by the configuration of the system of claim 12 and is rejected for the same reasons as claim 12.
Claim 1 is ineligible.
Claim 18 (Statutory Category – Machine)
Regarding claim 18, the Applicant has amended the claims to positively recite the non-transitory CRM as an element. Claim 18 is the software and likely intended to be the CRM of claim 12, so claim 18 is rejected for the same reasons as claim 12.
Claim 18 is ineligible.
Dependent Claims
The dependent claims are also ineligible for the following reasons. Note that elements recognized as generic computing elements in the independent claims fail to confer eligibility under MPEP 2106.05(f) for the same reasons in the dependent claims. Similarly, the data description specific to the technological environment merely restrict the abstract idea to a particular technological environment and, under MPEP 2106.05(h), fail to confer eligibility.
Claim 6
wherein receiving simulation instructions from the at least one user of the tele-operation system includes receiving instructions that describe a new object representation for the simulation of the external environment, and
This is an element of the mere data gathering and fails to confer eligibility for the same reasons as the receiving steps of the independent claim.
Also, the nature of the data merely limits the abstract idea to a particular technological field and, under MPEP 2106.05(h), fails to confer eligibility at Step 2A, Prong 2 and Step 2B.
wherein updating the simulation of the external environment based on the simulation instructions includes applying the simulation instructions to add the new object representation to the simulation of the external environment, the new object representation corresponding to a real-world counterpart in the external environment characterized, at least in part, by the data collected by at least one sensor of the agent.
This is an evaluation that is an element of the updating, and is an element of the abstract idea for the same reasons as the updating step in the independent claims.
Claim 6 fails to recite any additional limitations that confer eligibility at Step 2A, Prong 2 and Step 2B.
Claim 6 is ineligible
Claim 7
providing additional data collected by at least one sensor of the agent to the at least one user of the tele-operation system; receiving additional simulation instructions from the at least one user of the tele- operation system; re-updating the simulation of the external environment based on the additional simulation instructions; and displaying the re-updated simulation of the external environment to the at least one user of the tele-operation system.
This is just a repetition of steps of the independent claims and is rejected for the same reasons as the steps in the prior iteration expressed in the independent claims.
Claim 7 fails to recite any additional limitations that confer eligibility at Step 2A, Prong 2 and Step 2B.
Claim 7 is ineligible
Claim 8
wherein the agent includes a robot system including a robot body and the at least one sensor of the agent includes at least one image sensor on-board the robot body, and wherein:
A robot agent with a robot body is a computing element recited at a high level of generality, so under MPEP 2106.05(f), it fails to confer eligibility at Step 2A, Prong 2 and Step 2B.
displaying a simulation of the external environment of the agent to at least one user of the tele-operation system includes displaying a simulation of the external environment of the robot body to at least one user of the tele-operation system;
This fails to confer eligibility for the same reasons as the display steps of the independent claims.
receiving data collected by at least one sensor of the agent includes receiving data collected by at least one image sensor of the robot body; and
This merely qualifies the data gathered and is insignificant extra-solution activity and WURC for the same reasons as the receiving steps of the independent claims. Accordingly, this fails to confer eligibility at Step 2A, Prong 2 and Step 2B.
providing the data collected by at least one sensor of the agent to the at least one user of the tele-operation system includes providing the data collected by at least one image sensor of the robot body to the at least one user of the tele-operation system.
This fails to confer eligibility for the same reasons as the providing step in the independent claims.
Claim 8 fails to recite any additional limitations that confer eligibility at Step 2A, Prong 2 and Step 2B.
Claim 8 is ineligible
Claim 9
further comprising: training the robot system to autonomously update the simulation of the external environment based on multiple iterations of:
the receiving data collected by at least one image sensor of the robot body;
the providing the data collected by at least one image sensor of the robot body to the at least one user of the tele-operation system
the receiving simulation instructions from the at least one user of the tele- operation system based at least in part on the data collected by at least one image sensor of the robot body; and
the updating the simulation of the external environment based on the simulation instructions.
This merely recites at a high level of generality that a generic AI training is performed based on the steps of the independent claims. Accordingly, this fails to confer eligibility as an apply it step on a generic computing element under MPEP 2106.05(f).
Also, the steps are mere repeats of the steps in the independent claims and are rejected for the same reasons as those steps.
Claim 9 fails to recite any additional limitations that confer eligibility at Step 2A, Prong 2 and Step 2B.
Claim 9 is ineligible
Claim 10
wherein the at least one user of the tele-operation system includes at least one tele-operator of the robot system.
This merely limits the abstract idea to a particular field of technology and so, under MPEP 2106.05(h), fails to confer eligibility at Step 2A, Prong 2 and Step 2B.
Claim 10 fails to recite any additional limitations that confer eligibility at Step 2A, Prong 2 and Step 2B.
Claim 10 is ineligible
Claims 11, 17, and 19
wherein the at least one user of the tele-operation system includes a plurality of tele-artists and the tele-operation system enables the plurality of tele-artists to concurrently update the simulation.
This merely limits the abstract idea to a particular field of technology and so, under MPEP 2106.05(h), fails to confer eligibility at Step 2A, Prong 2 and Step 2B.
Claim 11 fails to recite any additional limitations that confer eligibility at Step 2A, Prong 2 and Step 2B.
Claims 17 and 19 recite substantially the same features as claim 11.
Claims 11, 17, and 19 are ineligible
Claim 15
wherein the agent includes a robot system including a robot body, and the at least one sensor of the agent includes at least one image sensor on-board the robot body.
This merely recites at a high level of generality the use of a robot with sensors, including an image sensor (e.g., a camera). Accordingly, this fails to confer eligibility as an apply it step on a generic computing element under MPEP 2106.05(f).
This merely limits the abstract idea to a particular field of technology and so, under MPEP 2106.05(h), fails to confer eligibility at Step 2A, Prong 2 and Step 2B.
Claim 15 fails to recite any additional limitations that confer eligibility at Step 2A, Prong 2 and Step 2B.
Claim 15 is ineligible
Claims 16 and 20
train the robot system to autonomously update to the simulation of the external environment based on multiple iterations of:
receiving simulation instructions from the at least one user of the tele-operation system based at least in part on the data collected by at least one image sensor of the robot body; and
updating the simulation of the external environment based on the simulation instructions.
This represents a generic computing operation of training a robot system to autonomously update an environment recited at a high level, especially because there is no recited link between the repeated iterations of the steps and how they affect the training.
Also, the repeated steps are fail to confer eligibility for the same reasons as the corresponding steps in the independent claims.
Claim 16 fails to recite any additional limitations that confer eligibility at Step 2A, Prong 2 and Step 2B.
Claim 20 recites features similar to the features of claim 16 and fails to confer eligibility for at least the same reasons.
Claims 16 and 20 are ineligible
Claims 21-23
data and/or processor-executable instructions stored in the non-transitory processor-readable storage medium that, when executed by the at least one processor, cause the tele-operation system to:
These are generic computing elements recited at a high level for the same reasons as the elements were in the respective claims from which these claims depend, so the elements fail to confer eligibility under MPEP 2106.05(f).
load a stored instance of the simulation of the external environment of the agent, wherein the modification to at least one object representation corrects a discrepancy between stored instance of the simulation and the actual external environment of the agent
The loading operation is also a generic computing operation under 2106.05(f), so it fails to confer eligibility.
Further, this is insignificant extra-solution activity and WURC for the same reason as the receiving steps, so it fails to confer eligibility under MPEP 2106.05(g) and 2106.05(d) for at least the same reasons.
Claims 21-23 fail to recite any additional limitations that confer eligibility at Step 2A, Prong 2 and Step 2B.
Claims 21-23 are ineligible
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7-10, 12, 15-16, 18, and 20-23: Bigdeli and Reddy
Claims 1, 7-10, 12, 15-16, 18, and 20-23 are rejected under 35 U.S.C. 103 as being unpatentable over US 2021/0349478 A1 to Bigdeli et al. (Bigdeli) in view of NPL: “Shared Autonomy vie Deep Reinforcement Learning” by Reddy et al. (Reddy).
Claims 1, 12 and 18
Regarding claim 12, Bigdeli teaches:
A system comprising: at least one processor; and at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable storage medium storing data and/or processor-executable instructions that, when executed by the at least one processor, cause the tele-operation system to: (Bigdeli [0045] “Further, the architecture builds upon a 3D representation/model of the environment. While at the start of a mission, the 3D digital twin is initialized to its static state, observations are accumulated throughout the mission to provide a temporally updated model. More specifically, at any time t, given the collective and integrated observations across the platform, areas of change and location of moving objects are identified. Furthermore, the uncertainty about the locations of change as well as moving objects will be quantified and tracked. […] Again this prediction comes with quantification of uncertainty. Further, the gateway may be associated with continual correction and learning via (an active) filter. The most recent observations collected and reported by the drones (onboard mobile sensors) as well as static sensors distributed in an environment are compared against the predicted state of the digital twin, or more precisely its projection onto observation space. Further, the observations are not only enabling the gateway intelligence to make its knowledge of the space more precise but also allow for further learning. More specifically, a residual model will be deployed.” – This is a robot system that cause the simulation of the external environment around the robots to more closely resemble a real-time state using robots with sensors. [0055] “Further, the device 200 may include a storage device 206 communicatively coupled with the processing device 204.”)
display ; (Bigdeli [0098] “Output device(s) 1114 such as a display, speakers, a printer, etc. may also be included. The aforementioned devices are examples and others may be used.”)
receive data collected by at least one sensor of the [robot] ; provide the data collected by at least one sensor of the [robot] to the system; (Bigdeli [0045] “More specifically, at any time t, given the collective and integrated observations across the platform, areas of change and location of moving objects are identified.” [0083] “observations may be accumulated throughout the mission to provide a temporally updated model. More specifically, at any time t, given the collective and integrated observations across the platform, area of changes, and location of moving objects are identified.” – Sensors provide data about a change in an object represented in the environment prior to update)
receive simulation instructions from the system based at least in part on the data collected by at least one sensor of the [robot] , wherein the simulation instructions describe a modification to at least one object representation in the simulation of the external environment; (Bigdeli [0083] “observations may be accumulated throughout the mission to provide a temporally updated model. More specifically, at any time t, given the collective and integrated observations across the platform, area of changes, and location of moving objects are identified.” [0084] “The present invention comprises robots, drones, and sensors that are lateralized which are designed to specialize in specific information gathering and tasks. Note that the state of digital twin can be projected onto the space of observations and the lateralization of robots, drones, and sensors means that the observation spaces of the drones” – The system provides data representing changes to objects in the environment.)
update the simulation of the external environment based on the simulation instructions, wherein updating the simulation of the external environment based on the simulation instructions includes applying the modification to at least one object representation in the simulation of the external environment to cause the at least one object representation to more closely resemble a corresponding real-world counterpart object in the real-time external environment; and (Bigdeli [0086] “Further, continual correction and learning through an active filter maybe a fourth architecture of the four architectures used in the processing/path planning of the disclosed device, as shown in FIG. 5. The continual correction and learning through an active filter uses the most recent observation and information collected and reported by the robots, drones, and sensors distributed in an environment is compared against the predicted state of the digital twin, or more specifically its projection onto observation space.” – The model is updated to provide a simulation that better represents the environment at the current time.)
.
Bigdeli teaches a display and an automated system but does not appear to explicitly teach, but Bigdeli in view of Reddy teaches:
A tele-operation system comprising: at least one processor; and at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable storage medium storing data and/or processor-executable instructions that, when executed by the at least one processor, cause the tele-operation system to: (Reddy Page 2, II. Related Work “Robotic teleoperation. We build on shared autonomy work in which the system is initially unaware of the user’s goal [9, 7, 21, 13, 24, 16, 10] and explore problem statements with unknown dynamics, unknown user policy, and unknown goal representation.” See FIG. 1 (below) with a computer providing instructions to a robot.)
display a simulation of an external environment of an agent to at least one user of the tele-operation system; display a simulation of an external environment of an agent to at least one user of the tele-operation system, the at least one user being physically remote from the agent; (Reddy FIG. 1(a) illustrates the visualization produced for the environment
PNG
media_image1.png
538
628
media_image1.png
Greyscale
PNG
media_image2.png
315
326
media_image2.png
Greyscale
receive data collected by at least one sensor of the agent; provide the data collected by at least one sensor of the agent to the at least one user of the tele-operation system; (Reddy Page 3, Right Column, C. Incorporating User Control “Because we do not know dynamics in any of our problems of interest, we use a deep reinforcement learning agent which maps observations from its sensors to actions (or Q values for each action). – Data is collected from robot sensors.)
receive simulation instructions from the at least one user of the tele-operation system based at least in part on the data collected by at least one sensor of the agent, wherein the simulation instructions describe a modification to at least one object representation in the simulation of the external environment; (Page 7, Right Column, VII User Study With A Physical Robot “To evaluate our method in a more realistic environment, we formulate a “perching” task for a real human flying a real quadrotor: land the vehicle on a level, square landing pad at some distance from the initial take-off position, such that the drone’s first-person camera is pointed at a specific object in the drone’s surroundings, without flying out of bounds or running out of time. Perching a drone at an arbitrary vantage point enables it to be used as a mobile security camera for surveillance applications. Humans find it challenging to simultaneously point the camera at the desired scene and navigate to the precise location of a feasible landing pad under time constraints. An assistive copilot has little trouble navigating to and landing on the landing pad, but does not know where to point the camera because it does not know what the human wants to observe after landing. Together, the human can focus on pointing the camera and the copilot can focus on landing precisely on the landing pad.” – The user provides instructions to the robot as to where to direct the camera to provide the user with a desired visual perspective.)
[…]
display the updated simulation of the external environment to the at least one user of the tele-operation system. (Reddy Page 1, Fig. 1 (a)… the displayed simulation is a lunar lander game which updates with the control provided by the user and the simulated pilot of the robot.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claims to modify the simulation modeling of Bigdeli by the machine learning functionality of Reddy because the person of ordinary skill in the art would be motivated by the aim of Bigdeli to provide for live automated control of robots in a real-time environment, to look to Reddy, which provides a shared autonomy model for drone pilots that is better than completely automated or completely manual control alone. (Bigdeli [0048] “Further, the disclosed system may be based on a reinforcement-learning-based imitation learning method for correcting the trajectory of the drone. The network architecture includes two parts, as shown in FIG. 10. The first part aims for extracting the relative pose information from the image pair and the second part aims to predict the control correction.” [0091] “After the processing/path planning with different architecture, the path planning adjustments are next as the integrated information is used to give a real-time command to the drones as they traverse the surrounding space. The path planning adjustments are done in a two-step operation, macro-scale drone path planning with lateralized functionalities, and dynamic path adjustments and contextual optimization.”; Reddy Abstract “This approach poses the challenge of following user commands closely enough to provide the user with real-time action feedback and thereby ensure high-quality user input, but also deviating from the user’s actions when they are suboptimal. We balance these two needs by discarding actions whose values fall below some threshold, then selecting the remaining action closest to the user’s input. Controlled studies with users (n = 12) and synthetic pilots playing a video game, and a pilot study with users (n = 4) flying a real quadrotor, demonstrate the ability of our algorithm to assist users with real-time control tasks in which the agent cannot directly access the user’s private information through observations, but receives a reward signal and user input that both depend on the user’s intent. The agent learns to assist the user without access to this private information, implicitly inferring it from the user’s input. This enables the assisted user to complete the task more effectively than the user or an autonomous agent could on their own. This paper is a proof of concept that illustrates the potential for deep reinforcement learning to enable flexible and practical assistive systems.”)
Regarding claim 1, claim 1 recites the method executed by the system of claim 12 and is rejected for at least the same reasons as claim 12.
Regarding claim 1, claim 18 recites the CRM recited in claim 12, so it is rejected for at least the same reasons as claim 12.
Claim 7
Regarding claim 7, Bigdeli in view of Reddy teaches the features of claim 1 and further teaches:
providing additional data collected by at least one sensor of the agent to the at least one user of the tele-operation system; receiving additional simulation instructions from the at least one user of the tele- operation system; re-updating the simulation of the external environment based on the additional simulation instructions; and displaying the re-updated simulation of the external environment to the at least one user of the tele-operation system. (Reddy – This is a mere repetition of steps from the independent claims, representing the continued use of the lander simulation when more data is available, so claim 7 is rejected under the same rationale as the corresponding steps of the independent claims.)
Claim 8
Regarding claim 8, Bigdeli in view of Reddy teaches the features of claim 1 and further teaches:
wherein the agent includes a robot system including a robot body and the at least one sensor of the agent includes at least one image sensor on-board the robot body, and wherein: (Reddy Page 7, Right Column, Third Paragraph “To evaluate our method in a more realistic environment, we formulate a “perching” task for a real human flying a real quadrotor: land the vehicle on a level, square landing pad at some distance from the initial take-off position, such that the drone’s first-person camera is pointed at a specific object in the drone’s surroundings, without flying out of bounds or running out of time.” – The drone is a robot system with a visual sensor, a camera.)
displaying a simulation of the external environment of the agent to at least one user of the tele-operation system includes displaying a simulation of the external environment of the robot body to at least one user of the tele-operation system; receiving data collected by at least one sensor of the agent includes receiving data collected by at least one image sensor of the robot body; and providing the data collected by at least one sensor of the agent to the at least one user of the tele-operation system includes providing the data collected by at least one image sensor of the robot body to the at least one user of the tele-operation system. (Reddy See Fig. 4 – Fig. 4(a) shows the display of the simulation which models spatial relationships between the real drone and the designated landing pad in the real environment. As seen in Fig. 4(b), the robots drone camera collects visual data and provides it to the user for display.)
PNG
media_image3.png
325
504
media_image3.png
Greyscale
Claim 9
Regarding claim 9, Bigdeli in view of Reddy teaches the features of claim 8 and further teaches:
further comprising: training the robot system to autonomously update the simulation of the external environment based on multiple iterations of: (Reddy Page 3, Right Column, Method Overview “Our method takes observations of the environment and the user’s controls or inferred goal (when available) as input, and produces a high value action or control output that is as close as possible to the user’s control. We learn state-action values via Q-learning with neural network function approximation. In this section, we will describe how the agent combines user input with environmental observations, motivate and describe our choice of deep Q-learning for training the agent, and describe how the agent shares control with the user.”- The system is trained to autonomously update the simulation based on iterations of the steps taught in the independent claims.)
the receiving data collected by at least one image sensor of the robot body; the providing the data collected by at least one image sensor of the robot body to the at least one user of the tele-operation system the receiving simulation instructions from the at least one user of the tele- operation system based at least in part on the data collected by at least one image sensor of the robot body; and the updating the simulation of the external environment based on the simulation instructions. (Reddy – These are just repetitions of steps of the independent claims as the simulation progresses, so they are rejected for the same reasons as the corresponding steps in the independent claims.
Claim 10
Regarding claim 10, Bigdeli in view of Reddy teaches the features of claim 8 and further teaches:
wherein the at least one user of the tele-operation system includes at least one tele-operator of the robot system. (Reddy Fig. 4(b) illustrates that the user of the tele-operating system includes a tele-operator of the robot system)
PNG
media_image4.png
317
507
media_image4.png
Greyscale
Claim 15
Regarding claim 15, Bigdeli in view of Reddy teaches the features of claim 12 and further teaches:
wherein the agent includes a robot system including a robot body, and the at least one sensor of the agent includes at least one image sensor on-board the robot body. (Reddy Page 7, Right Column, Third Paragraph “To evaluate our method in a more realistic environment, we formulate a “perching” task for a real human flying a real quadrotor: land the vehicle on a level, square landing pad at some distance from the initial take-off position, such that the drone’s first-person camera is pointed at a specific object in the drone’s surroundings, without flying out of bounds or running out of time.” – The drone is a robot system with a visual sensor, a camera.)
Claims 16 and 20
Regarding claim 16, Bigdeli in view of Reddy teaches the features of claim 15 and further teaches:
further comprising: data and/or processor-executable instructions stored in the non-transitory processor-readable storage medium that, when executed by the at least one processor, cause the tele-operation system to: (Bigdeli – This is rejected based on the same rationale as presented in the independent claim for the CRM/memory.)
train the robot system to autonomously update to the simulation of the external environment based on multiple iterations of: (Reddy Page 3, Right Column, 3. Method Overview “Our method takes observations of the environment and the user’s controls or inferred goal (when available) as input, and produces a high value action or control output that is as close as possible to the user’s control. We learn state-action values via Q-learning with neural network function approximation. In this section, we will describe how the agent combines user input with environmental observations, motivate and describe our choice of deep Q-learning for training the agent, and describe how the agent shares control with the user.
receiving simulation instructions from the at least one user of the tele-operation system based at least in part on the data collected by at least one image sensor of the robot body; and updating the simulation of the external environment based on the simulation instructions. (Reddy – This is a repetition of steps in the independent claim and each step is rejected for the same reason as the corresponding step in the independent claim.)
Regarding claim 20, claim 16 teaches the features of claim 20, so claim 20 is rejected for at least the same reasons as claim 16.
Claims 21-23
Regarding claims 21-23, Bigdeli in view of Reddi teaches the features of claims 1, 12, and 18, and further teaches:
data and/or processor-executable instructions stored in the non-transitory processor-readable storage medium that, when executed by the at least one processor, cause the tele-operation system to: (Bigdeli “In other words, embodiments of the present disclosure may take the form of a computer program product on a computer-usable or computer-readable storage medium having computer-usable or computer-readable program code embodied in the medium for use by or in connection with an instruction execution system.”)
load a stored instance of the simulation of the external environment of the agent, wherein the modification to at least one object representation corrects a discrepancy between stored instance of the simulation and the actual external environment of the agent. (Bigdeli [0083] “observations may be accumulated throughout the mission to provide a temporally updated model. More specifically, at any time t, given the collective and integrated observations across the platform, area of changes, and location of moving objects are identified.” [0086] “Further, continual correction and learning through an active filter maybe a fourth architecture of the four architectures used in the processing/path planning of the disclosed device, as shown in FIG. 5. The continual correction and learning through an active filter uses the most recent observation and information collected and reported by the robots, drones, and sensors distributed in an environment is compared against the predicted state of the digital twin, or more specifically its projection onto observation space.” – The model continually updates iteratively over time for each time step.)
Claim 6: Bigdeli, Reddy, and Rosenfeld
Claims 2-6 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over US 2021/0349478 A1 to Bigdeli et al. (Bigdeli) in view of NPL: “Shared Autonomy vie Deep Reinforcement Learning” by Reddy et al. (Reddy) and of NPL: “Intelligent agent supporting human-multi-robot team collaboration” by Rosenfeld et al. (Rosenfeld).
Claim 6
Regarding claim 6, Reddy teaches the features of claim 1. Reddy appears to fail to explicitly teach, but Rosenfeld teaches:
wherein receiving simulation instructions from the at least one user of the tele-operation system includes receiving instructions that describe a new object representation for the simulation of the external environment, and (Rosenfeld Page 218, First Paragraph “In order to learn the effect that fallen products have on the robots’ productivity, we use between 0 and 3 products that are uniformly scheduled to fall on the warehouse floor during each session. As no human operator is present during the session, the fallen products caused robots to get stuck. Recall that the human worker can keep the robots from going through problem spots by marking these spots as ‘restricted spots’.” Page 216, Last Paragraph “We allow the human operator to mark the position of an obstacle and thereby instruct the robots to avoid the problematic spot. We refer to these instructions as “restricted cells” (more details are available in AppendixA).” Page 217, First Paragraph “A is the action set that the operator can take. Recall that Ais also the advice set from which the agent can propose advice. We define Aas the instantiations of the following 7 action schemes: “Robot i is waiting for your command”, “Unload item x at station y.”, “Complete the packing of order z.”, “Clear an obstacle from the floor.”, “Obstacle was detected – restrict its cell.”, “Clear a critical obstacle from the floor.”, and “Critical obstacle was detected – restrict its cell.”” Page 228, Third Paragraph “For example, during the robots’ work, products may fall in the work area, potentially causing the robots to get stuck. Once a product has fallen on the warehouse floor, the human worker can alert the robots so that they will avoid the problematic spot or s/he can remove the fallen product altogether. If the human worker decides to keep the robots from going through the problematic spot, then the robots will work out new, and possibly longer, paths. However, if the human worker decides to remove the product that fell on the warehouse floor, the robots will be restricted from approaching the area in order to avoid any danger to the human worker. The human worker decides when and where to restrict the robots’ movements. Namely, a spot can be restricted even if no obstacle is present (e.g., keep it clear for future use) and a spot may be unrestricted even if an obstacle is present (e.g., “taking a chance” that the robot will not collide with the obstacle). The command to restrict (unrestrict) a spot in the warehouse is executed instantaneously once given by the human worker. Note that by restricting the robots’ movements, the completion of the robots’ tasks may be impaired. In this work, we differ between two types of obstacles or restricted cells: critical, which prevents the completion of a task (e.g., if it blocks a robot’s only path to a packing station) and non-critical. The robots can also malfunction regardless of fallen products. For example, a technical problem could cause one of the robots to deviate from its course. In such a situation, the human worker will need to manually drive or guide the robot to its destination.” – The user can input information with regard to whether something is an obstacle and whether it is critical enough that the robots should avoid the obstacle in the environment. The user can also pick up the obstacle and inform the teleoperation system that the obstacle is no longer at that position. The labeling by the user of the obstacle or the space occupied thereby is a modification to an object representation in the environment.)
wherein updating the simulation of the external environment based on the simulation instructions includes applying the simulation instructions to add the new object representation to the simulation of the external environment, the new object representation corresponding to a real-world counterpart in the external environment characterized, at least in part, by the data collected by at least one sensor of the agent. (Rosenfeld Page 228, Third Paragraph “The human worker decides when and where to restrict the robots’ movements. Namely, a spot can be restricted even if no obstacle is present (e.g., keep it clear for future use) and a spot may be unrestricted even if an obstacle is present (e.g., “taking a chance” that the robot will not collide with the obstacle). The command to restrict (unrestrict) a spot in the warehouse is executed instantaneously once given by the human worker. Note that by restricting the robots’ movements, the completion of the robots’ tasks may be impaired. In this work, we differ between two types of obstacles or restricted cells: critical, which prevents the completion of a task (e.g., if it blocks a robot’s only path to a packing station) and non-critical. The robots can also malfunction regardless of fallen products. For example, a technical problem could cause one of the robots to deviate from its course. In such a situation, the human worker will need to manually drive or guide the robot to its destination.” – The user adds a new object to the environment. Page 228, Fourth Paragraph “When a robot needs the operator’s attention, its graphical representation blinks for 5sand the interface plays a prerecorded, human-voice message in the operator’s headset. The operator can view all of the robots’ requests and provide solutions using the alarm log. For the operator’s convenience, we maintain a first-come-first-serve request log indicating the (active) robot requests.” – The user’s designation of a restricted cell is responsive to an alert from the robot that there is an issue/obstacle.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claims to modify the shared autonomy of Reddy by the collaborative human-robot systems provided in Rosenfeld because the person of ordinary skill in the art would be motivated based on the expressed aim of Reddy to use human-in-the-loop reinforcement learning to learn a mapping from a combination environmental observation and user input to look to Rosenfeld to handle the robot-human collaboration using methods that are extensively tested and demonstrated to show improvement over other human-robot collaborative models. (Reddy Abstract “We use human-in-the-loop reinforcement learning with neural network function approximation to learn an end-to-end mapping from environmental observation and user input to agent action values, with task reward as the only form of supervision. This approach poses the challenge of following user commands closely enough to provide the user with real-time action feedback and thereby ensure high-quality user input, but also deviating from the user’s actions when they are suboptimal.”; Rosenfeld Abstract “In this article we propose a novel approach for utilizing automated advising agents in assisting an operator to better manage a team of multiple robots in complex environments. We introduce an advice provision methodology and exemplify its implementation using automated advising agents in two real-world human–multi-robot team collaboration tasks: the Search And Rescue (SAR) and the warehouse operation tasks. Our intelligent advising agents were evaluated through extensive field trials, with over 150 human operators using both simulated and physical mobile robots, and showed a significant improvement in the team’s performance.”)
Claims 11, 17, and 19: Bigdeli, Reddy, and Caren
Claim(s) 11, 17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over US 2021/0349478 A1 to Bigdeli et al. (Bigdeli) in view of NPL: “Shared Autonomy vie Deep Reinforcement Learning” by Reddy et al. (Reddy) and of NPL: “You Can Give A Robot A Paintbrush, But Does It Create Art” by Caren (Caren).
Claims 11, 17 and 19
Regarding claim 11, Bigdeli in view of Reddy teaches the features of claim 1. Bigdeli in view of Reddy does not appear to explicitly teach, but Bigdeli in view of Reddi and Caren teaches:
wherein the at least one user of the tele-operation system includes a plurality of tele-artists and the tele-operation system enables the plurality of tele-artists to concurrently update the simulation. (Caren Page 2, First-Second Paragraphs “The concept is simple: A user uploads a picture, and the robot (a.k.a."bitPaintr") paints it. The added bonus? Users can remotely jump into the software and join the robot to help it paint, whether that user is in the same room or 3,000 miles away.” – This system allows tele-artists to cooperatively paint with A robot. The changes made by the user and the robot would be updated in the simulation of Reddy.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claims to modify the human-robot cooperative guidance of a drone in Reddy by the human-robot cooperative guidance of a painting element in Arman because the person of ordinary skill in the art would be motivated based on the expressed desire in Reddy to address issues of high memory usage for inferring user intent by allowing added user feedback as to the intention as in Arman that receives the further user feedback and does not have to infer intent. (Reddy Conclusion “Several weaknesses and open questions remain to be addressed. Inferring user intent in general will require memory. Several existing techniques may accomplish this, including concatenating m previous frames with the current observation, or adding recurrent connections to the copilot policy architecture as in [11].”; Arman P “The concept is simple: A user uploads a picture, and the robot (a.k.a."bitPaintr") paints it. The added bonus? Users can remotely jump into the software and join the robot to help it paint, whether that user is in the same room or 3,000 miles away.” Page 3, First Paragraph “He likes how it's hard to distinguish whether bitPaintr's paintings, which start at $50, were created by a human or a robot. He says they "dance on the edge" of something in between.”)
Claims 17 and 19 recite substantially the same features and are rejected for at least the same reasons as claim 11.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
(From This Action)
US 2021/0157312 A1 to Cella et al. (Teaches essentially all of the features of claims, similarly to Bigdeli)
NPL: “Drone-based AI and 3D Reconstruction for Digital Twin Augmentation” by To et al. (Teaches using drone and AI fusion technology to map buildings)
NPL: “Distributed Situational Awareness in Robot Swarms” by Jones et al. (Teaches AI-based drone collective situational awareness in an environment)
NPL: “Distributed Data Storage and Fusion for Collective Perception in Resource-Limited Mobile Robot Swarms” by Majcherczyk et al. (Teaches collective distributed computing amongst drones for object classification in an environment)
(From Previous Actions)
NPL: “The Telegarden” by Goldberg et al. (Teaches human-robot collaborative planting methods)
NPL: “Active exploration and parameterized reinforcement learning applied to a simulated human-robot interaction task” by Khamassi et al. (Teaches using reinforcement learning to train a robot to collaborate with human in manufacturing)
NPL: “Reinforcement learning-based dynamic field exploration and reconstruction using multi-robot systems for environmental monitoring” by Lu et al. (Teaches reinforcement learning for robot exploration)
NPL: “The PumaPaint Project” by Stein (Teaches an early example of human-robot collaborative painting)
NPL: “Deep Reinforcement learning for real autonomous mobile robot navigation in indoor environments” by Surmann et al. (Teaches training robots to navigate using reinforcement learning)
NPL: “Remote Robot Control with Human-in-the-Loop Over Long Distances using Digital Twins” by Tsokalo et al. (Teaches robot control in a digital twin environment with a human-in-the-loop)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAY MICHAEL WHITE whose telephone number is (571) 272-7073. The examiner can normally be reached Mon-Fri 11:00-7:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ryan Pitaro can be reached at (571) 272-4071. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.M.W./Examiner, Art Unit 2188
/RYAN F PITARO/Supervisory Patent Examiner, Art Unit 2188
Prosecution Insights