Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 11/05/2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
On January 7, 2019, the USPTO released new examination guidelines setting forth a two-step inquiry for determining whether a claim is directed to non-statutory subject matter. According to the guidelines, a claim is directed to non-statutory subject matter if:
STEP 1: the claim does not fall within one of the four statutory categories of invention (process, machine, manufacture or composition of matter), or
STEP 2: the claim recites a judicial exception, e.g. an abstract idea, without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis:
STEP 2A (PRONG 1): Does the claim recite an abstract idea, law of nature, or natural phenomenon?
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
Using the two-step inquiry, it is clear that claim 15 is directed toward non-statutory subject matter, as shown below:
STEP 1: Does claim 15 fall within one of the statutory categories? Claim 15 is directed towards a method, and as such falls within one of the statutory categories.
STEP 2A (PRONG 1): Is the claim directed to a law of nature, a natural phenomenon or an abstract idea? Yes, claim 15 is directed to mental processes.
With regard to STEP 2A (PRONG 1), the guidelines provide three groupings of subject matter that are considered abstract ideas:
Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations;
Certain methods of organizing human activity – fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions); and
Mental processes – concepts that are practicably performed in the human mind (including an observation, evaluation, judgment, opinion).
The method of claim 15 contains a mental process that can be practicably performed in the human mind and, therefore, an abstract idea. A human can determine a sequence of robot motion that shall be performed based on a human input. As an example if a human input is a circular gesture and a recorded motion matches the gesture, a human can determine that a robot should wipe a surface.
Therefore, limitation of “determining,
Furthermore, “displaying, on a display, during a second time period that is subsequent to the first time period, an augmented reality graphical user interface including a graphical representation of the recorded human motions that is superimposed on the environment such that the graphical representation appears within the environment at the plurality of recorded positions of the human in the environment” is collecting information, analyzing it and displaying results of collection and analysis. Therefore, is also considered an abstract idea. See Electric Power Group v. Alstom, S.A., 830 F.3d 1350, 1353-54, 119 USPQ2d 1739, 1741-42 (Fed. Cir. 2016).
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application? No, the claim does not recite additional elements that integrate the judicial exception into a practical application.
With regard to STEP 2A (prong 2), whether the claim recites additional elements that integrate the judicial exception into a practical application, the guidelines provide the following exemplary considerations that are indicative that an additional element (or combination of elements) may have integrated the judicial exception into a practical application:
an additional element reflects an improvement in the functioning of a computer, or an improvement to other technology or technical field;
an additional element that applies or uses a judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition;
an additional element implements a judicial exception with, or uses a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim;
an additional element effects a transformation or reduction of a particular article to a different state or thing; and
an additional element applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception.
While the guidelines further state that the exemplary considerations are not an exhaustive list and that there may be other examples of integrating the exception into a practical application, the guidelines also list examples in which a judicial exception has not been integrated into a practical application:
an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea;
an additional element adds insignificant extra-solution activity to the judicial exception; and
an additional element does no more than generally link the use of a judicial exception to a particular technological environment or field of use.
Claim 15 does not recite any of the exemplary considerations that are indicative of an abstract idea having been integrated into a practical application. Also, as noted above, merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea is indicative that the judicial exception has not been integrated into a practical application.
The steps identified in step 2A prong 1 are performed by a processor. Thus, it is clear that the abstract idea is merely implemented on a computer, which is indicative of the abstract idea having not been integrated into a practical application. See MPEP 2106.05(f).
Furthermore, the steps performed by controller above are performed “recording, with at least one sensor, during a first time period, human motions of a human as the human demonstrates the human-robot collaborative task in an environment, the recorded human motions including a plurality of recorded positions of the human in the environment over a period of time”, this is mere data gathering, hence an insignificant extra solution activity. See MPEP 2106.05(g).
Furthermore, “storing, in a memory, the recorded human motions and the sequence of robot motions to be performed by the robot” is storing information in memory, and is an insignificant extra solution activity. See Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015).
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No, the claim does not recite additional elements that amount to significantly more than the judicial exception.
With regard to STEP 2B, whether the claims recite additional elements that provide significantly more than the recited judicial exception, the guidelines specify that the pre-guideline procedure is still in effect. Specifically, that examiners should continue to consider whether an additional element or combination of elements:
adds a specific limitation or combination of limitations that are not well-understood, routine, conventional activity in the field, which is indicative that an inventive concept may be present; or
simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, which is indicative that an inventive concept may not be present.
Claim 15 does not recite any specific limitation or combination of limitations that are not well-understood, routine, conventional (WURC) activity in the field. Limitations identified as “apply it” in step 2A qualify as apply it in step 2B as well.
With respect to “recording, with at least one sensor, during a first time period, human motions of a human as the human demonstrates the human-robot collaborative task in an environment, the recorded human motions including a plurality of recorded positions of the human in the environment over a period of time”, this is receiving or transmitting data. And as such has been recognized as well-understood routine and conventional. See MPEP 2106.05(d).
Furthermore, “storing, in a memory, the recorded human motions and the sequence of robot motions to be performed by the robot” is storing information in memory, and is recognized a well understood routing and conventional activity. See MPEP 2106.05(d).
CONCLUSION
Thus, since claim 15 is: (a) directed toward an abstract idea, (b) does not recite additional elements that integrate the judicial exception into a practical application, and (c) does not recite additional elements that amount to significantly more than the judicial exception, it is clear that claim 15 is directed towards non-statutory subject matter.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Thackston (US 20200130190) in view of Guerin (US 20160257000).
For claim 16, Thackston reaches: A method for operating a robot to collaborate with a human to perform a human-robot collaborative task, the method comprising (Abstract, disclosing monitoring action of operator and controlling the robotic device to perform the intended action), the method comprising:
storing, in a memory, (i) previously recorded human motions including a plurality of recorded positions of a human in an environment over a period of time and (ii) a sequence of robot motions to be performed by the robot in concert with performance of human motions that match the previously recorded human motions ([0029], disclosing gesture tracking system. [0026-0028], disclosing Based on the history of events (e.g., placing knives and forks on the table) i.e., previously recorded human motions, the system may identify the drawer of spoons and infer that the spoons are the intended target. Based on the history of events (e.g., placing knives and forks on the table), the system may identify the drawer of spoons and infer that the spoons are the intended target. [0002, disclosing controlling the robotic device to perform the intended action);
detecting, with at least one sensor, a real-time performance of human motions that match the recorded human motions ([0029], disclosing gesture tracking system. [0024], disclosing intended target may be inferred based on observations of the operator with respect to operating the robotic device. That is, one or more of a gaze tracking system, gesture tracking system, motion tracking system, and/or other action tracking system may monitor the operator's actions. The monitored actions may be used to identify the intended target. The target may be an object or a spatial location in an environment of the robotic device)
generating, with the processor, and transmitting to the robot, with a transceiver, a plurality of commands configured to operate the robot to perform the sequence of robot motions in concert with the real-time performance of human motions that match corresponding to the recorded human motions ([0031], disclosing actions of the robotic device 106 may be controlled via the motion controllers 104. For example, the motion controllers 104 capture the operator's 100 gestures and the robotic device 106 mimics the captured gestures)
during the performance of human motions that match the recorded human motions, displaying, in an augmented reality graphical user interface on a display of an augmented reality device, a graphical representation of a portion of the recorded human motions that have not yet been performed, which is superimposed on the environment such that the graphical representation appears within the environment at the plurality of recorded positions of the human in the environment ([0031], disclosing actions of the robotic device 106 may be controlled via the motion controllers 104. For example, the motion controllers 104 capture the operator's 100 gestures and the robotic device 106 mimics the captured gestures. [0026], disclosing learning actions based on previous interactions. [0026-0027], disclosing system identifies spoons are intended target i.e. operator has not yet performed pick and place motion of spoon. And system may request the operator to confirm that the spoons are the intended target. Prior to controlling the robotic device to place the spoons on the table, the system may propose the intended action to the operator for approval. [0029-0030], disclosing a vision system. Confirmation and proposal is necessarily performed through display. [0035], disclosing the on-screen indication area 254 provides notifications of inferred targets and/or inferred actions i.e., actions not yet performed. [0039-0040], disclosing system illustrates operator to confirm intended target. A confirmation dialog provides a list and intended target is also highlighted)
Although displaying a list of intended target and highlighting intended target is augmented reality. As an alternative, Guerin teaches augmented reality for collaborative robot control ([0041], disclosing Augmented reality (AR) and virtual reality (VR) for robotic systems are both meant to make interaction with robots more flexible by combining digital information in a more intuitive manner with a physical system. AR involves projecting virtual information on top of a real physical environment, while VR involves displaying a representation of the physical system in a virtual digital environment. And [0034], disclosing the user may alternate between various perspectives such as egocentric (first person) and exocentric (third person) perspectives to control the robot directly, train or program motions for the robot to make later, or interactively collaborate with the robot during a task execution).
As Thackston and Guerin are analogous arts as they are in same field of endeavor i.e., training robots, it would have been obvious to one having ordinary skill in the art before effective filing date of claimed invention to modify art of Thackston to utilize augmented reality display as taught by Guerin to make robot interaction more intuitive for operator.
For claim 17, modified Thackston teaches: The method of claim 16, the detecting the performance of human motions that match the recorded human motions further comprising:
recording, with the at least one sensor, a real-time position of the one of the human ([0031], disclosing actions of the robotic device 106 may be controlled via the motion controllers 104. For example, the motion controllers 104 capture the operator's 100 gestures and the robotic device 106 mimics the captured gestures. [0026], disclosing learning actions based on previous interactions) and a further human; and
comparing, with the processor, the real-time position to the plurality of recorded positions of the human in the recorded human motions ([0060], disclosing identified objects may be output to an intent module 410 to infer an intended target. That is, based on the identified objects and one or more operator actions received via the transceiver 416, the intent module 410 infers an intended target for the robotic device. [0026], disclosing operator may then gesture towards a drawer with spoons. Based on the history of events (e.g., placing knives and forks on the table), the system may identify the drawer of spoons and infer that the spoons are the intended target. [0037], disclosing inference of intended target may be based on a probabilistic determination. In one configuration, the probabilistic determination is based on previous actions and/or a current action (e.g., gesture, gaze, motion, etc.). Gesture of operator includes position of operator and inference of intended target based on previous gesture and current gesture necessarily compares previous gesture with current gesture).
For claim 18, modified Thackston teaches: The method of claim 16, wherein:
the recorded human motions are grouped into a plurality of motion groups, each including a respective subset of the recorded human motions ([0028], disclosing action may be based on a learned action or a library of actions associated with the object (e.g., the spoon). For example, for a spoon, the library of actions may include setting the spoon on a table, using the spoon to scoop a surrounding object, or putting the spoon in a drawer for storage. The system may also use a combination of learned and modeled actions, such as adjusting the available modeled actions based upon a previously performed actions within a time frame);
the sequence of robot motions to be performed by the robot includes a respective set of robot motions to be performed by the robot in concert with a performance of human motions that match each motion group of the plurality of motion groups ([0069], disclosing for a mop, the library of actions may include putting the mop in a bucket of water, mopping the floor, squeeze the mop, or put the mop back in the closet. [0028], disclosing action may be based on a learned action or a library of actions associated with the object (e.g., the spoon). For example, for a spoon, the library of actions may include setting the spoon on a table, using the spoon to scoop a surrounding object, or putting the spoon in a drawer for storage); and
each motion group in the plurality of motion groups is designated as one of (i) a first type in which the respective set of robot motions is to be performed by the robot in synchronization with a progress of a performance of human motions that match the respective motion group ([0068], disclosing the operator controls the robotic device to fill a bucket with soap and water. The operator may then gesture towards a mop. Based on the history of events (e.g., fill a bucket with soap and water), the system may identify the mop as the intended target. Furthermore, the system may infer that the operator intends to place the mop in the bucket and then mop the floor), and
(ii) a second type in which the respective set of robot motions is to be performed by the robot responsive to a completion of a performance of human motions that match the respective motion group ([0069], disclosing for a mop, the library of actions may include putting the mop in a bucket of water, mopping the floor, squeeze the mop, or put the mop back in the closet. The intent-based system may use a combination of the learned and modeled approach).
First type is when intended action is already learned second type is when robot follows human movements.
For claim 19, modified Thackston teaches: The method of claim 18 further comprising:
detecting, with the at least one sensor, a plurality of real-time positions of one of (i) the human and (ii) a further human during a performance of human motions that match a respective motion group of the plurality of motion groups that is designated as the first type ([0031], disclosing actions of the robotic device 106 may be controlled via the motion controllers 104. For example, the motion controllers 104 capture the operator's 100 gestures and the robotic device 106 mimics the captured gestures);
mapping, with the processor, the plurality of real-time positions to the respective subset of recorded human motions of the respective motion group ([0031], disclosing actions of the robotic device 106 may be controlled via the motion controllers 104. For example, the motion controllers 104 capture the operator's 100 gestures and the robotic device 106 mimics the captured gestures. Gestures are necessarily mapped in real time to be accurately mimicked); and
based on the mapping of the plurality of real-time positions, generating, with the processor, and transmitting to the robot, with a transceiver, at least one command configured to operate the robot to perform the respective set of robot motions that match the respective motion group in synchronization with a progress of the performance of human motions that match the respective motion group ([0031], disclosing actions of the robotic device 106 may be controlled via the motion controllers 104. For example, the motion controllers 104 capture the operator's 100 gestures and the robotic device 106 mimics the captured gestures. Gestures are necessarily mapped in real time to be accurately mimicked. Control of robotic device requires control command to be sent to robot).
Claim 20 recites limitations similar in scope to claim 19, hence is similarly rejected. Furthermore, second type is when robot follows human motion ([0069], disclosing for a mop, the library of actions may include putting the mop in a bucket of water, mopping the floor, squeeze the mop, or put the mop back in the closet. The intent-based system may use a combination of the learned and modeled approach. [0031], disclosing robot mimics tracked movements).
Furthermore, although examiner has rejected both claims 19 and 20, under BRI one claim is optional because:
Claim 18 recites “wherein each motion group in the sequence of motion groups is designated as one of (i) a first type… and (ii) a second type”, hence only one type is necessary.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-15 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-11 of U.S. Patent No. 12145267. Although the claims at issue are not identical, they are not patentably distinct from each other because:
Claim of Instant Application 18937756
Claim of US Patent 12145267
1. A method for authoring a human-robot collaborative task in which a robot collaborates with a human, the method comprising: recording, with at least one sensor, during a first time period, human motions of a human as the human demonstrates the human-robot collaborative task in an environment, the recorded human motions including a plurality of recorded positions of the human in the environment over a period of time; displaying, on a display, during a second time period that is subsequent to the first time period, an augmented reality graphical user interface including a graphical representation of the recorded human motions that is superimposed on the environment such that the graphical representation appears within the environment at the plurality of recorded positions of the human in the environment; determining, with a processor, based on user inputs received from the human during the second time period, a sequence of robot motions to be performed by a robot in concert with a performance of human motions that match the recorded human motions; and generating, with the processor, and transmitting to the robot, with a transceiver, during a third time period that is subsequent to the second time period, a plurality of commands configured to operate the robot to perform the sequence of robot motions in concert with the performance of human motions that match the recorded human motions.
1. A method for authoring a human-robot collaborative task in which a robot collaborates with a human, the method comprising:recording, with at least one sensor, during a first time period, human motions of a human as the human demonstrates the human-robot collaborative task in an environment, the recorded human motions including a plurality of recorded positions of the human in the environment over a period of time;
displaying, on a display, during a second time period that is subsequent to the first time period, an augmented reality graphical user interface including a graphical representation of the recorded human motions that is superimposed on the environment such that the graphical representation appears within the environment at the plurality of recorded positions of the human in the environment;
displaying, in the graphical user interface on the display, during the second time period while the graphical representation of the recorded human motions is displayed, a virtual representation of the robot that is superimposed on the environment and which can be manipulated by the human by providing user inputs;
receiving, via a user interface, during the second time period, user inputs defining manipulations of the virtual representation of the robot, the manipulations being graphically represented by the virtual representation of the robot;
determining, with a processor, during the second time period, a sequence of robot motions to be performed by a robot in concert with a performance of human motions that match the recorded human motions, based on the manipulations of the virtual representation of the robot;
storing, in a memory, during the second time period, the recorded human motions and the sequence of robot motions to be performed by the robot;
detecting, during a third time period that is subsequent to the second time period, the performance of human motions that match the recorded human motions by one of the human and a further human, the detecting including (i) recording, with the at least one sensor, a real-time position of the one of the human and the further human and (ii) comparing, with the processor, the real-time position to the plurality of recorded positions of the human in the recorded human motions; and
generating, with the processor, and transmitting to the robot, with a transceiver, during the third time period, a plurality of commands configured to operate the robot to perform the sequence of robot motions in concert with the performance of the human motions that match the recorded human motions.
2. The method of claim 1 further comprising: detecting, during the third time period, with the least one sensor, the performance of human motions that match the recorded human motions by one of the human and a further human.
1… detecting, during a third time period that is subsequent to the second time period, the performance of human motions that match the recorded human motions by one of the human and a further human, the detecting including (i) recording, with the at least one sensor, a real-time position of the one of the human and the further human and (ii) comparing, with the processor, the real-time position to the plurality of recorded positions of the human in the recorded human motion
3. The method of claim 2, the detecting the performance of human motions that match the recorded human motions further comprising: recording, with the at least one sensor, a real-time position of the one of the human and the further human; and comparing, with the processor, the real-time position to the plurality of recorded positions of the human in the recorded human motions.
1… detecting, during a third time period that is subsequent to the second time period, the performance of human motions that match the recorded human motions by one of the human and a further human, the detecting including (i) recording, with the at least one sensor, a real-time position of the one of the human and the further human and (ii) comparing, with the processor, the real-time position to the plurality of recorded positions of the human in the recorded human motions;
4. The method of claim 1 further comprising: during the performance of human motions that match the recorded human motions: displaying, in the graphical user interface on the display, a virtual representation of a portion of the recorded human motions that have not yet been performed by the one of (i) the human and (ii) the further human, which is superimposed on the environment based on the plurality of recorded positions of the human in the environment.
4. The method of claim 1 further comprising:during the performance of human motions that match the recorded human motions by the one of (i) the human and (ii) the further human:
displaying, in the graphical user interface on the display, a virtual representation of a portion of the recorded human motions that have not yet been performed by the one of (i) the human and (ii) the further human, which is superimposed on the environment based on the plurality of recorded positions of the human in the environment
5. The method of claim 1, the recording the human motions further comprising: displaying, in the graphical user interface on the display, a virtual representation of an object that is superimposed on the environment and which can be manipulated by the human by providing user inputs; and recording, with the at least one sensor, the human motions of the human as the human demonstrates the human-robot collaborative task in the environment, the recorded human motions including the plurality of recorded positions of the human in the environment over the period of time and manipulations of the virtual representation of the object that are performed by the human by providing user inputs during the period of time.
10. The method of claim 1, wherein the sequence of robot motions to be performed by a robot includes movements between a plurality of positions of the robot in the environment over a period of time, the displaying the graphical user interface further comprising:
displaying, in the graphical user interface on the display, a plurality of graphical representations of the robot, each superimposed on the environment at a respective one of the plurality of positions of the robot in the environment.
6. The method of claim 1, the determining the sequence of robot motions to be performed by the robot further comprising: displaying, in the graphical user interface on the display, during the second time period while the graphical representation of the recorded human motions is displayed, a virtual representation of the robot that is superimposed on the environment and which can be manipulated by the human by providing user inputs; receiving, via a user interface, during the second time period, user inputs defining manipulations of the virtual representation of the robot, the manipulations being graphically represented by the virtual representation of the robot; and determining, with the processor, the sequence of robot motions to be performed by the robot based on the manipulations of the virtual representation of the robot.
10. The method of claim 1, wherein the sequence of robot motions to be performed by a robot includes movements between a plurality of positions of the robot in the environment over a period of time, the displaying the graphical user interface further comprising:
displaying, in the graphical user interface on the display, a plurality of graphical representations of the robot, each superimposed on the environment at a respective one of the plurality of positions of the robot in the environment.
1…receiving, via a user interface, during the second time period, user inputs defining manipulations of the virtual representation of the robot, the manipulations being graphically represented by the virtual representation of the robot;
7. The method of claim 1, the determining the sequence of robot motions to be performed by the robot further comprising: displaying, in the graphical user interface on the display, a virtual representation of an object, which is superimposed on the environment and which can be manipulated by the virtual representation of the robot by the human by providing user inputs, wherein the sequence of robot motions include manipulations to be performed of a real-world object corresponding to the virtual representation of the object.
4. The method of claim 1, the determining the sequence of robot motions to be performed by the robot further comprising:displaying, in the graphical user interface on the display, a virtual representation of an object, which is superimposed on the environment and which can be manipulated by the virtual representation of the robot by the human by providing user inputs,
wherein the sequence of robot motions include manipulations to be performed of a real- world object corresponding to the virtual representation of the object.
8. The method of claim 1 further comprising: grouping, with a processor, respective subsets of the recorded human motions into a plurality of motion groups based on user inputs received from the human; and determining, with the processor, based on user inputs received from the human, for each respective motion group of the plurality of motion groups, a respective set of robot motions to be performed by the robot in concert with a performance of human motions that match the respective motion group, wherein the sequence of robot motions to be performed by the robot is formed by the set of robot motions to be performed by the robot for each of the plurality of motion groups.
5. The method of claim 1 further comprising:grouping, with a processor, respective subsets of the recorded human motions into a plurality of motion groups based on user inputs received from the human; and
determining, with the processor, based on user inputs received from the human, for each respective motion group of the plurality of motion groups, a respective set of robot motions to be performed by the robot in concert with a performance of human motions that match the respective motion group,
wherein the sequence of robot motions to be performed by the robot is formed by the set of robot motions to be performed by the robot for each of the plurality of motion groups.
9. The method of claim 8 further comprising: receiving, for each respective motion group of the plurality of motion groups, a user input from the human designating the respective motion group as one of: (i) a first type in which the respective set of robot motions is to be performed by the robot in synchronization with a progress of a performance of human motions that match the respective motion group; and (ii) a second type in which the respective set of robot motions is to be performed by the robot responsive to a completion of a performance of human motions that match the respective motion group.
6. The method of claim 5 further comprising:receiving, for each respective motion group of the plurality of motion groups, a user input from the human designating the respective motion group as one of:
(i) a first type in which the corresponding set of robot motions is to be performed by the robot in synchronization with a progress of a performance of human motions that match the respective motion group; and
(ii) a second type in which the corresponding set of robot motions is to be performed by the robot responsive to a completion of a performance of human motions that match the respective motion group.
10. The method of claim 9 further comprising: detecting, with the at least one sensor, a plurality of real-time positions of one of (i) the human and (ii) a further human during a performance of human motions that match a respective motion group of the plurality of motion groups that is designated as the first type; mapping, with the processor, the plurality of real-time positions to the respective subset of recorded human motions of the respective motion group; and based on the mapping of the plurality of real-time positions, generating, with the processor, and transmitting to the robot, with the transceiver, at least one command configured to operate the robot to perform the respective set of robot motions that match the respective motion group in synchronization with a progress of the performance of human motions that match the respective motion group.
7. The method of claim 6 further comprising:detecting, with the at least one sensor, a plurality of real-time positions of one of (i) the human and (ii) a further human during a performance of human motions that match a respective motion group of the plurality of motion groups that is designated as the first type;
mapping, with the processor, the plurality of real-time positions to the respective subset of recorded human motions of the respective motion group; and
based on the mapping of the plurality of real-time positions, generating, with the processor, and transmitting to the robot, with the transceiver, at least one command configured to operate the robot to perform the respective set of robot motions that match the respective motion group in synchronization with a progress of the performance of human motions that match the respective motion group.
11. The method of claim 9 further comprising: detecting, with the at least one sensor, a plurality of real-time positions of one of the human and a further human during a performance of human motions that match a respective motion group of the plurality of motion groups that is designated as the second type; mapping, with the processor, the plurality of real-time positions to the respective subset of recorded human motions of the respective motion group; and based on the mapping of the plurality of real-time positions, generating, with the processor, and transmitting to the robot, with a transceiver, at least one command configured to operate the robot to perform the respective set of robot motions that match the respective motion group responsive to a completion of the performance of human motions that match the respective motion group.
7. The method of claim 6 further comprising:detecting, with the at least one sensor, a plurality of real-time positions of one of (i) the human and (ii) a further human during a performance of human motions that match a respective motion group of the plurality of motion groups that is designated as the second type;
mapping, with the processor, the plurality of real-time positions to the respective subset of recorded human motions of the respective motion group; and
based on the mapping of the plurality of real-time positions, generating, with the processor, and transmitting to the robot, with a transceiver, at least one command configured to operate the robot to perform the respective set of robot motions that match the respective motion group responsive to a completion of the performance of human motions that match the respective motion group.
12. The method of claim 1, the displaying the graphical user interface further comprising: displaying, in the graphical user interface on the display, a plurality of graphical representations of the human, each superimposed on the environment at a respective one of the plurality of recorded positions of the human in the environment.
9. The method of claim 1, the displaying the graphical user interface further comprising:displaying, in the graphical user interface on the display, a plurality of graphical representations of the human, each superimposed on the environment at a respective one of the plurality of recorded positions of the human in the environment.
13. The method of claim 1, wherein the sequence of robot motions to be performed by a robot includes movements between a plurality of positions of the robot in the environment over a period of time, the displaying the graphical user interface further comprising: displaying, in the graphical user interface on the display, a plurality of graphical representations of the robot, each superimposed on the environment at a respective one of the plurality of positions of the robot in the environment.
10. The method of claim 1, wherein the sequence of robot motions to be performed by a robot includes movements between a plurality of positions of the robot in the environment over a period of time, the displaying the graphical user interface further comprising:displaying, in the graphical user interface on the display, a plurality of graphical representations of the robot, each superimposed on the environment at a respective one of the plurality of positions of the robot in the environment.
14. The method of claim 1 further comprising: displaying, in the graphical user interface on the display, an animation in which a virtual representation of the human performs the recorded human motions and a virtual representation of the robot performs the sequence of robot motions in concert with the performance of the recorded human motions by the virtual representation of the human.
12. The method of claim 1 further comprising:displaying, in the graphical user interface on the display, an animation in which a virtual representation of the human performs the recorded human motions and a virtual representation of the robot performs the sequence of robot motions in concert with the performance of the recorded human motions by the virtual representation of the human.
Claim 15 recites limitations similar in scope to claim 1, hence is similarly rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Rod (US-20210069894) teaches of robot performing a task in conjunction with a change of a posture of an operator. See abstract and figure 1.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ARSLAN AZHAR whose telephone number is (571)270-1703. The examiner can normally be reached Mon-Fri 7:30 - 5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wade Miles can be reached at (571) 270-7777. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ARSLAN AZHAR/Examiner, Art Unit 3656
Prosecution Insights