Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This communication is responsive to Application No. 19/536,407 and the amendments filed on 7/30/2026.
3. Claims 1-20 are presented for examination.
Response to Arguments
4. Applicant’s arguments, see page 8, filed 7/30/2026, with respect to the objection to claims 16 and 17 for minor informalities have been fully considered and are persuasive. The objection of 4/30/3036 has been withdrawn.
5. Applicant's arguments filed 4/30/2026 with respect to the rejection of claims 1-20 under 35 U.S.C. 102 and/or 35 U.S.C. 103 have been fully considered but they are not persuasive.
Regarding independent claim 1, the Applicant argues that reference US 20230234231 A1 to Dong fails to anticipate all of claim 1. The Applicant argues that Dong does not identically disclose all of the limitations of claim 1. Specifically, the Applicant argues that Dong fails to properly teach the limitation of “accessing at least one instance of human data, each instance of the at least one instance of human data comprising data collected by a third at least one sensor positioned at a human and representing the human performing the task,” recited in lines 9-11 of claim 1, claiming that the Examiner has erroneously mapped the human operator of Dong to be both of the pilot and the human recited in claim 1. The Applicant contends that the pilot and the human are separate characterized concepts. However, the Examiner respectfully disagrees with all of these arguments, in which will be described below.
The invention of Dong teaches a teleoperation assist device where a plethora of sensor data is acquired in order for a robot end effector to perform a task, wherein the sensor data includes information on the motion of an operator (See abstract of Dong). The operator of Dong is a human operator ([0040] of Dong) who wears multiple sensors to teleoperate the robot ([0029] and Figure 1 of Dong).
The Applicant argues that the pilot and human of current claim 1 are separate characterized concepts, to which the Examiner interprets the Applicant as stating that the pilot and the human recited in claim 1 are not the same entity (i.e., different people). However, the Examiner respectfully disagrees. Under the Examiner’s broadest reasonable interpretation (BRI), claim 1 fails to distinguish that the pilot and human are two different entities. Claim 1 may introduce “a pilot” (line 5) and “a human” (lines 10-11), but no further distinction is made that the pilot and the human are separate. Under the Examiner’s BRI, the pilot and the human may be one in the same.
The Applicant’s own specification provides support that the pilot and the human may be the same entity. For example, paragraph [053] of the specification of the instant application (hereinafter referred to as the specification) recites “whereby a human pilot performs operations which are captured by the teleoperation system.” Paragraph [063] of the specification states “detects real physical actions performed by a human pilot 182” and “the data collected by any of the above sensors which captures movement of the human pilot is referred to herein as “pilot data”.” Pilot 182 shown in Figure 1 of the drawings also appears to be a human. The specification and drawings make clear that the pilot may be a human. Specifically, a human who wears various sensors and can control the motion of the robot. This definition of the pilot exactly matches the human operator of Dong.
Paragraph [066] of the specification defines the human, stating “In particular, Figure 2A illustrates a variety of sensors which can be worn by human 201, useful for collecting human data used to train or refine a foundation model as discussed later with reference to Figure 4. Throughout this disclosure, “human data” generally refers to data which represents at least one human (e.g. data representing movement or action of a human to perform a task). … That is, the sensors worn by the human 201 capture data which represents or approximates what human 201 experiences (e.g. sees or feels).” Figures 2A, 3, and 6 of the drawings also show the human 201. This definition of the human also exactly matches the human operator of Dong.
The specification has defined that the pilot may be human, and has defined that both of the pilot and the human wear various types of sensors that the robot uses to perform a task. These concepts are also present in claim 1.
The Examiner also notes that people may be referred to as multiple titles. For example, a banker may be referred to as “a banker,” “a human,” “a stamp collecting enthusiast,” etc.
Whether taking the claim on its own or factoring in the definitions from the specification, the pilot and the human may be of the same singular entity. For these reasons, the Examiner submits that Dong anticipates all of the limitations of claim 1, and that the previous rejection of claim 1 under 35 U.S.C. 102(a)(1) mailed 4/30/2026 is being maintained.
Regarding independent claim 11, as this claim contains similar limitations to claim 1, is still rejected, in which will be described later.
Regarding dependent claims 2-10 and 12-20, as all of these claims depend from either claims 1 or 11, are still rejected, in which will be described later.
Claim Rejections - 35 USC § 102
6. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
7. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
8. Claim(s) 1, 2, 3, and 4 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Dong et al. (US 20230234231 A1 hereinafter Dong).
Regarding Claim 1, Dong teaches a method comprising: accessing at least one instance of teleoperation data ([0029] via “FIG. 1 is a diagram schematically illustrating teleoperation of a robot.”), each instance of the at least one instance of teleoperation data comprising:
pilot data, the pilot data collected by a first at least one sensor of a piloted teleoperation system where a pilot provides at least one input to cause a robot to perform a task ([0041] via “The finger movement sensor 111 is, for example, a six-axis sensor, an acceleration sensor, or a vibration sensor. The finger movement sensor 111 is provided, for example, in the controller 61. The finger movement sensor 111 detects a fingertip command value including a position of each finger of the human operator and movement of each finger and outputs the detected fingertip command value to the teleoperation assist device 3.”), ([0049] via “The end effector operation determining unit 33 determines an amount of operation of the end effector 222 using the detection information detected by the finger movement sensor 111 and the taxonomy information set by the parameter setting unit 32.”), (Note: The Examiner interprets the finger movement sensor 111 of Dong as the first at least one sensor, and the position and movement of each finger of the human operator of Dong as the at least one instance of pilot data.);
robot data, the robot data collected by a second at least one sensor of the robot and representing operation of the robot which performs the task ([0029] via “In a work space, an environmental sensor 71 (71a, 71b) is installed. The environmental sensor 71 may be attached to a robot 2. … The environmental sensor 71 includes, for example, an RGB camera and a depth sensor.”), ([0044] via “The environmental status determining unit 12 acquires environment information in which the robot 2 is operated. The environmental status determining unit 12 recognizes a position of a target object, a posture of the target object, a type of the target object, a shape of the target object, and the like using detection information detected by the environmental sensor 71 and outputs the recognized information on the target object to the teleoperation assist device 3. The environmental status determining unit 12 recognizes a target object and a position, a size, a shape, or the like of an obstacle when the obstacle is interposed between the target object and the end effector 222 using the acquired information.”), (Note: The Examiner interprets the environmental sensor 71 of Dong as the second at least one sensor.);
accessing at least one instance of human data, each instance of the at least one instance of human data comprising data collected by a third at least one sensor positioned at a human and representing the human performing the task ([0042] via “The gaze sensor 112 is a sensor that detects movement of a gaze of the human operator. The gaze sensor 112 is provided, for example, in the HMD 51. The gaze sensor 112 detects movement of a gaze of the human operator and outputs the detected gaze information to the teleoperation assist device 3.”), (Note: The Examiner interprets the gaze sensor 112 of Dong as the third at least one sensor, and the gaze of the human operator as the at least one instance of human data.); and
training at least one model based on the at least one instance of teleoperation data and the at least one instance of human data ([0047] via “The intention estimating unit 31 inputs the acquired information on a target object and the acquired detection information on the movement of the fingers of the human operator, and the acquired gaze information to a trained model 311 and estimates a task and a target (a target object) which are intended by the operator.”), ([0077] via “At the time of training, the model 311 is trained by inputting training data and a position of a finger, gaze information, a position of a wrist, rotation information of the wrist, and environmental information to the model 311. The environmental information includes, for example, a captured image and information obtained by recognizing a target object (for example, a position, a shape, and a size). The training data includes task information and target information.”), (Note: See Figure 7 of Dong as well.).
Regarding Claim 2, Dong teaches the method of claim 1, wherein training at least one model based on the at least one instance of teleoperation data and the at least one instance of human data comprises: training a first model based on the at least one instance of teleoperation data ([0077] via “At the time of training, the model 311 is trained by inputting training data and a position of a finger, gaze information, a position of a wrist, rotation information of the wrist, and environmental information to the model 311. The environmental information includes, for example, a captured image and information obtained by recognizing a target object (for example, a position, a shape, and a size).”), (Note: See Figure 3 of Dong as well.); and
refining the first model based on the at least one instance of human data to produce a second model ([0079] via “The model 311 is, for example, a network such as a deep neutral network (DNN), a recurrent neural network (RNN), a convolutional neural network (CNN), a long/short-term memory (LSTM), or a generative adversarial network (GAM).”), ([0094] via “(Step S3) The intention estimating unit 31 estimates a task and a target which are intended by the operator by inputting the acquired operator information and the environmental information to the trained model 311.”), ([0098] via “(Step S7) The teleoperation assist device 3 determines whether the task has been completed. … When it is determined that the task has been completed (Step S7: NO), the teleoperation assist device 3 returns the routine to Step S1.”), (Note: See Figure 7 of Dong as well.).
Regarding Claim 3, Dong teaches the method of claim 1, wherein training at least one model based on the at least one instance of teleoperation data and the at least one instance of human data comprises: training a first model based on the at least one instance of human data ([0077] via “At the time of training, the model 311 is trained by inputting training data and a position of a finger, gaze information, a position of a wrist, rotation information of the wrist, and environmental information to the model 311. The environmental information includes, for example, a captured image and information obtained by recognizing a target object (for example, a position, a shape, and a size).”), (Note: See Figure 3 of Dong as well.); and
refining the first model based on the at least one instance of teleoperation data to produce a second model ([0079] via “The model 311 is, for example, a network such as a deep neutral network (DNN), a recurrent neural network (RNN), a convolutional neural network (CNN), a long/short-term memory (LSTM), or a generative adversarial network (GAM).”), ([0094] via “(Step S3) The intention estimating unit 31 estimates a task and a target which are intended by the operator by inputting the acquired operator information and the environmental information to the trained model 311.”), ([0098] via “(Step S7) The teleoperation assist device 3 determines whether the task has been completed. … When it is determined that the task has been completed (Step S7: NO), the teleoperation assist device 3
returns the routine to Step S1.”), (Note: See Figure 7 of Dong as well.).
Regarding Claim 4, Dong teaches the method of claim 1, wherein training at least one model based on the at least one instance of teleoperation data and the at least one instance of human data comprises: training a single model based on the at least one instance of teleoperation data and the at least one instance of human data ([0077] via “At the time of training, the model 311 is trained by inputting training data and a position of a finger, gaze information, a position of a wrist, rotation information of the wrist, and environmental information to the model 311. The environmental information includes, for example, a captured image and information obtained by recognizing a target object (for example, a position, a shape, and a size). The training data includes task information and target information.”), (Note: See Figure 3 of Dong as well.).
Claim Rejections - 35 USC § 103
9. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
10. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
11. Claim(s) 5 and 11-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. (US 20230234231 A1 hereinafter Dong) in view of Kuang et al. (US 20250091214 A1 hereinafter Kuang).
Regarding Claim 5, Dong teaches the method of claim 1, wherein: accessing the at least one instance of teleoperation data comprises, for each instance of the teleoperation data, capturing, by at least one image sensor of the second at least one sensor as worn by the robot and aligned with a perspective of the robot, at least a portion of the robot data as robot-centric image data including a visual representation of the robot performing the task ([0029] via “In a work space, an environmental sensor 71 (71a, 71b) is installed. The environmental sensor 71
may be attached to a robot 2. … The environmental sensor 71 includes, for example, an RGB camera and a depth sensor.”), ([0044] via “The environmental status determining unit 12
acquires environment information in which the robot 2 is operated. The environmental status determining unit 12 recognizes a position of a target object, a posture of the target object, a type of the target object, a shape of the target object, and the like using detection information detected by the environmental sensor 71 and outputs the recognized information on the target object to the teleoperation assist device 3. The environmental status determining unit 12 recognizes a target object and a position, a size, a shape, or the like of an obstacle when the obstacle is interposed between the target object and the end effector 222 using the acquired information.”).
Dong is silent on wherein accessing the at least one instance of human data comprises, for each instance of the human data, capturing, by at least one image sensor of the third at least one sensor as worn by the human and aligned with a perspective of the at least one human, at least a portion of the human data as human-centric image data including a visual representation of the at least one human performing the task.
However, Kuang teaches wherein accessing the at least one instance of human data comprises, for each instance of the human data, capturing, by at least one image sensor of the third at least one sensor as worn by the human and aligned with a perspective of the at least one human, at least a portion of the human data as human-centric image data including a visual representation of the at least one human performing the task ([0049] via “At block 206, motion data 208 and gaze data 210 associated with the human operator's actions are captured during the performance of the sequence of tasks 204. … Gaze data 210 is captured to track the direction of the human operator's gaze. Both motion data 208 and gaze data 210 are captured while the human operator operates the controls of the robotic system which may include movements of actual elements of the robotic system. Motion data 208 and gaze data 210 may be acquired using any suitable technology including but not limited to commercially available or custom designed head-worn or eye-based gear adorned by operators, hand-held controllers complemented with accelerometers, internal or external cameras or image capturing devices, …, and so forth.”), (Note: See paragraph [0066] of Kaung as well.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Kuang wherein accessing the at least one instance of human data comprises, for each instance of the human data, capturing, by at least one image sensor of the third at least one sensor as worn by the human and aligned with a perspective of the at least one human, at least a portion of the human data as human-centric image data including a visual representation of the at least one human performing the task. The courts have determined under the case KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-07 (2007), a number of rationales in which obviousness is concluded. The rationale that pertains to the present invention is rationale B: Simple Substitution of One Known Element for Another to Obtain Predictable Results. Specifically, in this case item 3 of rationale B is satisfied: a finding that one of ordinary skill in the art could have substituted one known element for another, and the results of the substitution would have been predictable. Image sensors are a common type of visual sensor found in robotic and teleoperation control to sense information of an environment. While the invention of Dong includes a gaze sensor that senses a human operator’s gaze, despite the lack of mention that this gaze sensor is an image sensor, the functionalities of the invention would still produce the same outcomes, and therefore the simple substitution of an image sensor would have been obvious to implement.
Regarding Claim 11, Dong teaches a system comprising: at least one non-transitory processor-readable storage medium ([0015] via “A non-transitory computer-readable storage medium according to another aspect of the present invention is a non-transitory computer-readable storage medium storing a program causing a computer of a teleoperation assist device remotely operating at least an end effector to perform: ….”), and cause the system to:
access at least one instance of teleoperation data ([0029] via “FIG. 1 is a diagram schematically illustrating teleoperation of a robot.”), each instance of the at least one instance of teleoperation data comprising:
pilot data, the pilot data collected by a first at least one sensor of a piloted teleoperation system where a pilot provides at least one input to cause a robot to perform a task ([0041] via “The finger movement sensor 111 is, for example, a six-axis sensor, an acceleration sensor, or a vibration sensor. The finger movement sensor 111 is provided, for example, in the controller 61. The finger movement sensor 111 detects a fingertip command value including a position of each finger of the human operator and movement of each finger and outputs the detected fingertip command value to the teleoperation assist device 3.”), ([0049] via “The end effector operation determining unit 33 determines an amount of operation of the end effector 222 using the detection information detected by the finger movement sensor 111 and the taxonomy information set by the parameter setting unit 32.”), (Note: The Examiner interprets the finger movement sensor 111 of Dong as the first at least one sensor, and the position and movement of each finger of the human operator of Dong as the at least one instance of pilot data.);
robot data, the robot data collected by a second at least one sensor of the robot and representing operation of the robot which performs the task ([0029] via “In a work space, an environmental sensor 71 (71a, 71b) is installed. The environmental sensor 71 may be attached to a robot 2. … The environmental sensor 71 includes, for example, an RGB camera and a depth sensor.”), ([0044] via “The environmental status determining unit 12 acquires environment information in which the robot 2 is operated. The environmental status determining unit 12 recognizes a position of a target object, a posture of the target object, a type of the target object, a shape of the target object, and the like using detection information detected by the environmental sensor 71 and outputs the recognized information on the target object to the teleoperation assist device 3. The environmental status determining unit 12 recognizes a target object and a position, a size, a shape, or the like of an obstacle when the obstacle is interposed between the target object and the end effector 222 using the acquired information.”), (Note: The Examiner interprets the environmental sensor 71 of Dong as the second at least one sensor.);
access at least one instance of human data, each instance of the at least one instance of human data comprising data collected by a third at least one sensor positioned at a human and representing the human performing the task ([0042] via “The gaze sensor 112 is a sensor that detects movement of a gaze of the human operator. The gaze sensor 112 is provided, for example, in the HMD 51. The gaze sensor 112 detects movement of a gaze of the human operator and outputs the detected gaze information to the teleoperation assist device 3.”), (Note: The Examiner interprets the gaze sensor 112 of Dong as the third at least one sensor, and the gaze of the human operator as the at least one instance of human data.); and
train, by the at least one processor, at least one model based on the at least one instance of teleoperation data and the at least one instance of human data ([0047] via “The intention estimating unit 31 inputs the acquired information on a target object and the acquired detection information on the movement of the fingers of the human operator, and the acquired gaze information to a trained model 311 and estimates a task and a target (a target object) which are intended by the operator.”), ([0077] via “At the time of training, the model 311 is trained by inputting training data and a position of a finger, gaze information, a position of a wrist, rotation information of the wrist, and environmental information to the model 311. The environmental information includes, for example, a captured image and information obtained by recognizing a target object (for example, a position, a shape, and a size). The training data includes task information and target information.”), (Note: See Figure 7 of Dong as well.).
Dong is silent on the system comprising: at least one processor; and the at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least on non-transitory processor-readable storage medium storing processor-executable instructions which, when executed by the at least one processor, cause the system to perform.
However, Kuang teaches at least one processor ([0029] via “Computer components may include, for example, a process running on a processor, a processor, an object, an executable, a thread of execution, and a computer.”); and
the at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least on non-transitory processor-readable storage medium storing processor-executable instructions which, when executed by the at least one processor, cause the system to perform ([0032] via ““Computer-readable medium,” as used herein, refers to a non-transitory medium that stores instructions and/or data. A computer-readable medium may take forms, including, but not limited to, …, and other media from which a computer, a processor or other electronic device may read.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Kuang wherein the system comprises: at least one processor; and the at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least on non-transitory processor-readable storage medium storing processor-executable instructions which, when executed by the at least one processor, cause the system to perform. The courts have determined under the case KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-07 (2007), a number of rationales in which obviousness is concluded. The rationale that pertains to the present invention is rationale B: Simple Substitution of One Known Element for Another to Obtain Predictable Results. Specifically, in this case item 3 of rationale B is satisfied: a finding that one of ordinary skill in the art could have substituted one known element for another, and the results of the substitution would have been predictable. Computer processors communicatively coupled to computer storage mediums are a common type of computing component found in robotic control. While the invention of Dong includes non-transitory computer-readable storage medium communicatively coupled to a computer, despite the lack of mention that this computer comprises a processor, the functionalities of the invention would still produce the same outcomes, and therefore the simple substitution of a computer processor would have been obvious to implement.
Regarding Claim 12, modified reference Dong teaches the system of claim 11, wherein the processor-executable instructions which cause the at least one processor to train at least one model based on the at least one instance of teleoperation data and the at least one instance of human data cause the at least one processor to: train a first model based on the at least one instance of teleoperation data ([0077] via “At the time of training, the model 311 is trained by inputting training data and a position of a finger, gaze information, a position of a wrist, rotation information of the wrist, and environmental information to the model 311. The environmental information includes, for example, a captured image and information obtained by recognizing a target object (for example, a position, a shape, and a size).”), (Note: See Figure 3 of Dong as well.); and
refine the first model based on the at least one instance of human data to produce a second model ([0079] via “The model 311 is, for example, a network such as a deep neutral network (DNN), a recurrent neural network (RNN), a convolutional neural network (CNN), a long/short-term memory (LSTM), or a generative adversarial network (GAM).”), ([0094] via “(Step S3) The intention estimating unit 31 estimates a task and a target which are intended by the operator by inputting the acquired operator information and the environmental information to the trained model 311.”), ([0098] via “(Step S7) The teleoperation assist device 3 determines whether the task has been completed. … When it is determined that the task has been completed (Step S7: NO), the teleoperation assist device 3 returns the routine to Step S1.”), (Note: See Figure 7 of Dong as well.).
Regarding Claim 13, modified reference Dong teaches the system of claim 11, wherein the processor-executable instructions which cause the at least one processor to train at least one model based on the at least one instance of teleoperation data and the at least one instance of human data cause the at least one processor to: train a first model based on the at least one instance of human data ([0077] via “At the time of training, the model 311 is trained by inputting training data and a position of a finger, gaze information, a position of a wrist, rotation information of the wrist, and environmental information to the model 311. The environmental information includes, for example, a captured image and information obtained by recognizing a target object (for example, a position, a shape, and a size).”), (Note: See Figure 3 of Dong as well.); and
refine the first model based on the at least one instance of teleoperation data to produce a second model ([0079] via “The model 311 is, for example, a network such as a deep neutral network (DNN), a recurrent neural network (RNN), a convolutional neural network (CNN), a long/short-term memory (LSTM), or a generative adversarial network (GAM).”), ([0094] via “(Step S3) The intention estimating unit 31 estimates a task and a target which are intended by the operator by inputting the acquired operator information and the environmental information to the trained model 311.”), ([0098] via “(Step S7) The teleoperation assist device 3 determines whether the task has been completed. … When it is determined that the task has been completed (Step S7: NO), the teleoperation assist device 3
returns the routine to Step S1.”), (Note: See Figure 7 of Dong as well.).
Regarding Claim 14, modified reference Dong teaches the system of claim 11, wherein the processor-executable instructions which cause the at least one processor to train at least one model based on the at least one instance of teleoperation data and the at least one instance of human data cause the at least one processor to: train a single model based on the at least one instance of teleoperation data and the at least one instance of human data ([0077] via “At the time of training, the model 311 is trained by inputting training data and a position of a finger, gaze information, a position of a wrist, rotation information of the wrist, and environmental information to the model 311. The environmental information includes, for example, a captured image and information obtained by recognizing a target object (for example, a position, a shape, and a size). The training data includes task information and target information.”), (Note: See Figure 3 of Dong as well.).
Regarding Claim 15, modified reference Dong teaches the system of claim 11, further comprising: at least one image sensor of the second at least one sensor worn by the robot and aligned with a perspective of the robot ([0029] via “In a work space, an environmental sensor 71 (71a, 71b) is installed. The environmental sensor 71 may be attached to a robot 2. … The environmental sensor 71 includes, for example, an RGB camera and a depth sensor.”); and
wherein the processor-executable instructions which cause the system to access the at least one instance of teleoperation data cause the at least one image sensor of the second at least one sensor to capture at least a portion of the robot data as robot-centric image data including a visual representation of the robot performing the task ([0029] via “In a work space, an environmental sensor 71 (71a, 71b) is installed. The environmental sensor 71 may be attached to a robot 2. … The environmental sensor 71 includes, for example, an RGB camera and a depth sensor.”), ([0044] via “The environmental status determining unit 12 acquires environment information in which the robot 2 is operated. The environmental status determining unit 12 recognizes a position of a target object, a posture of the target object, a type of the target object, a shape of the target object, and the like using detection information detected by the environmental sensor 71 and outputs the recognized information on the target object to the teleoperation assist device 3. The environmental status determining unit 12 recognizes a target object and a position, a size, a shape, or the like of an obstacle when the obstacle is interposed between the target object and the end effector 222 using the acquired information.”).
Dong is silent on at least one image sensor of the third at least one sensor worn by the human and aligned with a perspective of the at least one human, and wherein the processor-executable instructions which cause the system to access the at least one instance of human data cause the at least one image sensor of the third at least one sensor to capture at least a portion of the at least one instance of human data as human-centric image data including a visual representation of the human performing the task.
However, Kuang teaches at least one image sensor of the third at least one sensor worn by the human and aligned with a perspective of the at least one human ([0049] via “Gaze data
210 is captured to track the direction of the human operator's gaze. … Motion data 208 and gaze data 210 may be acquired using any suitable technology including but not limited to commercially available or custom designed head-worn or eye-based gear adorned by operators, hand-held controllers complemented with accelerometers, internal or external cameras or image capturing devices, … and so forth.”), and
wherein the processor-executable instructions which cause the system to access the at least one instance of human data cause the at least one image sensor of the third at least one sensor to capture at least a portion of the at least one instance of human data as human-centric image data including a visual representation of the human performing the task ([0049] via “At block 206, motion data 208 and gaze data 210 associated with the human operator's actions are captured during the performance of the sequence of tasks 204. … Gaze data 210 is captured to track the direction of the human operator's gaze. Both motion data 208 and gaze data 210 are captured while the human operator operates the controls of the robotic system which may include movements of actual elements of the robotic system. Motion data 208 and gaze data 210 may be acquired using any suitable technology including but not limited to commercially available or custom designed head-worn or eye-based gear adorned by operators, hand-held controllers complemented with accelerometers, internal or external cameras or image capturing devices, …, and so forth.”), (Note: See paragraph [0066] of Kaung as well.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Kuang wherein the system comprises: at least one image sensor of the third at least one sensor worn by the human and aligned with a perspective of the at least one human, and wherein the processor-executable instructions which cause the system to access the at least one instance of human data cause the at least one image sensor of the third at least one sensor to capture at least a portion of the at least one instance of human data as human-centric image data including a visual representation of the human performing the task. The courts have determined under the case KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-07 (2007), a number of rationales in which obviousness is concluded. The rationale that pertains to the present invention is rationale B: Simple Substitution of One Known Element for Another to Obtain Predictable Results. Specifically, in this case item 3 of rationale B is satisfied: a finding that one of ordinary skill in the art could have substituted one known element for another, and the results of the substitution would have been predictable. Image sensors are a common type of visual sensor found in robotic and teleoperation control to sense information of an environment. While the invention of Dong includes a gaze sensor that senses a human operator’s gaze, despite the lack of mention that this gaze sensor is an image sensor, the functionalities of the invention would still produce the same outcomes, and therefore the simple substitution of an image sensor would have been obvious to implement.
12. Claim(s) 6 and 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. (US 20230234231 A1 hereinafter Dong) in view of Golda et al. (US 12103182 B1 hereinafter Golda).
Regarding Claim 6, Dong teaches the method of claim 1, but is silent on wherein: accessing the at least one instance of teleoperation data comprises, for each instance of the teleoperation data, capturing, by at least one inertial sensor of the second at least one sensor, at least a portion of the robot data as inertial data including an inertial representation of the robot performing the task; and accessing the human data comprises, for each instance of the human data, capturing, by at least one inertial sensor of the third at least one sensor, at least a portion of the human data as inertial data including an inertial representation of the human performing the task.
However, Golda teaches wherein: accessing the at least one instance of teleoperation data comprises, for each instance of the teleoperation data, capturing, by at least one inertial sensor of the second at least one sensor, at least a portion of the robot data as inertial data including an inertial representation of the robot performing the task (Col. 17 lines 5-15, where “FIG. 10 is an example of a control system 1000 for controlling the robotic device 104. The control system 1000 may include … a motion controller 1006 to control a robotic motion performed by the robotic device 104, and a sensed motion 1008 (e.g., a trajectory, position, orientation, velocity, or acceleration, sensed via one or more inertial sensors of the sensor array 108) of the robotic device 104.”); and
accessing the human data comprises, for each instance of the human data, capturing, by at least one inertial sensor of the third at least one sensor, at least a portion of the human data as inertial data including an inertial representation of the human performing the task (Col. 7 lines 25-39, where “For example, with additional reference to FIG. 2, a user can wear the wearable device 102 to perform a task, such as a glove used to pick up an object 122 (e.g., a soldering iron) and apply the object 122 in relation to a target 124 (e.g., a circuit board). The information may include motion data indicating a motion of the wearable device 102 and force data indicating a force applied to the wearable device 102. … The motion data (e.g., a trajectory, position, orientation, velocity, or acceleration) may also be obtained from the sensor array 106 (e.g., motion sensing, via one or more inertial sensors coupled with the wearable device 102, such as one or more IMUs).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Golda wherein: accessing the at least one instance of teleoperation data comprises, for each instance of the teleoperation data, capturing, by at least one inertial sensor of the second at least one sensor, at least a portion of the robot data as inertial data including an inertial representation of the robot performing the task; and accessing the human data comprises, for each instance of the human data, capturing, by at least one inertial sensor of the third at least one sensor, at least a portion of the human data as inertial data including an inertial representation of the human performing the task. Doing so trains the robot to perform motion control more similarly to how a human would, as stated by Golda (Col. 3 lines 11-27, where “Implementations of this disclosure address problems such as these by utilizing a wearable device and/or a robotic device with a tactile sensor array configured to replicate human-scale tactile sensing, touch, grasp, and/or dexterity. In some implementations, the wearable device may include an article configured to be worn by a user and a sensor array coupled with the article. … The sensor array may include sensors that provide multimodal sensing (e.g., micro-sensing of various conditions, such as normal forces, shear forces, temperatures, proximities, and/or images associated with objects, via normal force sensors, shear force sensors, thermal sensors, proximity sensors, image sensors (e.g., RGB cameras), and/or inertial sensors, such as inertial measurement units (IMU)).”).
Regarding Claim 7, Dong teaches the method of claim 1, but is silent on wherein: accessing the at least one instance of teleoperation data comprises, for each instance of the teleoperation data, capturing, by at least one haptic sensor of the second at least one sensor, at least a portion of the robot data as haptic data including a haptic representation of the robot performing the task; and accessing the at least one instance of human data comprises, for each instance of the human data, capturing, by at least one haptic sensor of the third at least one sensor, at least a portion of the human data as haptic data including a haptic representation of the human performing the task.
However, Golda teaches wherein: accessing the at least one instance of teleoperation data comprises, for each instance of the teleoperation data, capturing, by at least one haptic sensor of the second at least one sensor, at least a portion of the robot data as haptic data including a haptic representation of the robot performing the task (Col. 17 lines 5-15, where “FIG. 10 is an example of a control system 1000 for controlling the robotic device 104. The control system 1000 may include a force controller 1002 to control a robotic force applied by the robotic device 104, a sensed force 1004 (e.g., tactile data, such a pressure readings, sensed via sensors of the sensor array 108) applied to the robotic device 104, ….”); and
accessing the at least one instance of human data comprises, for each instance of the human data, capturing, by at least one haptic sensor of the third at least one sensor, at least a portion of the human data as haptic data including a haptic representation of the human performing the task (Col. 7 lines 18-35, where “In operation, the wearable controller 114 can receive information generated by the wearable device 102 when the wearable device 102 is utilized to perform a task in the demonstration environment 110. … The information may include motion data indicating a motion of the wearable device 102 and force data indicating a force applied to the wearable device 102. The force data may be obtained from the sensor array 106 (e.g., tactile sensing, via normal force sensors, shear force sensors, and/or other multimodal sensing) generating digital outputs.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Golda wherein: accessing the at least one instance of teleoperation data comprises, for each instance of the teleoperation data, capturing, by at least one haptic sensor of the second at least one sensor, at least a portion of the robot data as haptic data including a haptic representation of the robot performing the task; and accessing the at least one instance of human data comprises, for each instance of the human data, capturing, by at least one haptic sensor of the third at least one sensor, at least a portion of the human data as haptic data including a haptic representation of the human performing the task. Doing so trains the robot to perform force control more similarly to how a human would, as stated by Golda (Col. 3 lines 11-27, where “Implementations of this disclosure address problems such as these by utilizing a wearable device and/or a robotic device with a tactile sensor array configured to replicate human-scale tactile sensing, touch, grasp, and/or dexterity. In some implementations, the wearable device may include an article configured to be worn by a user and a sensor array coupled with the article. … The sensor array may include sensors that provide multimodal sensing (e.g., micro-sensing of various conditions, such as normal forces, shear forces, ….”).
13. Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. (US 20230234231 A1 hereinafter Dong) in view of Kolluri et al. (US 20210362333 A1 hereinafter Kolluri).
Regarding Claim 8, Dong teaches the method of claim 1, but is silent on wherein accessing the at least one instance of teleoperation data comprises generating, by at least one processor, at least a portion of the robot data as simulated sensor data representing motion or physical contact of a simulated instance of the robot controlled in accordance with the pilot data to perform the task.
However, Kolluri teaches wherein accessing the at least one instance of teleoperation data comprises generating, by at least one processor ([0275] via “… refers to data processing hardware … including by way of example a programmable processor, a computer, or multiple processors or computers.”), at least a portion of the robot data as simulated sensor data representing motion or physical contact of a simulated instance of the robot controlled in accordance with the pilot data to perform the task ([0247] via “The system generates simulated local demonstration data from the virtual model of the portion of the workcell (1430). For example, from a single camera image or scan of a socket, the system can generate multiple different perspectives to be used in training the demonstration subtask.”), ([0252] via “In addition, while the training system is tuning the base control policy with the simulated local demonstration data, the user can be collecting actual local demonstration data from the physical robot. Then, during or after tuning with the simulated local demonstration data is completed, the system can begin further refining the control policy using the actual local demonstration data collected from the physical robot.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Kolluri wherein accessing the at least one instance of teleoperation data comprises generating, by at least one processor, at least a portion of the robot data as simulated sensor data representing motion or physical contact of a simulated instance of the robot controlled in accordance with the pilot data to perform the task. Doing so only requires a small amount of data to be physically acquired in order to generate a large amount of data used to train the robot, as stated by Kolluri ([0253] via “This arrangement provides for very data-efficient training process. The user needs to upload only a relatively small amount of perceptual data. From there, the cloud-based training system can massively parallelize the process on hundreds or thousands of machines that all perform simulated training iterations using a virtual model constructed from the perceptual data.”), which speeds up the robot training process, as stated by Kolluri ([0241] via “FIG. 14 is a flowchart of an example process for training a skill template using a simulated working environment. Although using local demonstration data is fast relative to traditional methods, collecting the data still requires a nontrivial time investment in order to collect the amount of data required to make the models very precise. The process can be further sped up by parallel training on simulated workcell data. These techniques are particularly suited for demonstration subtasks that rely on perceptual streams of information.”).
14. Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. (US 20230234231 A1 hereinafter Dong) in view of Iwahara et al. (US 20220203517 A1 hereinafter Iwahara).
Regarding Claim 9, Dong teaches the method of claim 1, but is silent on wherein: the at least one instance of teleoperation data includes a first quantity of instances; and the at least one instance of human data includes a second quantity of instances greater than the first quantity of instances.
However, Iwahara teaches wherein: the at least one instance of teleoperation data includes a first quantity of instances ([0022] via “In the robot system in FIG. 1, a first work area WA1 in which a worker TP performs teaching work and a second work area WA2 in which the robot 100 executes work are set. … The second work area WA2 can be imaged by the second camera 220.”); and
the at least one instance of human data includes a second quantity of instances greater than the first quantity of instances ([0022] via “In the robot system in FIG. 1, a first work area WA1 in which a worker TP performs teaching work and a second work area WA2 in which the robot 100 executes work are set. … The first work area WA1 can be imaged by the first camera 210.”), ([0023] via “In the first work area WA1, the third camera 230 for imaging a hand and fingers of the worker TP and a workpiece is placed. It is preferable that the third camera 230 is placed in a position closer to the first work area WA1 than that of the first camera 210 for imaging the hand and fingers and the workpiece closer than the first camera
210. The positions of the hand and fingers and the workpiece are recognized using an image captured by the third camera 230, and thereby, the positions of the hand and fingers and the workpiece may be recognized more accurately compared to a case using only the first camera 210.”), (Note: See Figure 1 of Iwahara where there is more human data generated via cameras 210 and 230 than teleoperation data generated by lone camera 220.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Iwahara wherein: the at least one instance of teleoperation data includes a first quantity of instances; and the at least one instance of human data includes a second quantity of instances greater than the first quantity of instances. Doing so more accurately captures the human demonstrating the task to the robot, as stated above by Iwahara in paragraph [0023].
15. Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. (US 20230234231 A1 hereinafter Dong) in view of Wake et al. (US 20210402593 A1 hereinafter Wake).
Regarding Claim 10, Dong teaches the method of claim 1, but is silent on wherein: for each instance of the at least one instance of teleoperation data, the robot data represents operation of the robot which performs the task including engaging with a first set of at least one object; and for each instance of the at least one instance of human data, the human data represents the human performing the task including engaging with a second set of at least one object which corresponds to the first set of at least one object.
However, Wake teaches wherein: for each instance of the at least one instance of teleoperation data, the robot data represents operation of the robot which performs the task including engaging with a first set of at least one object; and for each instance of the at least one instance of human data, the human data represents the human performing the task including engaging with a second set of at least one object which corresponds to the first set of at least one object ([0024] via “FIG. 1A illustrates a “clean” environment 100 in which a human 102 demonstrates a task (or task sequence) and a robot 106 learns to perform the task (or task sequence) by observing the human 102. As illustrated, the task sequence is performed on an object (e.g., target object), such as cup 104. The task sequence involves “picking up” the cup 104 from table 108 and “placing” the cup 104 on shelf 110. To learn the task sequence, the robot 106 observes the human 102 picking up the cup 104 from table 108
and placing the cup 104 on the shelf 110. The robot 106 then performs the task sequence of picking up a cup 114 from a table 116 and placing the cup 114 on a shelf 118 based on Learning-from-Observation (LFO) 112.”), (Note: The Examiner interprets the cup 114 as the first set of at least one object and the cup 104 as the second set of at least one object corresponding to the first set of at least one object.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wake wherein: for each instance of the at least one instance of teleoperation data, the robot data represents operation of the robot which performs the task including engaging with a first set of at least one object; and for each instance of the at least one instance of human data, the human data represents the human performing the task including engaging with a second set of at least one object which corresponds to the first set of at least one object. Doing so allows the robot to replicate the same task as demonstrated by the operator with the same object, as stated above by Wake and shown in Figure 1A of Wake.
16. Claim(s) 16 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. (US 20230234231 A1 hereinafter Dong) in view of Kuang et al. (US 20250091214 A1 hereinafter Kuang), and further in view of Golda et al. (US 12103182 B1 hereinafter Golda).
Regarding Claim 16, modified reference Dong teaches the system of claim 11, but is silent on the system further comprising: at least one inertial sensor of the second at least one sensor; and at least one inertial sensor of the third at least one sensor, wherein the processor-executable instructions which cause the system to access the at least one instance of teleoperation data cause the at least one inertial sensor of the second at least one sensor to capture at least a portion of the robot data as inertial data including an inertial representation of the robot performing the task; and the processor-executable instructions which cause the system to access the at least one instance of human data cause the at least one inertial sensor of the third at least one sensor to capture at least a portion of the at least one instance of human data as inertial data including an inertial representation of the human performing the task.
However, Golda teaches at least one inertial sensor of the second at least one sensor (Col. 17 lines 6-15, where “The control system 1000 may include … a motion controller 1006
to control a robotic motion performed by the robotic device 104, and a sensed motion 1008
(e.g., a trajectory, position, orientation, velocity, or acceleration, sensed via one or more inertial sensors of the sensor array 108) of the robotic device 104.”); and
at least one inertial sensor of the third at least one sensor (Col. 7 lines 35-39, where “The motion data (e.g., a trajectory, position, orientation, velocity, or acceleration) may also be obtained from the sensor array 106 (e.g., motion sensing, via one or more inertial sensors coupled with the wearable device 102, such as one or more IMUs).”),
wherein the processor-executable instructions which cause the system to access the at least one instance of teleoperation data cause the at least one inertial sensor of the second at least one sensor to capture at least a portion of the robot data as inertial data including an inertial representation of the robot performing the task (Col. 17 lines 5-15, where “FIG. 10 is an example of a control system 1000 for controlling the robotic device 104. The control system 1000 may include … a motion controller 1006 to control a robotic motion performed by the robotic device 104, and a sensed motion 1008 (e.g., a trajectory, position, orientation, velocity, or acceleration, sensed via one or more inertial sensors of the sensor array 108) of the robotic device 104.”); and
the processor-executable instructions which cause the system to access the at least one instance of human data cause the at least one inertial sensor of the third at least one sensor to capture at least a portion of the at least one instance of human data as inertial data including an inertial representation of the human performing the task (Col. 7 lines 25-39, where “For example, with additional reference to FIG. 2, a user can wear the wearable device 102 to perform a task, such as a glove used to pick up an object 122 (e.g., a soldering iron) and apply the object 122 in relation to a target 124 (e.g., a circuit board). The information may include motion data indicating a motion of the wearable device 102 and force data indicating a force applied to the wearable device 102. … The motion data (e.g., a trajectory, position, orientation, velocity, or acceleration) may also be obtained from the sensor array 106 (e.g., motion sensing, via one or more inertial sensors coupled with the wearable device 102, such as one or more IMUs).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Golda wherein the system further comprises: at least one inertial sensor of the second at least one sensor; and at least one inertial sensor of the third at least one sensor, wherein the processor-executable instructions which cause the system to access the at least one instance of teleoperation data cause the at least one inertial sensor of the second at least one sensor to capture at least a portion of the robot data as inertial data including an inertial representation of the robot performing the task; and the processor-executable instructions which cause the system to access the at least one instance of human data cause the at least one inertial sensor of the third at least one sensor to capture at least a portion of the at least one instance of human data as inertial data including an inertial representation of the human performing the task. Doing so trains the robot to perform motion control more similarly to how a human would, as stated by Golda (Col. 3 lines 11-27, where “Implementations of this disclosure address problems such as these by utilizing a wearable device and/or a robotic device with a tactile sensor array configured to replicate human-scale tactile sensing, touch, grasp, and/or dexterity. In some implementations, the wearable device may include an article configured to be worn by a user and a sensor array coupled with the article. … The sensor array may include sensors that provide multimodal sensing (e.g., micro-sensing of various conditions, such as normal forces, shear forces, temperatures, proximities, and/or images associated with objects, via normal force sensors, shear force sensors, thermal sensors, proximity sensors, image sensors (e.g., RGB cameras), and/or inertial sensors, such as inertial measurement units (IMU)).”).
Regarding Claim 17, modified reference Dong teaches the system of claim 11, but is silent on the system further comprising: at least one haptic sensor of the second at least one sensor; and at least one haptic sensor of the third at least one sensor, wherein the processor-executable instructions which cause the system to access the at least one instance of teleoperation data cause the at least one haptic sensor of the second at least one sensor to capture at least a portion of the robot data as haptic data including a haptic representation of the robot performing the task; and the processor-executable instructions which cause the system to access the at least one instance of human data cause the at least one haptic sensor of the third at least one sensor to capture at least a portion of the at least one instance of human data as haptic data including a haptic representation of the human performing the task.
However, Golda teaches at least one haptic sensor of the second at least one sensor (Col. 17 lines 5-15, where “FIG. 10 is an example of a control system 1000 for controlling the robotic device 104. The control system 1000 may include a force controller 1002 to control a robotic force applied by the robotic device 104, a sensed force 1004 (e.g., tactile data, such a pressure readings, sensed via sensors of the sensor array 108) applied to the robotic device
104, ….”); and
at least one haptic sensor of the third at least one sensor (Col. 7 lines 32-35, where “The force data may be obtained from the sensor array 106 (e.g., tactile sensing, via normal force sensors, shear force sensors, and/or other multimodal sensing) generating digital outputs.”),
wherein the processor-executable instructions which cause the system to access the at least one instance of teleoperation data cause the at least one haptic sensor of the second at least one sensor to capture at least a portion of the robot data as haptic data including a haptic representation of the robot performing the task (Col. 17 lines 5-15, where “FIG. 10 is an example of a control system 1000 for controlling the robotic device 104. The control system 1000 may include a force controller 1002 to control a robotic force applied by the robotic device 104, a sensed force 1004 (e.g., tactile data, such a pressure readings, sensed via sensors of the sensor array 108) applied to the robotic device 104, ….”); and
the processor-executable instructions which cause the system to access the at least one instance of human data cause the at least one haptic sensor of the third at least one sensor to capture at least a portion of the at least one instance of human data as haptic data including a haptic representation of the human performing the task (Col. 7 lines 18-35, where “In operation, the wearable controller 114 can receive information generated by the wearable device 102 when the wearable device 102 is utilized to perform a task in the demonstration environment 110. … The information may include motion data indicating a motion of the wearable device 102 and force data indicating a force applied to the wearable device 102. The force data may be obtained from the sensor array 106 (e.g., tactile sensing, via normal force sensors, shear force sensors, and/or other multimodal sensing) generating digital outputs.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Golda wherein the system further comprises: at least one haptic sensor of the second at least one sensor; and at least one haptic sensor of the third at least one sensor, wherein the processor-executable instructions which cause the system to access the at least one instance of teleoperation data cause the at least one haptic sensor of the second at least one sensor to capture at least a portion of the robot data as haptic data including a haptic representation of the robot performing the task; and the processor-executable instructions which cause the system to access the at least one instance of human data cause the at least one haptic sensor of the third at least one sensor to capture at least a portion of the at least one instance of human data as haptic data including a haptic representation of the human performing the task. Doing so trains the robot to perform force control more similarly to how a human would, as stated by Golda (Col. 3 lines 11-27, where “Implementations of this disclosure address problems such as these by utilizing a wearable device and/or a robotic device with a tactile sensor array configured to replicate human-scale tactile sensing, touch, grasp, and/or dexterity. In some implementations, the wearable device may include an article configured to be worn by a user and a sensor array coupled with the article. … The sensor array may include sensors that provide multimodal sensing (e.g., micro-sensing of various conditions, such as normal forces, shear forces, ….”).
17. Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. (US 20230234231 A1 hereinafter Dong) in view of Kuang et al. (US 20250091214 A1 hereinafter Kuang), and further in view of Kolluri et al. (US 20210362333 A1 hereinafter Kolluri).
Regarding Claim 18, modified reference Dong teaches the system of claim 11, but is silent on wherein the processor-executable instructions which cause the system to access the at least one instance of teleoperation data cause the at least one processor to generate at least a portion of the robot data as simulated sensor data representing motion or physical contact of a simulated instance of the robot controlled in accordance with the pilot data to perform the task.
However, Kolluri teaches wherein the processor-executable instructions which cause the system to access the at least one instance of teleoperation data cause the at least one processor ([0275] via “… refers to data processing hardware … including by way of example a programmable processor, a computer, or multiple processors or computers.”) to generate at least a portion of the robot data as simulated sensor data representing motion or physical contact of a simulated instance of the robot controlled in accordance with the pilot data to perform the task ([0247] via “The system generates simulated local demonstration data from the virtual model of the portion of the workcell (1430). For example, from a single camera image or scan of a socket, the system can generate multiple different perspectives to be used in training the demonstration subtask.”), ([0252] via “In addition, while the training system is tuning the base control policy with the simulated local demonstration data, the user can be collecting actual local demonstration data from the physical robot. Then, during or after tuning with the simulated local demonstration data is completed, the system can begin further refining the control policy using the actual local demonstration data collected from the physical robot.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Kolluri wherein the processor-executable instructions which cause the system to access the at least one instance of teleoperation data cause the at least one processor to generate at least a portion of the robot data as simulated sensor data representing motion or physical contact of a simulated instance of the robot controlled in accordance with the pilot data to perform the task. Doing so only requires a small amount of data to be physically acquired in order to generate a large amount of data used to train the robot, as stated by Kolluri ([0253] via “This arrangement provides for very data-efficient training process. The user needs to upload only a relatively small amount of perceptual data. From there, the cloud-based training system can massively parallelize the process on hundreds or thousands of machines that all perform simulated training iterations using a virtual model constructed from the perceptual data.”), which speeds up the robot training process, as stated by Kolluri ([0241] via “FIG. 14 is a flowchart of an example process for training a skill template using a simulated working environment. Although using local demonstration data is fast relative to traditional methods, collecting the data still requires a nontrivial time investment in order to collect the amount of data required to make the models very precise. The process can be further sped up by parallel training on simulated workcell data. These techniques are particularly suited for demonstration subtasks that rely on perceptual streams of information.”).
18. Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. (US 20230234231 A1 hereinafter Dong) in view of Kuang et al. (US 20250091214 A1 hereinafter Kuang), and further in view of Iwahara et al. (US 20220203517 A1 hereinafter Iwahara).
Regarding Claim 19, modified reference Dong teaches the system of claim 11, but is silent on wherein: the at least one instance of teleoperation data includes a first quantity of instances; and the at least one instance of human data includes a second quantity of instances greater than the first quantity of instances.
However, Iwahara teaches wherein: the at least one instance of teleoperation data includes a first quantity of instances ([0022] via “In the robot system in FIG. 1, a first work area WA1 in which a worker TP performs teaching work and a second work area WA2 in which the robot 100 executes work are set. … The second work area WA2 can be imaged by the second camera 220.”); and
the at least one instance of human data includes a second quantity of instances greater than the first quantity of instances ([0022] via “In the robot system in FIG. 1, a first work area WA1 in which a worker TP performs teaching work and a second work area WA2 in which the robot 100 executes work are set. … The first work area WA1 can be imaged by the first camera 210.”), ([0023] via “In the first work area WA1, the third camera 230 for imaging a hand and fingers of the worker TP and a workpiece is placed. It is preferable that the third camera 230 is placed in a position closer to the first work area WA1 than that of the first camera 210 for imaging the hand and fingers and the workpiece closer than the first camera
210. The positions of the hand and fingers and the workpiece are recognized using an image captured by the third camera 230, and thereby, the positions of the hand and fingers and the workpiece may be recognized more accurately compared to a case using only the first camera 210.”), (Note: See Figure 1 of Iwahara where there is more human data generated via cameras 210 and 230 than teleoperation data generated by lone camera 220.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Iwahara wherein: the at least one instance of teleoperation data includes a first quantity of instances; and the at least one instance of human data includes a second quantity of instances greater than the first quantity of instances. Doing so more accurately captures the human demonstrating the task to the robot, as stated above by Iwahara in paragraph [0023].
19. Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. (US 20230234231 A1 hereinafter Dong) in view of Kuang et al. (US 20250091214 A1 hereinafter Kuang), and further in view of Wake et al. (US 20210402593 A1 hereinafter Wake).
Regarding Claim 20, modified reference Dong teaches the system of claim 11, but is silent on wherein: for each instance of the at least one instance of teleoperation data, the robot data represents operation of the robot which performs the task including engaging with a first set of at least one object; and for each instance of the at least one instance of human data, the human data represents the human performing the task including engaging with a second set of at least one object which corresponds to the first set of at least one object.
However, Wake teaches wherein: for each instance of the at least one instance of teleoperation data, the robot data represents operation of the robot which performs the task including engaging with a first set of at least one object; and for each instance of the at least one instance of human data, the human data represents the human performing the task including engaging with a second set of at least one object which corresponds to the first set of at least one object ([0024] via “FIG. 1A illustrates a “clean” environment 100 in which a human 102 demonstrates a task (or task sequence) and a robot 106 learns to perform the task (or task sequence) by observing the human 102. As illustrated, the task sequence is performed on an object (e.g., target object), such as cup 104. The task sequence involves “picking up” the cup 104 from table 108 and “placing” the cup 104 on shelf 110. To learn the task sequence, the robot 106 observes the human 102 picking up the cup 104 from table 108
and placing the cup 104 on the shelf 110. The robot 106 then performs the task sequence of picking up a cup 114 from a table 116 and placing the cup 114 on a shelf 118 based on Learning-from-Observation (LFO) 112.”), (Note: The Examiner interprets the cup 114 as the first set of at least one object and the cup 104 as the second set of at least one object corresponding to the first set of at least one object.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wake wherein: for each instance of the at least one instance of teleoperation data, the robot data represents operation of the robot which performs the task including engaging with a first set of at least one object; and for each instance of the at least one instance of human data, the human data represents the human performing the task including engaging with a second set of at least one object which corresponds to the first set of at least one object. Doing so allows the robot to replicate the same task as demonstrated by the operator with the same object, as stated above by Wake and shown in Figure 1A of Wake.
Examiner’s Note
20. The Examiner has cited particular paragraphs or columns and line numbers in the
references applied to the claims above for the convenience of the Applicant. Although the
specified citations are representative of the teachings of the art and are applied to specific
limitations within the individual claim, other passages and figures may apply as well. It is
respectfully requested of the Applicant in preparing responses, to fully consider the references
in their entirety as potentially teaching all or part of the claimed invention, as well as the
context of the passage as taught by the prior art or disclosed by the Examiner. See MPEP
2141.02 [R-07.2015] VI. A prior art reference must be considered in its entirety, i.e., as a whole,
including portions that would lead away from the claimed Invention. W.L. Gore & Associates,
Inc. v. Garlock, Inc., 721 F.2d 1540, 220 USPQ 303 (Fed. Cir. 1983), cert, denied, 469 U.S. 851
(1984). See also MPEP §2123.
Conclusion
21. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
22. Any inquiry concerning this communication or earlier communications from the
examiner should be directed to BYRON X KASPER whose telephone number is (571)272-3895.
The examiner can normally be reached Monday - Friday 8 am - 5 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing
using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is
encouraged to use the USPTO Automated Interview Request (AIR) at
http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s
supervisor, Adam Mott can be reached on (571) 270-5376. The fax phone number for the
organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be
obtained from Patent Center. Unpublished application information in Patent Center is available
to registered users. To file and manage patent submissions in Patent Center, visit:
https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for
more information about Patent Center and https://www.uspto.gov/patents/docx for
information about filing in DOCX format. For additional questions, contact the Electronic
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO
Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BYRON XAVIER KASPER/Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657
