DETAILED ACTION
Status of Claims
Claims 1-21 are currently pending and have been examined in this application. This non-final rejection is in response to the amendment submitted on 1/29/2026.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Response to Arguments and Amendments
Applicant’s arguments, filed on 1/29/2026, with respect to the rejection of Claims 1-21 under 35 USC 103 have been fully considered but they are moot in view of the new grounds of rejection provided below, which was necessitated based on Applicant’s amendments to the claims, which changed the scope of the claims. Examiner notes wherein Applicant’s arguments are directed towards the newly amended claim limitation(s), which are addressed by the newly found prior art, as indicated below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 16, 17 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Dan (US 20230373098 A1) as modified by Atherton (US 20180173200 A1) in view of Tokuchi (US 20190082102 A1)
Claim 1:
Dan teaches the following limitations:
A robot control system, comprising: processing circuitry configured to
(Dan - [0014] A general architecture for the interactive programming framework is illustrated in FIG. 1. The system may include four types of necessary modules: 1) operator communication channels 10, i.e., speech processing 10A and/or a graphical user interface (GUI) 10B in the exemplary architecture; 2) environmental sensors 12 providing awareness of the robot 18 surroundings, e.g., a vision system 12; 3) actuation 14 interfacing with a specific robot 18 control channel, e.g., ABB's RobotWare or ABB's EGM or other similar offerings; and 4) a controller 16 coordinating the previous modules to provide the intuitive and natural interaction for the operator to teach and interact with the robot 18.) generate an inquiry for a user while a robot is executing a task in response to missing information regarding the task, (Dan – [0012] … In a second step, if an instruction is not understood, then the operator is prompted for clarification (e.g., which object?) and several choices may be offered (e.g., by showing the objects that are visible to the robot's camera). The operator is then able to provide feedback to the robot using one of the several input modalities (e.g., speech input, touch screen, hand gesture) so that the intended task can be completed.)
identify, using a sensor, an action of the user performed in response to the inquiry, ( Dan – [0015] … the environment sensor module 12 may utilize different sensors including a camera for object identification, robotic skin or force sensors for contact, gripping or collision detection, lidar/radar for distance estimation, thermal sensors to identify possible operating hazards or process steps, etc. Each channel and sensor may be handled by a specific module 10, 12.) the action being one of a hand gesture and (Dan – [0010] … Speech input/output, recognition of objects and hand gestures, and a graphical user interface guide users through the co-creation of robotic tasks and help them handle potential errors.)
the action providing the robot with instruction on how to execute the task, addressing the missing information,
(Dan – [0010] … Speech input/output, recognition of objects and hand gestures, and a graphical user interface guide users through the co-creation of robotic tasks and help them handle potential errors. … This way, robot programming becomes accessible to ordinary operators and the initial setup and reconfiguration of the robot's functionality can be completed within a short time.;
control the robot such that the robot executes at least part of the task.
(Dan – [0012] … The operator is then able to provide feedback to the robot using one of the several input modalities (e.g., speech input, touch screen, hand gesture) so that the intended task can be completed. )
Dan does not explicitly teach the following limitations, however Atherton teaches:
a pointing gesture toward a position,
(Atherton - [0025] Interpreter module 200 is configured to receive sensor data 122 from sensor array 134 and to process this data to generate gesture data 202 and location data 204. Gesture data 202 indicates the specific gestures performed by end-user 140. For example, gesture data 202 could indicate that end-user 140 performs a pointing gesture. Location data 204 indicates one or more locations associated with the gestures performed by end-user 140. For example, location data 204 could indicate a specific position on work piece 150 where end-user 140 points.)
Dan in combination with Atherton does not explicitly teach the following limitations, however Tokuchi teaches:
pause execution of the task upon generation of the inquiry,
(Tokuchi - [0162] When executing Step 5, the controller 441 notifies the computer 110 of the content of scheduled operation (scheduled operation) and waits for a response from the computer 110. …)
resume the task using the instruction from the inquiry on how to execute the task to address the missing information, and
(Tokuchi - [0162] … Here, when receiving a response indicating that the scheduled operation is accepted (when an affirmative result is obtained in Step 6), the controller 441 instructs the device concerned (for example, the cleaning robot 158) to perform the operation specified in Step 4 (Step 12).)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan with a method of using a pointing gesture as an indication of a desired position for a robot or workpiece as taught in Atherton and to further pause a task while waiting for the answer to an inquiry and resume a task after receiving a satisfactory resolution to an inquiry as taught in Tokuchi. Having the ability to pause or resume a task based on its inquiry status and to also associate operator hand gestures with robot or workpiece positions through the natural act of pointing reduces the need for complex programming and further allows the operator to interactively communicate a target location in real time with a cooperative robot.
Claim 2:
Dan teaches the following limitations:
The robot control system according to claim 1, wherein the sensor includes a camera, and wherein the processing circuitry is further configured to extract, from an image captured by the camera, a recognition target, wherein the recognition target is predefined for the task in response to a determination that parameters of the task include a variable in place of an explicit identifier for the recognition target, and the recognition target is subsequently employed when
(Dan – [0012] … if an instruction is not understood, then the operator is prompted for clarification (e.g., which object?) and several choices may be offered (e.g., by showing the objects that are visible to the robot's camera). The operator is then able to provide feedback to the robot using one of the several input modalities (e.g., speech input, touch screen, hand gesture) so that the intended task can be completed.)
Examiner Note:
Examiner notes wherein Applicant has cited a hand gesture of the user or a reference object as examples of a “recognition target”
Dan in combination with Atherton does not explicitly teach the following limitations, however Tokuchi teaches:
the robot resumes the paused task.
(Tokuchi - [0162] … Here, when receiving a response indicating that the scheduled operation is accepted (when an affirmative result is obtained in Step 6), the controller 441 instructs the device concerned (for example, the cleaning robot 158) to perform the operation specified in Step 4 (Step 12).)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan with a method of pausing a task while waiting for the answer to an inquiry clarifying a task as taught in Tokuchi. Having the ability to pause a task while clarifying its meaning reduces errors and increases the efficiency of the tasks being performed in real time by a cooperative robot.
Claim 16:
Dan teaches the following limitations:
A robot control method, comprising: generating, by processing circuitry of a robot control system, (Dan - [0014] A general architecture for the interactive programming framework is illustrated in FIG. 1. The system may include four types of necessary modules: 1) operator communication channels 10, i.e., speech processing 10A and/or a graphical user interface (GUI) 10B in the exemplary architecture; 2) environmental sensors 12 providing awareness of the robot 18 surroundings, e.g., a vision system 12; 3) actuation 14 interfacing with a specific robot 18 control channel, e.g., ABB's RobotWare or ABB's EGM or other similar offerings; and 4) a controller 16 coordinating the previous modules to provide the intuitive and natural interaction for the operator to teach and interact with the robot 18.) an inquiry for a user while a robot is executing a task in response to missing information regarding the task; (Dan – [0012] … In a second step, if an instruction is not understood, then the operator is prompted for clarification (e.g., which object?) and several choices may be offered (e.g., by showing the objects that are visible to the robot's camera). The operator is then able to provide feedback to the robot using one of the several input modalities (e.g., speech input, touch screen, hand gesture) so that the intended task can be completed.)
identifying, using a sensor and by the processing circuitry of the robot control system, an action of the user performed in response to the inquiry,
( Dan – [0015] … the environment sensor module 12 may utilize different sensors including a camera for object identification, robotic skin or force sensors for contact, gripping or collision detection, lidar/radar for distance estimation, thermal sensors to identify possible operating hazards or process steps, etc. Each channel and sensor may be handled by a specific module 10, 12.) the action being one of a hand gesture and
(Dan – [0010] … Speech input/output, recognition of objects and hand gestures, and a graphical user interface guide users through the co-creation of robotic tasks and help them handle potential errors.)
the action providing the robot with instruction on how to execute the task, addressing the missing information;
(Dan – [0010] … Speech input/output, recognition of objects and hand gestures, and a graphical user interface guide users through the co-creation of robotic tasks and help them handle potential errors. … This way, robot programming becomes accessible to ordinary operators and the initial setup and reconfiguration of the robot's functionality can be completed within a short time.)
controlling, by the processing circuitry of the robot control system, the robot such that the robot executes at least part of the task.
(Dan – [0012] … The operator is then able to provide feedback to the robot using one of the several input modalities (e.g., speech input, touch screen, hand gesture) so that the intended task can be completed. )
Dan does not explicitly teach the following limitations, however Atherton teaches:
a pointing gesture toward a position,
(Atherton - [0025] Interpreter module 200 is configured to receive sensor data 122 from sensor array 134 and to process this data to generate gesture data 202 and location data 204. Gesture data 202 indicates the specific gestures performed by end-user 140. For example, gesture data 202 could indicate that end-user 140 performs a pointing gesture. Location data 204 indicates one or more locations associated with the gestures performed by end-user 140. For example, location data 204 could indicate a specific position on work piece 150 where end-user 140 points.)
Dan in combination with Atherton does not explicitly teach the following limitations, however Tokuchi teaches:
pausing, by processing circuitry of a robot control system, execution of the task upon generation of the inquiry;
(Tokuchi - [0162] When executing Step 5, the controller 441 notifies the computer 110 of the content of scheduled operation (scheduled operation) and waits for a response from the computer 110. …)
resuming, by using the instruction from the inquiry on how to execute the task to address the missing information: and
(Tokuchi - [0162] … Here, when receiving a response indicating that the scheduled operation is accepted (when an affirmative result is obtained in Step 6), the controller 441 instructs the device concerned (for example, the cleaning robot 158) to perform the operation specified in Step 4 (Step 12).)
Therefore, prior to the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Dan with a method of using a pointing gesture as an indication of a desired position for a robot or workpiece as taught in Atherton and to further pause a task while waiting for the answer to an inquiry and resume a task after receiving a satisfactory resolution to an inquiry as taught in Tokuchi. Having the ability to pause or resume a task based on its inquiry status and to also associate operator hand gestures with robot or workpiece positions through the natural act of pointing reduces the need for complex programming and further allows the operator to interactively communicate a target location in real time with a cooperative robot.
Claim 17:
Dan teaches the following limitations:
A non-transitory computer-readable storage medium including computer executable instructions that when executed by a computer, cause the computer to perform the method of Claim 16.
(Dan - [0022] The operator inputs (and additional operator inputs) are then translated into software commands that will direct the robot 18 to perform the desired robot movement (26), for example, by an actuation module 10. … By contrast, the translated software commands are a form of specialized software code understood by engineers and the robot 18 but not readily used or understood by ordinary operators. It may be desirable to store the operator inputs, additional operator inputs and/or translated software commands in a library storage with data links therebetween so that future operator inputs may be able to be autonomously linked to such additional operator inputs and/or software commands (28). The library storage may be included as part of the actuation module 14 and may be a skill set library that recognizes operator inputs.)
Claim 21:
Dan does not explicitly teach the following limitations, however Atherton teaches:
The robot control system according to claim 1, wherein the processing circuitry is configured to identify, using a sensor, an action of the user performed in response to the inquiry, the action being one of a hand gesture and a pointing gesture toward a position, the pointing gesture being recognized as X-Y coordinates relative to the robot.
(Atherton - [0033] As shown in FIG. 3C, hand 300 of end-user 140 performs a sweeping-pointing gesture 306 relative to work piece 150. Sensor array 134 captures sensor data 122 that represents gesture 306, and control engine 118 then interprets this gesture as indicating that an arc should be cut across the surface of work piece 150. Control engine 118 then causes projector 136 to project image 152 onto work piece 150. In this example, image 152 includes a toolpath 320 indicating the precise trajectory across the surface of work piece 150 where the arc should be cut. Control engine 118 may then cause robot 130 to cut that arc using implement 138.)
Examiner’s Note:
It would be understood by one of ordinary skill in the art that the generation of a robot trajectory by pointing would require at least X and Y coordinates.
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan with a method of using a pointing gesture as an indication of a desired x-y coordinate position for a robot or workpiece as taught in Atherton. Having the ability to associate operator hand gestures with the x-y location of a robot or workpiece positions, through the natural act of pointing, reduces the need for complex programming and further allows the operator to interactively communicate a target location in real time with a cooperative robot.
Claims 3, 5, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Dan (US 20230373098 A1) as modified by Atherton (US 20180173200 A1) and Tokuchi (US 20190082102 A1) in view of Hayashi (US 20220281109 A1), here after referred to as Hayashi ‘109
Claim 3:
Dan in combination with Atherton and Tokuchi does not explicitly teach the following limitations, however Hayashi ‘109 teaches:
The robot control system according to claim 2, wherein the processing circuitry is further configured to move the camera according to a preceding action performed by the user
(Hayashi ‘109 – [0095] … An action trajectory of the robot body 1 based on teaching points created in accordance with pieces of information corresponding to the dominant hand and height attributes associated with the worker ID acquired by using the LUT 11 is displayed using a highlighted arrow. On this occasion, a target position when the robot body 1 approaches the worker is also displayed. … ; [0096] … In this case, action details are generated by using pieces of three-dimensional information acquired by the vision sensor 5 and the stereo cameras 22 and 23 and are overwritten.)
Dan further teaches the following limitations:
when instruction is still needed regarding how to execute the task.
(Dan – [0012] … In a second step, if an instruction is not understood, then the operator is prompted for clarification (e.g., which object?) and several choices may be offered (e.g., by showing the objects that are visible to the robot's camera). The operator is then able to provide feedback to the robot using one of the several input modalities (e.g., speech input, touch screen, hand gesture) so that the intended task can be completed.)
Dan in combination with Atherton does not explicitly teach the following limitations, however Tokuchi teaches:
while the inquiry remains active and the task remains paused.
(Tokuchi - [0162] When executing Step 5, the controller 441 notifies the computer 110 of the content of scheduled operation (scheduled operation) and waits for a response from the computer 110. …)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan, Atherton and Tokuchi with a method of tracking the user’s gestures with a camera as taught in Hayashi ‘109. Being able to have a camera track a user’s gestures allows the system to receive instructions and clarifications from the user thus improving the ability of the user to collaborate with the robot.
Claim 5:
Dan in combination with Atherton and Tokuchi does not explicitly teach the following limitations, however Hayashi ‘109 teaches:
The robot control system according to claim 3, wherein the camera is mounted on the robot, (Hayashi ‘109 – [0020] FIG. 1 is a block diagram schematically illustrating a control system of a robot system 1000 in this embodiment. In FIG. 1, the robot system 1000 includes a robot body 1, a robot control device 2 that performs drive control of the robot body 1, and a vision sensor 5 that monitors the operational status of the robot body 1. Although FIG. 1 illustrates one vision sensor 5, a plurality of vision sensors 5 may be provided, and the vision sensor 5 may be installed in the robot body 1 as necessary. …) the processing circuitry is configured to cause the robot to operate such that the camera approaches the recognition target and (Hayashi ‘109 – [0095] … An action trajectory of the robot body 1 based on teaching points created in accordance with pieces of information corresponding to the dominant hand and height attributes associated with the worker ID acquired by using the LUT 11 is displayed using a highlighted arrow. On this occasion, a target position when the robot body 1 approaches the worker is also displayed. … ; [0096] … In this case, action details are generated by using pieces of three-dimensional information acquired by the vision sensor 5 and the stereo cameras 22 and 23 and are overwritten.) extract the recognition target from an image captured by the camera that has approached the recognition target. (Hayashi ‘109 – [0044] Furthermore, the CPU 301 is connected to the vision sensor 5 via the interface 312 and the bus 315. The vision sensor 5 captures an image of the robot body 1 and the worker 100 and measures the image to thereby acquire the operational status of the robot body 1 and surrounding circumstances. … ; [0090] The image sensor 21 is an image sensor for identifying a worker wearing the head mounted display 13. The image sensor 21 acquires an image of an iris, a face, or the like and transfers the acquired image to the ID recognition unit 10 via the user interface control device 4.)
Examiner Note:
Examiner notes wherein Applicant has cited a hand gesture of the user or a reference object as examples of a “recognition target”
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan, Atherton, and Tokuchi with a method of targeting the user’s hands or a reference object as taught in Hayashi ‘109. Having the ability to accurately target gestures and reference objects allows the system to receive instructions and clarifications from the user thus improving the ability of the user to collaborate with the robot.
Claim 11:
Dan in combination with Atherton and Tokuchi does not explicitly teach the following limitations, however Hayashi ‘109 teaches:
The robot control system according to claim 2, wherein the recognition target is not the hand of the user, and the processing circuitry is configured to extract a reference object presented by the user from the image, identify a feature quantity of the reference object as a workpiece, wherein the workpiece is the subject of the action specified by the task.
(Hayashi ‘109 – [0043] Furthermore, the CPU 301 is connected to the parts supply device 16 via the interface 310 and the bus 315. The parts supply device 16 includes a sensor (not illustrated) that detects that a workpiece has been set, and the CPU 301 instructs the parts supply device 16 to set a new workpiece in accordance with an output of this sensor. ; [0065] FIGS. 5A to 5C each illustrate an example of an action of the robot body 1 in collaborative work between the robot body 1 and the worker 100, and the action has been registered in the LUT 11 in association with an ID. FIGS. 5A to 5C each illustrate work in which the robot body 1 takes a fully assembled workpiece 20 from a workbench 16 and transfers the workpiece 20 to the worker 100, the worker 100 examines the received workpiece 20 and transfers the workpiece 20 to the robot body 1, and the robot body 1 puts the workpiece 20 back on the workbench 16.)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan, Atherton, and Tokuchi with a method of targeting the user’s hands or a reference object as taught in Hayashi ‘109. Having the ability to accurately target gestures and reference objects allows the system to receive instructions and clarifications from the user thus improving the ability of the user to collaborate with the robot.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Dan (US 20230373098 A1) as modified by Atherton (US 20180173200 A1) and Tokuchi (US 20190082102 A1) in view of Hayashi (US 20220281109 A1), here after referred to as Hayashi ‘109 and in further view of Hayashi (US 20190184567 A1), here after referred to as Hayashi ‘567
Claim 4:
Dan in combination with Atherton, Tokuchi, and Hayashi ‘109 does not explicitly teach the following limitations, however Hayashi ‘567 teaches:
The robot control system according to claim 3, wherein the camera is mounted on the robot, the preceding action includes a sound emitted by the user, and the processing circuitry is configured to orient the camera in a direction in which the sound is emitted
(Hayashi ‘567 - [0009] The recognizing unit searches for a sound emitting body from an image in which each sound source direction is filmed when a multiple of sound source directions are detected by the microphone array. ; [0014] A behavior control program in an aspect of the invention causes a computer to execute a function of detecting a sound source direction using a microphone array, a function of filming in a detected sound source direction, thereby acquiring an image, and a function of, when a sound emitting body having predetermined characteristics is detected in the image, identifying the sound emitting body as a sound source.)
Dan in combination with Atherton does not explicitly teach the following limitations, however Tokuchi teaches:
while the task remains paused.
(Tokuchi - [0162] When executing Step 5, the controller 441 notifies the computer 110 of the content of scheduled operation (scheduled operation) and waits for a response from the computer 110. …)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan combined with Atherton and Hayashi ‘109 to include a method of directing the robot towards an emitted sound as taught in Hayashi ‘567 and to further pause a task while waiting for the answer to an inquiry clarifying a task as taught in Tokuchi. Having the ability to bring the robots attention towards an emitted sound and to also have the ability to pause a task while awaiting clarification provides an additional method of communication between the operator and the robot for the purpose of conveying instructions while also increasing the efficiency of the tasks being performed in real time by a cooperative robot.
Claims 6, 18, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Dan (US 20230373098 A1) as modified by Atherton (US 20180173200 A1) and Tokuchi (US 20190082102 A1) in view of Hayashi (US 20220281109 A1), here after referred to as Hayashi ‘109 and in further view of Chang (US 20190184567 A1)
Claim 6:
Dan in combination with Atherton, Tokuchi, and Hayashi ‘109 does not explicitly teach the following limitations, however Chang teaches:
The robot control system according to claim 5, wherein the processing circuitry is configured to cause the robot to operate such that the recognition target is positioned at a center of a field of view of the camera.
(Chang – [0026] … When a gesture that is performed by the user hand 9 in the sensing zone 41 and that is sensed by the sensor device 4 matches the following gesture data 610, the control device 6 executes the following action instruction 620 to cause the driver module 31 to drive the flange 32 to follow movement of the user hand 9; when the gesture that is performed by the user hand 9 in the sensing zone 41 and that is sensed by the sensor device 4 matches the first inching gesture data 611, the control device 6 executes the first inching action instruction 621 to cause the driver module 31 to drive movement of the flange 32 by a first inching distance in the first direction (X); when the gesture that is performed by the user hand 9 in the sensing zone 41 and that is sensed by the sensor device 4 matches the second inching gesture data 612, the control device 6 executes the second inching action instruction 622 to cause the driver module 31 to drive movement of the flange 32 by a second inching distance in the second direction (Y); and when the gesture that is performed by the user hand 9 in the sensing zone 41 and that is sensed by the sensor device 4 matches the third inching gesture data 613, the control device 6 executes the third inching action instruction 623 to cause the driver module 31 to drive movement of the flange 32 by a third inching distance in the third direction (Z). … ; [see also figure 6])
Examiner Note:
Wherein the examiner notes that the “inching” process corresponds to centering the camera.
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan in combination with Atherton, Tokuchi, and Hayashi ‘109 to include a method of +centering the operators hand or “target” as taught in Chang. Having the ability to keeping the operators hand centered in view of the vision sensors ensures that accurate operator hand gestures are being communicated to the robot at all times.
Claim 18:
Dan teaches the following limitations:
The robot control system according to claim 6, wherein the recognition target is a hand of the user, and the processing circuitry is configured to extract a first shape of the hand from a first image and identify the first shape as the action, (Dan - [0010] A new method and system are provided herein for enabling intuitive programming of robots. The method and system enables non-experts to co-create automation tasks with robots based on an intuitive programming method through natural interaction via multiple channels. Speech input/output, recognition of objects and hand gestures, and a graphical user interface guide users through the co-creation of robotic tasks and help them handle potential errors. …) display an inquiry for the user confirming the identified action, extract a second shape of the hand in response to the inquiry from a second image, and update the at least part of the task based on the second shape confirmed action. (Dan – [0012] … In a second step, if an instruction is not understood, then the operator is prompted for clarification (e.g., which object?) and several choices may be offered (e.g., by showing the objects that are visible to the robot's camera). The operator is then able to provide feedback to the robot using one of the several input modalities (e.g., speech input, touch screen, hand gesture) so that the intended task can be completed.
Claim 19:
Dan in combination with Atherton and Tokuchi does not explicitly teach the following limitations, however Hayashi ‘109 teaches:
The robot control system according to claim 6, wherein the processing circuitry is configured to extract a reference object presented by the user from the image, identify a feature quantity of the reference object as a workpiece, wherein the workpiece is the subject of the action specified by the task.
(Hayashi ‘109 – [0043] Furthermore, the CPU 301 is connected to the parts supply device 16 via the interface 310 and the bus 315. The parts supply device 16 includes a sensor (not illustrated) that detects that a workpiece has been set, and the CPU 301 instructs the parts supply device 16 to set a new workpiece in accordance with an output of this sensor. ; [0065] FIGS. 5A to 5C each illustrate an example of an action of the robot body 1 in collaborative work between the robot body 1 and the worker 100, and the action has been registered in the LUT 11 in association with an ID. FIGS. 5A to 5C each illustrate work in which the robot body 1 takes a fully assembled workpiece 20 from a workbench 16 and transfers the workpiece 20 to the worker 100, the worker 100 examines the received workpiece 20 and transfers the workpiece 20 to the robot body 1, and the robot body 1 puts the workpiece 20 back on the workbench 16.)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan, Atherton, and Tokuchi in combination with Chang with a method of targeting the user’s hands or a reference object as taught in Hayashi ‘109. Having the ability to accurately target gestures and reference objects allows the system to receive instructions and clarifications from the user thus improving the ability of the user to collaborate with the robot.
Claims 7, 9, and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Dan (US 20230373098 A1) as modified by Atherton (US 20180173200 A1) in view of Chang (US 20190184567 A1) and Tokuchi (US 20190082102 A1)
Claim 7:
Dan in combination with Atherton and Tokuchi does not explicitly teach the following limitations, however Chang teaches:
The robot control system according to claim 2, wherein the recognition target is a hand of the user, and the processing circuitry is configured to extract a shape of the hand from the image and to identify the shape to determine the instruction.
(Chang - [0022] Referring to FIGS. 2, 3 and 5, the control device 6 is electrically coupled to the driver module 31, the sensor device 4 and the light emitting device 5, and has a gesture database 61, and an action instruction database 62 corresponding to the gesture database 61. When the gesture that is performed by the user hand 9 in the sensing zone 41 and that is sensed by the sensor device 4 matches a gesture included in the gesture database 61, which may for example be determined according to comparison by the control device 6, the control device 6 causes the driver module 31 to execute an action instruction that is included in the action instruction database 62 and that corresponds to the matched gesture included in the gesture database 61.; [see also Figures 7-10])
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan combined with Atherton and Tokuchi to include a method of recognizing an operators hand shape and interpreting the shape as a gesture associated with an action as taught in Chang. Having the ability to associate operator hand gestures with robot actions reduces the need for complex programming and further allows the operator to interactively communicate in real time with a cooperative robot.
Claim 9:
Dan teaches the following limitations:
The robot control system according to claim 7, wherein the processing circuitry is configured to extract a first shape of the hand from a first image and to determine the instruction from the first shape, (Dan - [0010] A new method and system are provided herein for enabling intuitive programming of robots. The method and system enables non-experts to co-create automation tasks with robots based on an intuitive programming method through natural interaction via multiple channels. Speech input/output, recognition of objects and hand gestures, and a graphical user interface guide users through the co-creation of robotic tasks and help them handle potential errors. …) generate an additional inquiry for the user regarding whether or not to confirm the instruction, extract a second shape of the hand in response to the additional inquiry from a second image, confirm the instruction, and execute at least part of the task based on the confirmed instruction. (Dan – [0012] … In a second step, if an instruction is not understood, then the operator is prompted for clarification (e.g., which object?) and several choices may be offered (e.g., by showing the objects that are visible to the robot's camera). The operator is then able to provide feedback to the robot using one of the several input modalities (e.g., speech input, touch screen, hand gesture) so that the intended task can be completed.
Claim 10:
Dan in combination with Atherton and Tokuchi does not explicitly teach the following limitations, however Chang teaches:
The robot control system according to claim 7, further comprising: a memory that stores executes at least part of the task as a task part, wherein the processing circuitry is configured to acquire the task part from the memory when the identified action corresponds to a hand shape set in advance and control the robot such that the robot executes a task including the task part.
(Chang - … The control device is electrically coupled to the driver module and the sensor device, has a gesture database, and an action instruction database corresponding to the gesture database, and is configured to, when the gesture that is performed by the user hand in the sensing zone and that is sensed by the sensor device matches a piece of gesture data included in the gesture database, cause the driver module to execute an action instruction that is included in the action instruction database and that corresponds to the piece of gesture data.; [see also Figures 7-10])
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan combined with Atherton and Tokuchi to include a method of storing a library of hand gestures in a database, as taught in Chang. Having the ability to store and retrieve various hand gestures from a database allows the operator to have a large inventory of hand gestures which in turn improves the operators ability to communicate with a cooperative robot.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Dan (US 20230373098 A1) as modified by Atherton (US 20180173200 A1) and Tokuchi (US 20190082102 A1) in view of Chang (US 20190184567 A1) in further view of Huang (US 20150217450 A1)
Claim 8:
Dan, Atherton, and Tokuchi in combination with Chang does not explicitly teach the following limitations, however Huang teaches
The robot control system according to claim 7, wherein the processing circuitry is configured to update a target position for the task executed by the robot as the at least part of the task based on the shape.
(Huang - [0030] Then, the operator views the position of the arm unit 11 in the eye-to-hand frame 23. Normally, the arm unit 11 is farther away from the target work bench 13. Then, the operator shows an arm control gesture. After the arm control gesture is detected and recognized by the gesture recognition module 25, the arm control gesture is shown in the arm control gesture area 26a of the eye-to-hand frame 23. Meanwhile, the operator views the arm ambient image in the eye-to-hand frame 23, and controls the arm unit 11 to quickly move at a high velocity, such that the hand unit 12 disposed at the movement end of the arm unit 11 can quickly reach the target work bench 13 until the image of the target work bench 13 is shown in the eye-in-hand frame 24. ; [See also Figures 2 and 3])
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan, Atherton and Tokuchi in combination with Chang with a method of positioning a cooperative robot through the use of recognized hand gestures as taught in Huang. Having the ability to associate operator hand gestures with robot positions reduces the need for complex programming and further allows the operator to interactively communicate a target location in real time with a cooperative robot.
Claims 12 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Dan (US 20230373098 A1) as modified by Atherton (US 20180173200 A1) and Tokuchi (US 20190082102 A1) in view of Pirjanian (US 20040193322 A1)
Claim 12:
Dan in combination with Atherton and Tokuchi does not explicitly teach the following limitations, however Pirjanian teaches:
The robot control system according to claim 1, wherein the sensor is the camera, and the processing circuitry is configured to define the task, (Pirjanian - [0039] When in the ready state, the system's parsing and execution can be triggered in a number of ways. For example, in a "quick execute" mode, the system can repeatedly check whether the current program is valid; if so, the system will execute the program promptly, by switching to an "execute" state. In a normal mode, a "start" card can be used to trigger the parsing of the program, by switching to a "parse" state. If the program is correctly parsed, the machine switches to the execute state.)
inquire the user about a document indicating the task, define the task based on the document presented by the user, and (Pirjanian - [0008] Embodiments of the invention include methods and apparatus for programming and/or control of a computer system via a video camera or other imaging device that is coupled to the computer system. The computer system that is programmed and/or is controlled can correspond to, for example, a robot. Objects in the environment, such as printed cards, can be placed within the field of view of the video camera or other imaging device. Indicia on the cards can be recognized and associated with one or more programming instructions or computer commands for control.) complement at least part of the defined task based on the identified action. (Pirjanian - [0013] One embodiment is a computer program embodied in a tangible medium for controlling a device, where the computer program includes: a module with instructions configured to visually recognize indicia that are visible on at least one surface of one or more planar objects, where at least one surface of the planar objects includes indicia, where at least a portion of the indicia is machine readable and at least a portion is human recognizable; a module with instructions configured to automatically associate at least some of the recognized indicia with one or more executable program instructions; and a module with instructions configured to arrange the one or more executable program instructions to create at least a portion of a computer program.)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan in combination with Atherton and Tokuchi to include a method of using graphic symbols or markers to control robot actions, as taught in Pirjanian. Having the ability to control the robot via graphic interchangeable graphic symbols provides an alternative route of communication with a high degree of accuracy when interacting with a cooperative robot.
Claim 13:
Dan in combination with Atherton and Tokuchi does not explicitly teach the following limitations, however Pirjanian teaches:
The robot control system according to claim 12, wherein the document includes a syntax and a variable regarding the task, and the processing circuitry is further configured to define the task based on the syntax,
(Pirjanian – [0020] … Advantageously, objects in the environment, such as printed cards, can be placed within the field of view of the video camera or other imaging device. Indicia on the cards, as well as other visual features and/or cues in the environment, can be recognized and associated with one or more programming instructions or computer commands for control. … ; [0049] A programming system can be configured to respond to a variety of types of printed cards. One implementation of the system can respond to two types of printed cards: command cards that trigger certain actions (such as stopping or listing the current program) and token cards that comprise actual program elements (tokens) of a programming language. It will be understood that in another embodiment, the system can respond to one type of card or to more than two types of cards. ; [See also Figure 2])
Dan further teaches the following limitations:
generate the inquiry for the user regarding the variable, identify the action of the user in response to the inquiry, and update a value to be assigned to the variable based on the identified action.
(Dan - [0012] A first step of the interaction involves the operator instructing the robot to perform a sequence of operations by providing high level concepts (e.g., pick, move, place) and object names known to humans (e.g., block, pencil, screw, this object here). The instructions may be enhanced with attributes (e.g., red pen, big screw, rotate clockwise, etc.). In a second step, if an instruction is not understood, then the operator is prompted for clarification (e.g., which object?) and several choices may be offered (e.g., by showing the objects that are visible to the robot's camera). The operator is then able to provide feedback to the robot using one of the several input modalities (e.g., speech input, touch screen, hand gesture) so that the intended task can be completed.)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan, Atherton and Tokuchi to provide a syntax for instructions as taught in Pirjanian. Having the ability to clarify the operators instructions using a clear set of syntax and graphics ensures that the correct level of detail regarding task attributes is conveyed to the cooperative robot when requesting an action and improves communication between the user and robot.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Dan (US 20230373098 A1) as modified by Atherton (US 20180173200 A1) and Tokuchi (US 20190082102 A1) in view of Yang (US 20240359321 A1)
Claim 14:
Dan in combination with Atherton and Tokuchi does not explicitly teach the following limitations, however Yang teaches:
The robot control system according to claim 1, wherein the processing circuitry is configured to cause the robot to operate such that the robot assumes an inquiry posture, which is a posture corresponding to the inquiry, and generate the inquiry after the robot assumes the inquiry posture
(Yang - [0039] In step S1, the operator may use a hand gesture to trigger the teaching mode of the robotic arm 101. In response the robotic arm 101 may move to a standby position. Alternatively, the standby mode may be triggered by input commands different from a hand gesture, such as a voice command or input via a user interface element on a graphical user interface. ; [0040] In step S2, the operator may use a hand gesture to move the robotic arm 101 until the robotic arm 101 has reached the intended start position for the task to be taught. The operator may indicate that the start position has been reached by way of a hand gesture or other suitable command.)
Dan in combination with Atherton does not explicitly teach the following limitations, however Tokuchi teaches:
and pauses execution of the task.
(Tokuchi - [0162] When executing Step 5, the controller 441 notifies the computer 110 of the content of scheduled operation (scheduled operation) and waits for a response from the computer 110. …)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan in combination with Atherton to include a method of pausing a task while waiting for the answer to an inquiry clarifying a task as taught in Tokuchi and to further provide a method of orienting the robot in a standby position prior to instruction as taught in Yang. Having the ability to pause a task and then place the robot in the correct orientation to receive instruction ensures that graphic, auditory, and hand gesture instructions can be communicated with the robot in the best position to accept the information thereby improving accuracy, efficiency, and safety.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Pirjanian (US 20040193322 A1) as modified by Dan (US 20230373098 A1) in view of Tokuchi (US 20190082102 A1)
Claim 15:
Pirjanian teaches the following limitations:
A robot control system, comprising: processing circuitry configured to generate an inquiry for a user about a document indicating a task to be executed by a robot
(Pirjanian - [0039] When in the ready state, the system's parsing and execution can be triggered in a number of ways. For example, in a "quick execute" mode, the system can repeatedly check whether the current program is valid; if so, the system will execute the program promptly, by switching to an "execute" state. In a normal mode, a "start" card can be used to trigger the parsing of the program, by switching to a "parse" state. If the program is correctly parsed, the machine switches to the execute state.)
identify, using a sensor, the document presented by the user from an image captured by a camera, the identified document
(Pirjanian - [0008] Embodiments of the invention include methods and apparatus for programming and/or control of a computer system via a video camera or other imaging device that is coupled to the computer system. The computer system that is programmed and/or is controlled can correspond to, for example, a robot. Objects in the environment, such as printed cards, can be placed within the field of view of the video camera or other imaging device. Indicia on the cards can be recognized and associated with one or more programming instructions or computer commands for control.)
control the robot such that the robot executes the task.
(Pirjanian - [0013] One embodiment is a computer program embodied in a tangible medium for controlling a device, where the computer program includes: a module with instructions configured to visually recognize indicia that are visible on at least one surface of one or more planar objects, where at least one surface of the planar objects includes indicia, where at least a portion of the indicia is machine readable and at least a portion is human recognizable; a module with instructions configured to automatically associate at least some of the recognized indicia with one or more executable program instructions; and a module with instructions configured to arrange the one or more executable program instructions to create at least a portion of a computer program. ; [0061] In a single-player mode, the user can compete against time to see how quickly the user can get the robot to perform a task, or to see how few cards the user can use to accomplish the task. In a multi-player mode, users can compete with each other and even against other users who have posted their scores on an associated Web site. …)
Pirjanian does not explicitly teach the following limitations, however Dan teaches:
in response to missing information regarding the task,
(Dan - [0012] … In a second step, if an instruction is not understood, then the operator is prompted for clarification (e.g., which object?) and several choices may be offered (e.g., by showing the objects that are visible to the robot's camera). The operator is then able to provide feedback to the robot using one of the several input modalities (e.g., speech input, touch screen, hand gesture) so that the intended task can be completed.)
providing the robot with instruction on how to execute the task, addressing the missing information
(Dan - [0012] … In a second step, if an instruction is not understood, then the operator is prompted for clarification (e.g., which object?) and several choices may be offered (e.g., by showing the objects that are visible to the robot's camera). The operator is then able to provide feedback to the robot using one of the several input modalities (e.g., speech input, touch screen, hand gesture) so that the intended task can be completed.)
Pirjanian in combination with Dan does not explicitly teach the following limitations, however Tokuchi teaches:
pause execution of the task upon generation of the inquiry,
(Tokuchi - [0162] When executing Step 5, the controller 441 notifies the computer 110 of the content of scheduled operation (scheduled operation) and waits for a response from the computer 110. …)
while the task remains paused,
(Tokuchi - [0162] When executing Step 5, the controller 441 notifies the computer 110 of the content of scheduled operation (scheduled operation) and waits for a response from the computer 110. …)
resume the task using the instruction from the inquiry on how to execute the task to address the missing information.
(Tokuchi - [0162] … Here, when receiving a response indicating that the scheduled operation is accepted (when an affirmative result is obtained in Step 6), the controller 441 instructs the device concerned (for example, the cleaning robot 158) to perform the operation specified in Step 4 (Step 12).)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Pirjanian to include a method for having the robot inquire with the user regarding missing information, as taught in Dan and to further pause a task while waiting for the answer to an inquiry and resume a task after receiving a satisfactory resolution to an inquiry as taught in Tokuchi. Giving the robot the capability to prompt the user with an inquiry regarding missing instructions and then to pause or resume a task based on its inquiry status ensures the accuracy of robot tasks and improves the communication between the user and the robot.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Dan (US 20230373098 A1) as modified by Atherton (US 20180173200 A1) and Tokuchi (US 20190082102 A1) in view of Hayashi (US 20220281109 A1), here after referred to as Hayashi ‘109 and in further view of Chang (US 20190184567 A1) and Pirjanian (US 20040193322 A1)
Claim 20:
Dan, Atherton and Tokuchi in combination with Hayashi ‘109 and Chang does not explicitly teach the following limitations, however Pirjanian teaches:
The robot control system according to claim 6, wherein the processing circuitry is configured to define the task, (Pirjanian - [0039] When in the ready state, the system's parsing and execution can be triggered in a number of ways. For example, in a "quick execute" mode, the system can repeatedly check whether the current program is valid; if so, the system will execute the program promptly, by switching to an "execute" state. In a normal mode, a "start" card can be used to trigger the parsing of the program, by switching to a "parse" state. If the program is correctly parsed, the machine switches to the execute state.) inquire the user about a document indicating the task, define the task based on the document presented by the user, and (Pirjanian - [0008] Embodiments of the invention include methods and apparatus for programming and/or control of a computer system via a video camera or other imaging device that is coupled to the computer system. The computer system that is programmed and/or is controlled can correspond to, for example, a robot. Objects in the environment, such as printed cards, can be placed within the field of view of the video camera or other imaging device. Indicia on the cards can be recognized and associated with one or more programming instructions or computer commands for control.) execute at least part of the defined task based on the identified action. (Pirjanian - [0013] One embodiment is a computer program embodied in a tangible medium for controlling a device, where the computer program includes: a module with instructions configured to visually recognize indicia that are visible on at least one surface of one or more planar objects, where at least one surface of the planar objects includes indicia, where at least a portion of the indicia is machine readable and at least a portion is human recognizable; a module with instructions configured to automatically associate at least some of the recognized indicia with one or more executable program instructions; and a module with instructions configured to arrange the one or more executable program instructions to create at least a portion of a computer program.)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Dan in combination with Atherton, Tokuchi, Hayashi ‘109, and Chang to include a method of controlling robot tasks interchangeable graphic symbols, as taught in Pirjanian. Having the ability to control the robot via interchangeable graphic symbols provides an alternative route of communication with a high degree of accuracy when interacting with a cooperative robot.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure or directed to the state of the art is listed on the enclosed PTO-892.
The following is a brief description for relevant prior art that was cited but not applied:
Linnell (US 9737987 B1) describes systems and methods for detecting a graphic card that visually describes an operational mode of a rotatable interface component via a plurality of curves for rotationally-varying parameters, determining the operational mode that is visually described on the graphic card, and loading the operational mode to the rotatable interface component, where the operational mode specifies operations for a motor such that the motor generates torque on the interface component based on the curves for the rotationally-varying parameters that are shown on the graphic card.
Ando (US 20180354127 A1) describes an operation information generating apparatus includes: a task data recording unit configured to record, as task data, information regarding an operation of a person from a start to an end of a task; an operation classifying unit configured to divide an overall operation from the start to the end of the task into a plurality of partial operations; and an operation combining unit configured to select a best partial operation for each partial operation from a plurality of samples of the task data for the same task, and generates data of an optimum operation of the entire task by combining the selected best partial operations.
Forutanpour (US 20150091790 A1) describes an apparatus, a method, and a computer program product for gesture recognition. The apparatus classifies a gesture based on a movement of a body part as detected by a primary sensor. The apparatus determine a reliability level of a secondary sensor and obtains corroborating information associated with the movement of the body part using the secondary sensor when the reliability level satisfies a criterion. The apparatus then confirms or negates the classification of the gesture based on the corroborating information.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALAN LINDSAY OSTROW whose telephone number is (703)756-1854. The examiner can normally be reached M-F 8 - 5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Mott can be reached on (571) 270 5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALAN LINDSAY OSTROW/
Examiner, Art Unit 3657
/JONATHAN L SAMPLE/Primary Examiner, Art Unit 3657
Prosecution Insights