Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This Office action is in response to the amendment filed on 05/28/2026. Claims 1-5, 8-9, and 11-17 are currently pending with claims 1, 4, and 14-16 being amended, and claims 5-6 being cancelled.
Response to Amendment
The amendments to the claims submitted on 0/28/2026 overcome the claim objections set forth in the previous Office action except for those set forth in the claim objection section.
Response to Arguments
Examiner notes wherein Applicant argues the newly amended limitations, which have not been addressed by the prior art of record. As such, Examiner has augmented the below rejection(s) in view of the prior art of record to address the newly amended limitations.
Applicant’s amendments/arguments, see communications, filed 05/25/2026, with respect to the rejection(s) of claim(s) 1-6, 8-9, and 11-17 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Gupta et al. (US 20170236446 A1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-5, 8-9 and 13-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Takeuchi et al. (US 20190232493 A1), hereinafter Takeuchi in view of Jin et al. (US 20230236804 A1), hereinafter Jin and Gupta et al. (US 20170236446 A1), hereinafter Gupta.
Regarding claim 1, Takeuchi teaches:
1. (Currently Amended) A teaching device configured to create a control program of a robot comprising:
a processor (Paragraph 0057, “The robot control device 200 controls the arm 110, the end effector 140, the transport device 50, and the camera 30. The functions of the robot control device 200 are realized, for example, by a computer with a processor and a memory executing a computer program.”) configured to:
generate a program creation screen for performing program creation by a plurality of commands each representing a function constituting a control program of the robot (Paragraph 0005, "The robot control device includes a display control unit that displays an input screen including an operation flow creation area for creating an operation flow of work on a display device; a conversion unit that converts the created operation flow into a control program; and a control execution unit that executes the control program to control the robot. The input screen is configured to display a plurality of operation objects indicating a plurality of operations including an operation using force control, and one or more conditional branch objects indicating a conditional branch, as options. The operation flow creation area is configured to create an operation flow including the conditional branch by graphically placing an operation object selected from the plurality of operation objects and the conditional branch object." As well as Paragraph 0082, “FIG. 7 shows an example of operation classifications and operation objects constituting an operation flow, and FIGS. 8A to 8D show outlines of operations of some operation objects. A plurality of operation objects can be categorized into the following four categories. All of these operations involve force control.”) and …
cause a display to display, (Paragraph 0066, "Parameter setting area PR is an area for setting work parameters related to the entire work or operation parameters related to individual operations." and Paragraph 0081, " In FIG. 6D, when one of the blocks SB1 and OB1 to OB4 placed in the operation flow creation area FL is selected, the parameters corresponding to the selected block are displayed in the parameter setting area PR. For example, when the sequence block SB1 is selected, the work parameters related to the entire sequence are displayed. When one the objects OB1 to OB4 of the object is selected, the parameters related to the objects are displayed. In the example of FIG. 6D, the parameters related to the conditional branch block OB2 are displayed. These parameters are changed as necessary." Please also see Figures 6E and 7 which demonstrate that different operation objects may be selected to add to the program and information on these objects is displayed during selection.) when a command, … of the plurality of commands disposed in the program creation screen (Paragraph 0005, "The robot control device includes a display control unit that displays an input screen including an operation flow creation area for creating an operation flow of work on a display device; a conversion unit that converts the created operation flow into a control program; and a control execution unit that executes the control program to control the robot. The input screen is configured to display a plurality of operation objects indicating a plurality of operations including an operation using force control, and one or more conditional branch objects indicating a conditional branch, as options. The operation flow creation area is configured to create an operation flow including the conditional branch by graphically placing an operation object selected from the plurality of operation objects and the conditional branch object." As well as Paragraph 0082, “FIG. 7 shows an example of operation classifications and operation objects constituting an operation flow, and FIGS. 8A to 8D show outlines of operations of some operation objects. A plurality of operation objects can be categorized into the following four categories. All of these operations involve force control.”) is executed by execution of the control program or is selected, (Paragraph 0081, "In FIG. 6D, when one of the blocks SB1 and OB1 to OB4 placed in the operation flow creation area FL is selected, the parameters corresponding to the selected block are displayed in the parameter setting area PR. For example, when the sequence block SB1 is selected, the work parameters related to the entire sequence are displayed. When one the objects OB1 to OB4 of the object is selected, the parameters related to the objects are displayed. In the example of FIG. 6D, the parameters related to the conditional branch block OB2 are displayed. These parameters are changed as necessary." as well as Paragraphs 0157-0158) … and
execute the control program in order to control the robot, (Paragraphs 0157-0158, "FIG. 12 shows an example of a window W3 as an input screen displayed when the operation flow is converted into a control program according to the indication of the teacher. This window W3 includes a sequence selecting area F31 for selecting a sequence (that is an operation flow) to be converted into a control program and a program file name setting area F32 for setting the file name of the control program. The sequence selecting area F31 may be omitted. The window W3 may be configured to include an area for designating information such as a storage destination of parameters. In the window W3, when the “execution” button is pressed, the operation flow is converted into a control program by the conversion unit 244, and the control program is stored in the memory 220.
In step S140 of FIG. 5, the robot control device 200 controls the robot and causes the robot to execute the work according to the control program created in step S130. This work can be executed as a checking work for checking the operations of the robot 100 on the production line or an original work for manufacturing a product on the production line.") … and
…
Takeuchi does not specifically teach the display being a pop-up display. Or displaying information on a control block when that block is executed during execution of the program. However, Gupta, in the same field of endeavor of robotics, teaches:
… setting a parameter value of the plurality of commands; … with a set parameter value, (Paragraphs 0028-0029, "Events can be associated with one or more toy robot input systems (e.g., sensors or input devices), associated with other input systems, associated with time intervals, associated with progress through actions associated with a state, or associated with other suitable conditions. Each event preferably represents a fixed set of input variable values (e.g., a specific input received at a specific location relative to the robot, a specific wavelength of sound, a specific input threshold value being exceeded, a sustained input received for more than a threshold period of time, etc.), but can alternatively represent a variable input (e.g., a range of inputs, etc.), represent an output, represent any other suitable robot parameter, or represent any other suitable parameter. In a specific example, examples of events can include timers, robot tilt (e.g., relative to a gravity vector, relative to a previous robot orientation, etc.), robot acceleration (e.g., along or against a gravity vector, along a robot axis, along a vector at an angle relative to a gravity vector and/or robot axis, etc.), button depression, button release, sound detection, sound direction detection, secondary robot direction detection (e.g., based on the beacon strength of a secondary robot, based on visual system feedback, etc.), secondary robot proximity detection, obstruction detection (e.g., based on motor feedback or visual systems), overcoming the obstruction, robot component actuation (e.g., head turning, tail wagging, wheels turning, etc.), component movement cessation, robot vibration or shaking, or include any other suitable event.
[0029] Each icon can illustrate the toy robot in the state (e.g., respective associated state) or undergoing the event (e.g., respective associated event); can include any other suitable illustration or representation representative of the respective state(s) or event(s) associated with the icon (e.g., animation, text, etc.); be otherwise graphically associated with the state(s) or event(s); or otherwise suitably associated with a program element (e.g., state, event, etc.). In some embodiments, a state icon can illustrate a toy robot subsystem operating in a state associated with that subsystem, and an event icon can illustrate a toy robot input system undergoing an event associated with that input system.") … information related to an execution state of the executed or selected command, (Paragraph 0064, "The data is preferably determined (e.g., received) by the user device, but can alternatively be determined by the robot, a remote computing system, or by any other suitable system. The data is preferably measured by the robot sensors and sent by the robot to the user device, but can alternatively be measured by the user device (e.g., by the camera or microphone of the user device), be measured by a second user device, be measured by a second robot, be measured by an independent sensor or sensors (e.g., camera, microphone, proximity detector, trilateration system), or be measured by any other suitable input. All data measured by the robot is preferably sent to the user device (e.g., streamed to the user device, sent at a predetermined frequency, etc.), wherein the user device preferably processes the data and determines whether an event was satisfied, wherein the event is associated with the transition connected to the previous state for which machine code was sent to the robot. Examples of data indicative of robot inputs include robot acceleration (e.g., rolling, dropping, etc.), robot button presses, sounds, the identifiers of sound sensors detecting sound, light intensity, the identifiers of light sensors detecting light, component motor feedback, visual system measurements, or any other suitable data indicative of an event." as well as Paragraph 0068, " Executing the program S300 can include tracking the current state of program execution (e.g., as related to the state machine), preferably at the user device, but alternately at the robot, at another robot or user device, or at any suitable device. Tracking the current state can include tracking the instructed state (e.g., a first state, wherein the instruction sent in the most recent performance of Block S310 is an instruction to operate in the first state), tracking the state in which the robot is operating (e.g., based on confirmation received from the robot, determined from data indicative of robot operation), or tracking any other state. An indication of the current state (e.g., indication that the robot has entered the current state) can be displayed during program execution, not be displayed at all, or otherwise presented. The indication is preferably displayed on the graphical representation of the state machine (e.g., in the graphical representation, highlighting the state icon associated with the current state), but alternately can be displayed as a graphical representation of the state independent of any graphical representation of the state machine, can be displayed as text, or can be otherwise suitably displayed. The indication can be displayed beginning substantially concurrent with sending the instruction to operate in a state (e.g., within five seconds of sending the instruction, within 500 ms of sending the instruction, within 30 ms of sending the instruction, etc.), substantially concurrent with determining that the robot is operating in the state or has entered the instructed state, after some preset time delay, in response to receiving a request to display the current state, or at any other suitable time.") the information being displayed during execution of the executed or selected command; (Paragraph 0068, “Executing the program S300 can include tracking the current state of program execution (e.g., as related to the state machine), preferably at the user device, but alternately at the robot, at another robot or user device, or at any suitable device. Tracking the current state can include tracking the instructed state (e.g., a first state, wherein the instruction sent in the most recent performance of Block S310 is an instruction to operate in the first state), tracking the state in which the robot is operating (e.g., based on confirmation received from the robot, determined from data indicative of robot operation), or tracking any other state. An indication of the current state (e.g., indication that the robot has entered the current state) can be displayed during program execution, not be displayed at all, or otherwise presented. The indication is preferably displayed on the graphical representation of the state machine (e.g., in the graphical representation, highlighting the state icon associated with the current state), but alternately can be displayed as a graphical representation of the state independent of any graphical representation of the state machine, can be displayed as text, or can be otherwise suitably displayed. The indication can be displayed beginning substantially concurrent with sending the instruction to operate in a state (e.g., within five seconds of sending the instruction, within 500 ms of sending the instruction, within 30 ms of sending the instruction, etc.), substantially concurrent with determining that the robot is operating in the state or has entered the instructed state, after some preset time delay, in response to receiving a request to display the current state, or at any other suitable time.”) … the information related to the execution state includes at least one of a state, during the execution of the command, of a sensor or equipment associated with the command, information related to a detection value, during the execution of the command, of the sensor associated with the command, (Paragraph 0062, "Sending sensor data at the robot S330 functions to communicate data potentially indicative of the occurrence of an event. The sensor data is preferably robot sensor data, but can alternatively be user device data, or be any other suitable data. The sensor data can be sampled by the robot (e.g., by the robot sensors) and/or recorded by the robot (e.g., by the robot processor and/or memory), or otherwise determined. The sensor data can include sensor measurements, sensor signals, data synthesized from one or more sensor measurements (e.g., ratios between different types of sensor measurements, measurement rate of change, etc.), or any other suitable sensor data. The sensor data can be sent by the robot continuously, or sent in batches (e.g., sent at a predetermined frequency, sent in response to receiving a request to send one or more batches), or sent in any other suitable manner. The robot preferably samples, records, and/or sends the sensor data during instruction execution (e.g., during robot operation in the first state), but can alternatively obtain and/or process sensor data in any other suitable manner at any other suitable time. The robot can begin sampling, recording, and/or sending the sensor data substantially concurrently with a communication or execution incident (e.g., connecting to the user device, receiving an indication that execution will begin, receiving an instruction, operating according to an instruction S320, completing operation according to an instruction, receiving a request for sensor data, etc.), after a time interval (e.g., a predetermined time interval, a time interval specified by an instruction) following a communication or execution incident, in response to entering a state (e.g., an “on” state), or at any other suitable time. The robot can stop sampling, recording, and/or sending the sensor data substantially concurrently with a communication or execution incident (e.g. disconnecting from the user device, receiving an indication that execution will end, receiving an instruction, operating according to an instruction S320, completing operation according to an instruction, receiving a request to stop sending sensor data, losing connection to the user device, etc.), after a time interval following a communication or execution incident, in response to entering an off state, or at any other suitable time. The robot can continue sampling, recording, and/or sending the sensor data (e.g., continuously, sporadically, at a predetermined frequency, etc.) during the time interval between beginning sampling, recording, and/or sending the sensor data and ending sampling, recording, and/or sending the sensor data.") or an internal variable, during the execution of the command, of the control program related to the command. (Paragraphs 0028-0029, "Events can be associated with one or more toy robot input systems (e.g., sensors or input devices), associated with other input systems, associated with time intervals, associated with progress through actions associated with a state, or associated with other suitable conditions. Each event preferably represents a fixed set of input variable values (e.g., a specific input received at a specific location relative to the robot, a specific wavelength of sound, a specific input threshold value being exceeded, a sustained input received for more than a threshold period of time, etc.), but can alternatively represent a variable input (e.g., a range of inputs, etc.), represent an output, represent any other suitable robot parameter, or represent any other suitable parameter. In a specific example, examples of events can include timers, robot tilt (e.g., relative to a gravity vector, relative to a previous robot orientation, etc.), robot acceleration (e.g., along or against a gravity vector, along a robot axis, along a vector at an angle relative to a gravity vector and/or robot axis, etc.), button depression, button release, sound detection, sound direction detection, secondary robot direction detection (e.g., based on the beacon strength of a secondary robot, based on visual system feedback, etc.), secondary robot proximity detection, obstruction detection (e.g., based on motor feedback or visual systems), overcoming the obstruction, robot component actuation (e.g., head turning, tail wagging, wheels turning, etc.), component movement cessation, robot vibration or shaking, or include any other suitable event.
Each icon can illustrate the toy robot in the state (e.g., respective associated state) or undergoing the event (e.g., respective associated event); can include any other suitable illustration or representation representative of the respective state(s) or event(s) associated with the icon (e.g., animation, text, etc.); be otherwise graphically associated with the state(s) or event(s); or otherwise suitably associated with a program element (e.g., state, event, etc.). In some embodiments, a state icon can illustrate a toy robot subsystem operating in a state associated with that subsystem, and an event icon can illustrate a toy robot input system undergoing an event associated with that input system.")
However, Jin, in the same field of endeavor of robotics, teaches:
… wherein the processor is further configured to cause the display to display the information related to the execution state in a pop-up display, … (Paragraph 0017, "As another example, some embodiments include an apparatus (10) for automatic programming, comprising: a user interaction module (201), configured to receive a user's request on creating a first global parameter of a specific type by detecting the user's dragging a control corresponding to the specific type from a first user interface (701) and dropping the control on a second user interface (702); a displaying module (202), configured to display a first popup window (804) for editing the first global parameter on the second user interface (702); a processing module (203), configured to receive the user's editing operations on the first global parameter in the first popup window (804); and create the first global parameter according to the user's editing operations on the first global parameter; and the displaying module (202), further configured to display a new added first card (802) of the first global parameter in a list (802′) of global parameters on a third user interface (703).")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the workflow generation and robotic system as taught by Takeuchi with the methods to display current state of program execution during operation of the program as taught by Gupta and with ability to use a pop up display as taught by Jin. This would enable a user to continue their work without disruption while accessing focused information within the pop up window and would further allow a user to efficiently follow the execution of the program and provide necessary input and/or debug the program as needed.
Regarding claim 2, where all the limitations of claim 1 are discussed above, Takeuchi further teaches:
2. (Previously Presented) The teaching device according to claim 1, wherein the program creation screen includes a first region that displays a list of the plurality of commands, and (Paragraph 0063-0065, "(1) Main view area MV is an area for displaying options of operation objects and conditional branch objects to be described later, execution results of a control program, and the like.
(2) Operation flow creation area FL is an area for displaying the operation flows in which a plurality of objects are graphically placed in an editable manner. The work represented by the operation flow is also called “sequence”.
(3) Sequence display area SQ is an area for displaying a tree structure of the sequence." As well as Paragraph 0082, “FIG. 7 shows an example of operation classifications and operation objects constituting an operation flow, and FIGS. 8A to 8D show outlines of operations of some operation objects. A plurality of operation objects can be categorized into the following four categories. All of these operations involve force control.”)
a second region for creating the control program by disposing one or more commands selected from the first region, and (Paragraph 0064, "Operation flow creation area FL is an area for displaying the operation flows in which a plurality of objects are graphically placed in an editable manner. The work represented by the operation flow is also called “sequence”.")
the processor (Paragraph 0057, “The robot control device 200 controls the arm 110, the end effector 140, the transport device 50, and the camera 30. The functions of the robot control device 200 are realized, for example, by a computer with a processor and a memory executing a computer program.”) is further configured to cause the display to displays, in response to selection on a command in the first region or the second region or execution instruction on a command disposed in the second region, … (Paragraph 0066, "Parameter setting area PR is an area for setting work parameters related to the entire work or operation parameters related to individual operations." and Paragraph 0081, " In FIG. 6D, when one of the blocks SB1 and OB1 to OB4 placed in the operation flow creation area FL is selected, the parameters corresponding to the selected block are displayed in the parameter setting area PR. For example, when the sequence block SB1 is selected, the work parameters related to the entire sequence are displayed. When one the objects OB1 to OB4 of the object is selected, the parameters related to the objects are displayed. In the example of FIG. 6D, the parameters related to the conditional branch block OB2 are displayed. These parameters are changed as necessary." Please also see Figures 6E and 7 which demonstrate that different operation objects may be selected to add to the program and information on these objects is displayed during selection.)
While Takeuchi does teach displaying an execution information area. They do not explicitly state that this is the execution state information. However, Gupta, in the same field of endeavor of robotics, teaches:
… the information. (Paragraph 0064, "The data is preferably determined (e.g., received) by the user device, but can alternatively be determined by the robot, a remote computing system, or by any other suitable system. The data is preferably measured by the robot sensors and sent by the robot to the user device, but can alternatively be measured by the user device (e.g., by the camera or microphone of the user device), be measured by a second user device, be measured by a second robot, be measured by an independent sensor or sensors (e.g., camera, microphone, proximity detector, trilateration system), or be measured by any other suitable input. All data measured by the robot is preferably sent to the user device (e.g., streamed to the user device, sent at a predetermined frequency, etc.), wherein the user device preferably processes the data and determines whether an event was satisfied, wherein the event is associated with the transition connected to the previous state for which machine code was sent to the robot. Examples of data indicative of robot inputs include robot acceleration (e.g., rolling, dropping, etc.), robot button presses, sounds, the identifiers of sound sensors detecting sound, light intensity, the identifiers of light sensors detecting light, component motor feedback, visual system measurements, or any other suitable data indicative of an event." as well as Paragraph 0068, " Executing the program S300 can include tracking the current state of program execution (e.g., as related to the state machine), preferably at the user device, but alternately at the robot, at another robot or user device, or at any suitable device. Tracking the current state can include tracking the instructed state (e.g., a first state, wherein the instruction sent in the most recent performance of Block S310 is an instruction to operate in the first state), tracking the state in which the robot is operating (e.g., based on confirmation received from the robot, determined from data indicative of robot operation), or tracking any other state. An indication of the current state (e.g., indication that the robot has entered the current state) can be displayed during program execution, not be displayed at all, or otherwise presented. The indication is preferably displayed on the graphical representation of the state machine (e.g., in the graphical representation, highlighting the state icon associated with the current state), but alternately can be displayed as a graphical representation of the state independent of any graphical representation of the state machine, can be displayed as text, or can be otherwise suitably displayed. The indication can be displayed beginning substantially concurrent with sending the instruction to operate in a state (e.g., within five seconds of sending the instruction, within 500 ms of sending the instruction, within 30 ms of sending the instruction, etc.), substantially concurrent with determining that the robot is operating in the state or has entered the instructed state, after some preset time delay, in response to receiving a request to display the current state, or at any other suitable time.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the workflow generation and robotic system as taught by Takeuchi with the ability to provide information relating to the state of current operations during execution of the program as taught by Gupta. This would enable a user to efficiently follow the operation program and debug the program as needed.
Regarding claim 3, where all the limitations of claim 1 are discussed above, Takeuchi further teaches:
3. (Previously Presented) The teaching device according to claim 1, wherein the processor (Paragraph 0057, “The robot control device 200 controls the arm 110, the end effector 140, the transport device 50, and the camera 30. The functions of the robot control device 200 are realized, for example, by a computer with a processor and a memory executing a computer program.”) is further configured to cause the display to displays, during selection of a command disposed in the program creation screen, … (Paragraph 0066, "Parameter setting area PR is an area for setting work parameters related to the entire work or operation parameters related to individual operations." and Paragraph 0081, " In FIG. 6D, when one of the blocks SB1 and OB1 to OB4 placed in the operation flow creation area FL is selected, the parameters corresponding to the selected block are displayed in the parameter setting area PR. For example, when the sequence block SB1 is selected, the work parameters related to the entire sequence are displayed. When one the objects OB1 to OB4 of the object is selected, the parameters related to the objects are displayed. In the example of FIG. 6D, the parameters related to the conditional branch block OB2 are displayed. These parameters are changed as necessary." Please also see Figures 6E and 7 which demonstrate that different operation objects may be selected to add to the program and information on these objects is displayed during selection.)
While Takeuchi does teach displaying an execution information area. They do not explicitly state that this is the execution state information. However, Gupta, in the same field of endeavor of robotics, teaches:
… the information. (Paragraph 0064, "The data is preferably determined (e.g., received) by the user device, but can alternatively be determined by the robot, a remote computing system, or by any other suitable system. The data is preferably measured by the robot sensors and sent by the robot to the user device, but can alternatively be measured by the user device (e.g., by the camera or microphone of the user device), be measured by a second user device, be measured by a second robot, be measured by an independent sensor or sensors (e.g., camera, microphone, proximity detector, trilateration system), or be measured by any other suitable input. All data measured by the robot is preferably sent to the user device (e.g., streamed to the user device, sent at a predetermined frequency, etc.), wherein the user device preferably processes the data and determines whether an event was satisfied, wherein the event is associated with the transition connected to the previous state for which machine code was sent to the robot. Examples of data indicative of robot inputs include robot acceleration (e.g., rolling, dropping, etc.), robot button presses, sounds, the identifiers of sound sensors detecting sound, light intensity, the identifiers of light sensors detecting light, component motor feedback, visual system measurements, or any other suitable data indicative of an event." as well as Paragraph 0068, " Executing the program S300 can include tracking the current state of program execution (e.g., as related to the state machine), preferably at the user device, but alternately at the robot, at another robot or user device, or at any suitable device. Tracking the current state can include tracking the instructed state (e.g., a first state, wherein the instruction sent in the most recent performance of Block S310 is an instruction to operate in the first state), tracking the state in which the robot is operating (e.g., based on confirmation received from the robot, determined from data indicative of robot operation), or tracking any other state. An indication of the current state (e.g., indication that the robot has entered the current state) can be displayed during program execution, not be displayed at all, or otherwise presented. The indication is preferably displayed on the graphical representation of the state machine (e.g., in the graphical representation, highlighting the state icon associated with the current state), but alternately can be displayed as a graphical representation of the state independent of any graphical representation of the state machine, can be displayed as text, or can be otherwise suitably displayed. The indication can be displayed beginning substantially concurrent with sending the instruction to operate in a state (e.g., within five seconds of sending the instruction, within 500 ms of sending the instruction, within 30 ms of sending the instruction, etc.), substantially concurrent with determining that the robot is operating in the state or has entered the instructed state, after some preset time delay, in response to receiving a request to display the current state, or at any other suitable time.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the workflow generation and robotic system as taught by Takeuchi with the ability to provide information relating to the state of current operations during execution of the program as taught by Gupta. This would enable a user to efficiently follow the operation program and debug the program as needed.
Regarding claim 4, where all the limitations of claim 1 are discussed above, Takeuchi further teaches:
4. (previously presented) The teaching device according to claim 1, wherein the processor (Paragraph 0057, “The robot control device 200 controls the arm 110, the end effector 140, the transport device 50, and the camera 30. The functions of the robot control device 200 are realized, for example, by a computer with a processor and a memory executing a computer program.”) is further configured to cause the display to display, in response to an execution instruction on [[a]] the command disposed in the program creation screen, (Paragraph 0066, "Parameter setting area PR is an area for setting work parameters related to the entire work or operation parameters related to individual operations." and Paragraph 0081, " In FIG. 6D, when one of the blocks SB1 and OB1 to OB4 placed in the operation flow creation area FL is selected, the parameters corresponding to the selected block are displayed in the parameter setting area PR. For example, when the sequence block SB1 is selected, the work parameters related to the entire sequence are displayed. When one the objects OB1 to OB4 of the object is selected, the parameters related to the objects are displayed. In the example of FIG. 6D, the parameters related to the conditional branch block OB2 are displayed. These parameters are changed as necessary." Please also see Figure11A which shows the display and the information shown as a result of the generated program being executed.) …
While Takeuchi does teach displaying an execution information area. They do not explicitly state that this is the execution state information or that it is displayed during the execution of the operation program. However, Gupta, in the same field of endeavor of robotics, teaches:
… the information during the execution(Paragraph 0064, "The data is preferably determined (e.g., received) by the user device, but can alternatively be determined by the robot, a remote computing system, or by any other suitable system. The data is preferably measured by the robot sensors and sent by the robot to the user device, but can alternatively be measured by the user device (e.g., by the camera or microphone of the user device), be measured by a second user device, be measured by a second robot, be measured by an independent sensor or sensors (e.g., camera, microphone, proximity detector, trilateration system), or be measured by any other suitable input. All data measured by the robot is preferably sent to the user device (e.g., streamed to the user device, sent at a predetermined frequency, etc.), wherein the user device preferably processes the data and determines whether an event was satisfied, wherein the event is associated with the transition connected to the previous state for which machine code was sent to the robot. Examples of data indicative of robot inputs include robot acceleration (e.g., rolling, dropping, etc.), robot button presses, sounds, the identifiers of sound sensors detecting sound, light intensity, the identifiers of light sensors detecting light, component motor feedback, visual system measurements, or any other suitable data indicative of an event." as well as Paragraph 0068, " Executing the program S300 can include tracking the current state of program execution (e.g., as related to the state machine), preferably at the user device, but alternately at the robot, at another robot or user device, or at any suitable device. Tracking the current state can include tracking the instructed state (e.g., a first state, wherein the instruction sent in the most recent performance of Block S310 is an instruction to operate in the first state), tracking the state in which the robot is operating (e.g., based on confirmation received from the robot, determined from data indicative of robot operation), or tracking any other state. An indication of the current state (e.g., indication that the robot has entered the current state) can be displayed during program execution, not be displayed at all, or otherwise presented. The indication is preferably displayed on the graphical representation of the state machine (e.g., in the graphical representation, highlighting the state icon associated with the current state), but alternately can be displayed as a graphical representation of the state independent of any graphical representation of the state machine, can be displayed as text, or can be otherwise suitably displayed. The indication can be displayed beginning substantially concurrent with sending the instruction to operate in a state (e.g., within five seconds of sending the instruction, within 500 ms of sending the instruction, within 30 ms of sending the instruction, etc.), substantially concurrent with determining that the robot is operating in the state or has entered the instructed state, after some preset time delay, in response to receiving a request to display the current state, or at any other suitable time.") …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the workflow generation and robotic system as taught by Takeuchi with the ability to provide information relating to the state of current operations during execution of the program as taught by Gupta. This would enable a user to efficiently follow the operation program and debug the program as needed.
Regarding claim 8, where all the limitations of claim 1 are discussed above, Takeuchi further teaches:
8. (Previously Presented) The teaching device according to claim 1, wherein the sensor is a visual sensor. (Paragraph 0160, "FIG. 6E shows an example of a window W1a for creating an operation flow of second type work using the camera 30. The window W1a is similar to the window W1 for the first type work shown in FIG. 6D. However, in the main view area MV, the window W1a is different from the window W1 in that a camera image display area IM for displaying an image captured with the camera 30 is provided in the main view area MV. In the camera image display area IM, it is preferable to be able to designate an image processing area to be subjected to an image processing such as product checking among the images captured with the camera 30. In this way, the load of the image processing can be reduced so that the operation of the robot can be performed at high speed. The content of areas FL, SQ, PR, RS, and RN other than the window W1a for the second type work in the main view area MV are substantially the same as the content of these areas in the window W1 for the first type work. However, content different from that of the window W1 for the first type work may be displayed. The window W1 for the first type work corresponds to a “first type input screen”, and the window W1a for the second type work corresponds to a “second type input screen”." Please see areas "PR" and "RS" in Figure 6E.)
Regarding claim 9, where all the limitations of claim 1 are discussed above, Takeuchi further teaches:
9. (Previously Presented) The teaching device according to claim 1, wherein the sensor is a force sensor. (Paragraph 0054, "The robot 100 can set the end effectors at any positions in any orientations within a movable range of the arm 110. A force detector 130 and an end effector 140 are installed on the arm flange 120. In the present embodiment, the end effector 140 is a gripper, but any other type of end effector can be used. The force detector 130 is a six-axis sensor that measures three-axis force acting on the end effector 140 and torque acting around the three axes. The force detector 130 measures magnitude of force parallel to three measurement axes orthogonal to each other in a sensor coordinate system which is a unique coordinate system, and the magnitude of torque around the three measurement axes. A force sensor as a force detector may be provided at any one or more joints J1 to J5 other than the joint J6. The force detector may only measure the force and torque in a direction of control, and a unit for directly measuring the force and torque like the force detector 130 or a unit for measuring the torque of the joint of the robot to obtain the force and the torque indirectly may be used. The force detector may measure the force and torque only in the direction of controlling force." Please also see Figure 11B)
Regarding claim 13, where all the limitations of claim 1 are discussed above, Takeuchi further teaches:
13. (Previously Presented) The teaching device according to claim 1, wherein each of the plurality of commands is represented by an icon or a statement. (Paragraph 0076, "Main view area MV: a plurality of categories indicating the operations constituting the operation flow and the categories of the conditional branch, name and icon of the object belonging to each of the categories, the description of the contents of the object, and an image showing the outline of the object are displayed. The object displayed on the main view area MV can be arbitrarily added to the operation flow in the operation flow creation area FL by a work such as drag and drop.")
Regarding claim 14, Takeuchi further teaches:
14. (Currently Amended) A teaching device configured to create a control program of a robot comprising:
a processor (Paragraph 0057, “The robot control device 200 controls the arm 110, the end effector 140, the transport device 50, and the camera 30. The functions of the robot control device 200 are realized, for example, by a computer with a processor and a memory executing a computer program.”) configured to:
generate a program creation screen for performing program creation by a plurality of commands each representing a function constituting a control program of the robot (Paragraph 0005, "The robot control device includes a display control unit that displays an input screen including an operation flow creation area for creating an operation flow of work on a display device; a conversion unit that converts the created operation flow into a control program; and a control execution unit that executes the control program to control the robot. The input screen is configured to display a plurality of operation objects indicating a plurality of operations including an operation using force control, and one or more conditional branch objects indicating a conditional branch, as options. The operation flow creation area is configured to create an operation flow including the conditional branch by graphically placing an operation object selected from the plurality of operation objects and the conditional branch object." As well as Paragraph 0082, “FIG. 7 shows an example of operation classifications and operation objects constituting an operation flow, and FIGS. 8A to 8D show outlines of operations of some operation objects. A plurality of operation objects can be categorized into the following four categories. All of these operations involve force control.”) … and
cause a display to display, (Paragraph 0066, "Parameter setting area PR is an area for setting work parameters related to the entire work or operation parameters related to individual operations." and Paragraph 0081, " In FIG. 6D, when one of the blocks SB1 and OB1 to OB4 placed in the operation flow creation area FL is selected, the parameters corresponding to the selected block are displayed in the parameter setting area PR. For example, when the sequence block SB1 is selected, the work parameters related to the entire sequence are displayed. When one the objects OB1 to OB4 of the object is selected, the parameters related to the objects are displayed. In the example of FIG. 6D, the parameters related to the conditional branch block OB2 are displayed. These parameters are changed as necessary." Please also see Figures 6E and 7 which demonstrate that different operation objects may be selected to add to the program and information on these objects is displayed during selection.) when a command, … of the plurality of commands disposed in the program creation screen (Paragraph 0005, "The robot control device includes a display control unit that displays an input screen including an operation flow creation area for creating an operation flow of work on a display device; a conversion unit that converts the created operation flow into a control program; and a control execution unit that executes the control program to control the robot. The input screen is configured to display a plurality of operation objects indicating a plurality of operations including an operation using force control, and one or more conditional branch objects indicating a conditional branch, as options. The operation flow creation area is configured to create an operation flow including the conditional branch by graphically placing an operation object selected from the plurality of operation objects and the conditional branch object." As well as Paragraph 0082, “FIG. 7 shows an example of operation classifications and operation objects constituting an operation flow, and FIGS. 8A to 8D show outlines of operations of some operation objects. A plurality of operation objects can be categorized into the following four categories. All of these operations involve force control.”) is executed by execution of the control program or is selected, (Paragraph 0081, "In FIG. 6D, when one of the blocks SB1 and OB1 to OB4 placed in the operation flow creation area FL is selected, the parameters corresponding to the selected block are displayed in the parameter setting area PR. For example, when the sequence block SB1 is selected, the work parameters related to the entire sequence are displayed. When one the objects OB1 to OB4 of the object is selected, the parameters related to the objects are displayed. In the example of FIG. 6D, the parameters related to the conditional branch block OB2 are displayed. These parameters are changed as necessary." as well as Paragraphs 0157-0158) … and
execute the control program in order to control the robot, (Paragraphs 0157-0158, "FIG. 12 shows an example of a window W3 as an input screen displayed when the operation flow is converted into a control program according to the indication of the teacher. This window W3 includes a sequence selecting area F31 for selecting a sequence (that is an operation flow) to be converted into a control program and a program file name setting area F32 for setting the file name of the control program. The sequence selecting area F31 may be omitted. The window W3 may be configured to include an area for designating information such as a storage destination of parameters. In the window W3, when the “execution” button is pressed, the operation flow is converted into a control program by the conversion unit 244, and the control program is stored in the memory 220.
In step S140 of FIG. 5, the robot control device 200 controls the robot and causes the robot to execute the work according to the control program created in step S130. This work can be executed as a checking work for checking the operations of the robot 100 on the production line or an original work for manufacturing a product on the production line.") …
Takeuchi does not specifically teach displaying information on a control block when that block is executed during execution of the program. However, Gupta, in the same field of endeavor of robotics, teaches:
… setting a parameter value of the plurality of commands; … with a set parameter value, (Paragraphs 0028-0029, "Events can be associated with one or more toy robot input systems (e.g., sensors or input devices), associated with other input systems, associated with time intervals, associated with progress through actions associated with a state, or associated with other suitable conditions. Each event preferably represents a fixed set of input variable values (e.g., a specific input received at a specific location relative to the robot, a specific wavelength of sound, a specific input threshold value being exceeded, a sustained input received for more than a threshold period of time, etc.), but can alternatively represent a variable input (e.g., a range of inputs, etc.), represent an output, represent any other suitable robot parameter, or represent any other suitable parameter. In a specific example, examples of events can include timers, robot tilt (e.g., relative to a gravity vector, relative to a previous robot orientation, etc.), robot acceleration (e.g., along or against a gravity vector, along a robot axis, along a vector at an angle relative to a gravity vector and/or robot axis, etc.), button depression, button release, sound detection, sound direction detection, secondary robot direction detection (e.g., based on the beacon strength of a secondary robot, based on visual system feedback, etc.), secondary robot proximity detection, obstruction detection (e.g., based on motor feedback or visual systems), overcoming the obstruction, robot component actuation (e.g., head turning, tail wagging, wheels turning, etc.), component movement cessation, robot vibration or shaking, or include any other suitable event.
Each icon can illustrate the toy robot in the state (e.g., respective associated state) or undergoing the event (e.g., respective associated event); can include any other suitable illustration or representation representative of the respective state(s) or event(s) associated with the icon (e.g., animation, text, etc.); be otherwise graphically associated with the state(s) or event(s); or otherwise suitably associated with a program element (e.g., state, event, etc.). In some embodiments, a state icon can illustrate a toy robot subsystem operating in a state associated with that subsystem, and an event icon can illustrate a toy robot input system undergoing an event associated with that input system.") … information related to an execution state of the executed or selected command, (Paragraph 0064, "The data is preferably determined (e.g., received) by the user device, but can alternatively be determined by the robot, a remote computing system, or by any other suitable system. The data is preferably measured by the robot sensors and sent by the robot to the user device, but can alternatively be measured by the user device (e.g., by the camera or microphone of the user device), be measured by a second user device, be measured by a second robot, be measured by an independent sensor or sensors (e.g., camera, microphone, proximity detector, trilateration system), or be measured by any other suitable input. All data measured by the robot is preferably sent to the user device (e.g., streamed to the user device, sent at a predetermined frequency, etc.), wherein the user device preferably processes the data and determines whether an event was satisfied, wherein the event is associated with the transition connected to the previous state for which machine code was sent to the robot. Examples of data indicative of robot inputs include robot acceleration (e.g., rolling, dropping, etc.), robot button presses, sounds, the identifiers of sound sensors detecting sound, light intensity, the identifiers of light sensors detecting light, component motor feedback, visual system measurements, or any other suitable data indicative of an event." as well as Paragraph 0068, " Executing the program S300 can include tracking the current state of program execution (e.g., as related to the state machine), preferably at the user device, but alternately at the robot, at another robot or user device, or at any suitable device. Tracking the current state can include tracking the instructed state (e.g., a first state, wherein the instruction sent in the most recent performance of Block S310 is an instruction to operate in the first state), tracking the state in which the robot is operating (e.g., based on confirmation received from the robot, determined from data indicative of robot operation), or tracking any other state. An indication of the current state (e.g., indication that the robot has entered the current state) can be displayed during program execution, not be displayed at all, or otherwise presented. The indication is preferably displayed on the graphical representation of the state machine (e.g., in the graphical representation, highlighting the state icon associated with the current state), but alternately can be displayed as a graphical representation of the state independent of any graphical representation of the state machine, can be displayed as text, or can be otherwise suitably displayed. The indication can be displayed beginning substantially concurrent with sending the instruction to operate in a state (e.g., within five seconds of sending the instruction, within 500 ms of sending the instruction, within 30 ms of sending the instruction, etc.), substantially concurrent with determining that the robot is operating in the state or has entered the instructed state, after some preset time delay, in response to receiving a request to display the current state, or at any other suitable time.") the information being displayed during execution of the executed or selected command; (Paragraph 0068, “Executing the program S300 can include tracking the current state of program execution (e.g., as related to the state machine), preferably at the user device, but alternately at the robot, at another robot or user device, or at any suitable device. Tracking the current state can include tracking the instructed state (e.g., a first state, wherein the instruction sent in the most recent performance of Block S310 is an instruction to operate in the first state), tracking the state in which the robot is operating (e.g., based on confirmation received from the robot, determined from data indicative of robot operation), or tracking any other state. An indication of the current state (e.g., indication that the robot has entered the current state) can be displayed during program execution, not be displayed at all, or otherwise presented. The indication is preferably displayed on the graphical representation of the state machine (e.g., in the graphical representation, highlighting the state icon associated with the current state), but alternately can be displayed as a graphical representation of the state independent of any graphical representation of the state machine, can be displayed as text, or can be otherwise suitably displayed. The indication can be displayed beginning substantially concurrent with sending the instruction to operate in a state (e.g., within five seconds of sending the instruction, within 500 ms of sending the instruction, within 30 ms of sending the instruction, etc.), substantially concurrent with determining that the robot is operating in the state or has entered the instructed state, after some preset time delay, in response to receiving a request to display the current state, or at any other suitable time.”) … wherein the information related to the execution state includes at least one of a state, during the execution of the command, of a sensor or equipment associated with the command, information related to a detection value, during the execution of the command, of the sensor associated with the command, (Paragraph 0062, "Sending sensor data at the robot S330 functions to communicate data potentially indicative of the occurrence of an event. The sensor data is preferably robot sensor data, but can alternatively be user device data, or be any other suitable data. The sensor data can be sampled by the robot (e.g., by the robot sensors) and/or recorded by the robot (e.g., by the robot processor and/or memory), or otherwise determined. The sensor data can include sensor measurements, sensor signals, data synthesized from one or more sensor measurements (e.g., ratios between different types of sensor measurements, measurement rate of change, etc.), or any other suitable sensor data. The sensor data can be sent by the robot continuously, or sent in batches (e.g., sent at a predetermined frequency, sent in response to receiving a request to send one or more batches), or sent in any other suitable manner. The robot preferably samples, records, and/or sends the sensor data during instruction execution (e.g., during robot operation in the first state), but can alternatively obtain and/or process sensor data in any other suitable manner at any other suitable time. The robot can begin sampling, recording, and/or sending the sensor data substantially concurrently with a communication or execution incident (e.g., connecting to the user device, receiving an indication that execution will begin, receiving an instruction, operating according to an instruction S320, completing operation according to an instruction, receiving a request for sensor data, etc.), after a time interval (e.g., a predetermined time interval, a time interval specified by an instruction) following a communication or execution incident, in response to entering a state (e.g., an “on” state), or at any other suitable time. The robot can stop sampling, recording, and/or sending the sensor data substantially concurrently with a communication or execution incident (e.g. disconnecting from the user device, receiving an indication that execution will end, receiving an instruction, operating according to an instruction S320, completing operation according to an instruction, receiving a request to stop sending sensor data, losing connection to the user device, etc.), after a time interval following a communication or execution incident, in response to entering an off state, or at any other suitable time. The robot can continue sampling, recording, and/or sending the sensor data (e.g., continuously, sporadically, at a predetermined frequency, etc.) during the time interval between beginning sampling, recording, and/or sending the sensor data and ending sampling, recording, and/or sending the sensor data.") or an internal variable, during the execution of the command, of the control program related to the command. (Paragraphs 0028-0029, "Events can be associated with one or more toy robot input systems (e.g., sensors or input devices), associated with other input systems, associated with time intervals, associated with progress through actions associated with a state, or associated with other suitable conditions. Each event preferably represents a fixed set of input variable values (e.g., a specific input received at a specific location relative to the robot, a specific wavelength of sound, a specific input threshold value being exceeded, a sustained input received for more than a threshold period of time, etc.), but can alternatively represent a variable input (e.g., a range of inputs, etc.), represent an output, represent any other suitable robot parameter, or represent any other suitable parameter. In a specific example, examples of events can include timers, robot tilt (e.g., relative to a gravity vector, relative to a previous robot orientation, etc.), robot acceleration (e.g., along or against a gravity vector, along a robot axis, along a vector at an angle relative to a gravity vector and/or robot axis, etc.), button depression, button release, sound detection, sound direction detection, secondary robot direction detection (e.g., based on the beacon strength of a secondary robot, based on visual system feedback, etc.), secondary robot proximity detection, obstruction detection (e.g., based on motor feedback or visual systems), overcoming the obstruction, robot component actuation (e.g., head turning, tail wagging, wheels turning, etc.), component movement cessation, robot vibration or shaking, or include any other suitable event.
Each icon can illustrate the toy robot in the state (e.g., respective associated state) or undergoing the event (e.g., respective associated event); can include any other suitable illustration or representation representative of the respective state(s) or event(s) associated with the icon (e.g., animation, text, etc.); be otherwise graphically associated with the state(s) or event(s); or otherwise suitably associated with a program element (e.g., state, event, etc.). In some embodiments, a state icon can illustrate a toy robot subsystem operating in a state associated with that subsystem, and an event icon can illustrate a toy robot input system undergoing an event associated with that input system.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the workflow generation and robotic system as taught by Takeuchi with the methods to display current state of program execution during operation of the program as taught by Gupta. This would allow a user to efficiently follow the execution of the program and provide necessary input and/or debug the program as needed.
Regarding claim 15, Takeuchi further teaches:
15. (Currently Amended) A method of creating a control program of a robot, comprising: generating a program creation screen for performing program creation by a plurality of commands each representing a function constituting a control program of the robot (Paragraph 0005, "The robot control device includes a display control unit that displays an input screen including an operation flow creation area for creating an operation flow of work on a display device; a conversion unit that converts the created operation flow into a control program; and a control execution unit that executes the control program to control the robot. The input screen is configured to display a plurality of operation objects indicating a plurality of operations including an operation using force control, and one or more conditional branch objects indicating a conditional branch, as options. The operation flow creation area is configured to create an operation flow including the conditional branch by graphically placing an operation object selected from the plurality of operation objects and the conditional branch object." As well as Paragraph 0082, “FIG. 7 shows an example of operation classifications and operation objects constituting an operation flow, and FIGS. 8A to 8D show outlines of operations of some operation objects. A plurality of operation objects can be categorized into the following four categories. All of these operations involve force control.”) and …
causing a display to display, (Paragraph 0066, "Parameter setting area PR is an area for setting work parameters related to the entire work or operation parameters related to individual operations." and Paragraph 0081, " In FIG. 6D, when one of the blocks SB1 and OB1 to OB4 placed in the operation flow creation area FL is selected, the parameters corresponding to the selected block are displayed in the parameter setting area PR. For example, when the sequence block SB1 is selected, the work parameters related to the entire sequence are displayed. When one the objects OB1 to OB4 of the object is selected, the parameters related to the objects are displayed. In the example of FIG. 6D, the parameters related to the conditional branch block OB2 are displayed. These parameters are changed as necessary." Please also see Figures 6E and 7 which demonstrate that different operation objects may be selected to add to the program and information on these objects is displayed during selection.) when a command, … of the plurality of commands disposed in the program creation screen (Paragraph 0005, "The robot control device includes a display control unit that displays an input screen including an operation flow creation area for creating an operation flow of work on a display device; a conversion unit that converts the created operation flow into a control program; and a control execution unit that executes the control program to control the robot. The input screen is configured to display a plurality of operation objects indicating a plurality of operations including an operation using force control, and one or more conditional branch objects indicating a conditional branch, as options. The operation flow creation area is configured to create an operation flow including the conditional branch by graphically placing an operation object selected from the plurality of operation objects and the conditional branch object." As well as Paragraph 0082, “FIG. 7 shows an example of operation classifications and operation objects constituting an operation flow, and FIGS. 8A to 8D show outlines of operations of some operation objects. A plurality of operation objects can be categorized into the following four categories. All of these operations involve force control.”) is executed by execution of the control program or is selected, (Paragraph 0081, "In FIG. 6D, when one of the blocks SB1 and OB1 to OB4 placed in the operation flow creation area FL is selected, the parameters corresponding to the selected block are displayed in the parameter setting area PR. For example, when the sequence block SB1 is selected, the work parameters related to the entire sequence are displayed. When one the objects OB1 to OB4 of the object is selected, the parameters related to the objects are displayed. In the example of FIG. 6D, the parameters related to the conditional branch block OB2 are displayed. These parameters are changed as necessary." as well as Paragraphs 0157-0158) … and
executing the control program in order to control the robot, (Paragraphs 0157-0158, "FIG. 12 shows an example of a window W3 as an input screen displayed when the operation flow is converted into a control program according to the indication of the teacher. This window W3 includes a sequence selecting area F31 for selecting a sequence (that is an operation flow) to be converted into a control program and a program file name setting area F32 for setting the file name of the control program. The sequence selecting area F31 may be omitted. The window W3 may be configured to include an area for designating information such as a storage destination of parameters. In the window W3, when the “execution” button is pressed, the operation flow is converted into a control program by the conversion unit 244, and the control program is stored in the memory 220.
In step S140 of FIG. 5, the robot control device 200 controls the robot and causes the robot to execute the work according to the control program created in step S130. This work can be executed as a checking work for checking the operations of the robot 100 on the production line or an original work for manufacturing a product on the production line.") wherein …
Takeuchi does not specifically teach displaying information on a control block when that block is executed during execution of the program. However, Gupta, in the same field of endeavor of robotics, teaches:
… setting a parameter value of the plurality of commands; … with a set parameter value, (Paragraphs 0028-0029, "Events can be associated with one or more toy robot input systems (e.g., sensors or input devices), associated with other input systems, associated with time intervals, associated with progress through actions associated with a state, or associated with other suitable conditions. Each event preferably represents a fixed set of input variable values (e.g., a specific input received at a specific location relative to the robot, a specific wavelength of sound, a specific input threshold value being exceeded, a sustained input received for more than a threshold period of time, etc.), but can alternatively represent a variable input (e.g., a range of inputs, etc.), represent an output, represent any other suitable robot parameter, or represent any other suitable parameter. In a specific example, examples of events can include timers, robot tilt (e.g., relative to a gravity vector, relative to a previous robot orientation, etc.), robot acceleration (e.g., along or against a gravity vector, along a robot axis, along a vector at an angle relative to a gravity vector and/or robot axis, etc.), button depression, button release, sound detection, sound direction detection, secondary robot direction detection (e.g., based on the beacon strength of a secondary robot, based on visual system feedback, etc.), secondary robot proximity detection, obstruction detection (e.g., based on motor feedback or visual systems), overcoming the obstruction, robot component actuation (e.g., head turning, tail wagging, wheels turning, etc.), component movement cessation, robot vibration or shaking, or include any other suitable event.
Each icon can illustrate the toy robot in the state (e.g., respective associated state) or undergoing the event (e.g., respective associated event); can include any other suitable illustration or representation representative of the respective state(s) or event(s) associated with the icon (e.g., animation, text, etc.); be otherwise graphically associated with the state(s) or event(s); or otherwise suitably associated with a program element (e.g., state, event, etc.). In some embodiments, a state icon can illustrate a toy robot subsystem operating in a state associated with that subsystem, and an event icon can illustrate a toy robot input system undergoing an event associated with that input system.") … information related to an execution state of the executed or selected command, (Paragraph 0064, "The data is preferably determined (e.g., received) by the user device, but can alternatively be determined by the robot, a remote computing system, or by any other suitable system. The data is preferably measured by the robot sensors and sent by the robot to the user device, but can alternatively be measured by the user device (e.g., by the camera or microphone of the user device), be measured by a second user device, be measured by a second robot, be measured by an independent sensor or sensors (e.g., camera, microphone, proximity detector, trilateration system), or be measured by any other suitable input. All data measured by the robot is preferably sent to the user device (e.g., streamed to the user device, sent at a predetermined frequency, etc.), wherein the user device preferably processes the data and determines whether an event was satisfied, wherein the event is associated with the transition connected to the previous state for which machine code was sent to the robot. Examples of data indicative of robot inputs include robot acceleration (e.g., rolling, dropping, etc.), robot button presses, sounds, the identifiers of sound sensors detecting sound, light intensity, the identifiers of light sensors detecting light, component motor feedback, visual system measurements, or any other suitable data indicative of an event." as well as Paragraph 0068, " Executing the program S300 can include tracking the current state of program execution (e.g., as related to the state machine), preferably at the user device, but alternately at the robot, at another robot or user device, or at any suitable device. Tracking the current state can include tracking the instructed state (e.g., a first state, wherein the instruction sent in the most recent performance of Block S310 is an instruction to operate in the first state), tracking the state in which the robot is operating (e.g., based on confirmation received from the robot, determined from data indicative of robot operation), or tracking any other state. An indication of the current state (e.g., indication that the robot has entered the current state) can be displayed during program execution, not be displayed at all, or otherwise presented. The indication is preferably displayed on the graphical representation of the state machine (e.g., in the graphical representation, highlighting the state icon associated with the current state), but alternately can be displayed as a graphical representation of the state independent of any graphical representation of the state machine, can be displayed as text, or can be otherwise suitably displayed. The indication can be displayed beginning substantially concurrent with sending the instruction to operate in a state (e.g., within five seconds of sending the instruction, within 500 ms of sending the instruction, within 30 ms of sending the instruction, etc.), substantially concurrent with determining that the robot is operating in the state or has entered the instructed state, after some preset time delay, in response to receiving a request to display the current state, or at any other suitable time.") the information being displayed during execution of the executed or selected command; (Paragraph 0068, “Executing the program S300 can include tracking the current state of program execution (e.g., as related to the state machine), preferably at the user device, but alternately at the robot, at another robot or user device, or at any suitable device. Tracking the current state can include tracking the instructed state (e.g., a first state, wherein the instruction sent in the most recent performance of Block S310 is an instruction to operate in the first state), tracking the state in which the robot is operating (e.g., based on confirmation received from the robot, determined from data indicative of robot operation), or tracking any other state. An indication of the current state (e.g., indication that the robot has entered the current state) can be displayed during program execution, not be displayed at all, or otherwise presented. The indication is preferably displayed on the graphical representation of the state machine (e.g., in the graphical representation, highlighting the state icon associated with the current state), but alternately can be displayed as a graphical representation of the state independent of any graphical representation of the state machine, can be displayed as text, or can be otherwise suitably displayed. The indication can be displayed beginning substantially concurrent with sending the instruction to operate in a state (e.g., within five seconds of sending the instruction, within 500 ms of sending the instruction, within 30 ms of sending the instruction, etc.), substantially concurrent with determining that the robot is operating in the state or has entered the instructed state, after some preset time delay, in response to receiving a request to display the current state, or at any other suitable time.”) … the information related to the execution state includes at least one of a state, during the execution of the command, of a sensor or equipment associated with the command, information related to a detection value, during the execution of the command, of the sensor associated with the command, (Paragraph 0062, "Sending sensor data at the robot S330 functions to communicate data potentially indicative of the occurrence of an event. The sensor data is preferably robot sensor data, but can alternatively be user device data, or be any other suitable data. The sensor data can be sampled by the robot (e.g., by the robot sensors) and/or recorded by the robot (e.g., by the robot processor and/or memory), or otherwise determined. The sensor data can include sensor measurements, sensor signals, data synthesized from one or more sensor measurements (e.g., ratios between different types of sensor measurements, measurement rate of change, etc.), or any other suitable sensor data. The sensor data can be sent by the robot continuously, or sent in batches (e.g., sent at a predetermined frequency, sent in response to receiving a request to send one or more batches), or sent in any other suitable manner. The robot preferably samples, records, and/or sends the sensor data during instruction execution (e.g., during robot operation in the first state), but can alternatively obtain and/or process sensor data in any other suitable manner at any other suitable time. The robot can begin sampling, recording, and/or sending the sensor data substantially concurrently with a communication or execution incident (e.g., connecting to the user device, receiving an indication that execution will begin, receiving an instruction, operating according to an instruction S320, completing operation according to an instruction, receiving a request for sensor data, etc.), after a time interval (e.g., a predetermined time interval, a time interval specified by an instruction) following a communication or execution incident, in response to entering a state (e.g., an “on” state), or at any other suitable time. The robot can stop sampling, recording, and/or sending the sensor data substantially concurrently with a communication or execution incident (e.g. disconnecting from the user device, receiving an indication that execution will end, receiving an instruction, operating according to an instruction S320, completing operation according to an instruction, receiving a request to stop sending sensor data, losing connection to the user device, etc.), after a time interval following a communication or execution incident, in response to entering an off state, or at any other suitable time. The robot can continue sampling, recording, and/or sending the sensor data (e.g., continuously, sporadically, at a predetermined frequency, etc.) during the time interval between beginning sampling, recording, and/or sending the sensor data and ending sampling, recording, and/or sending the sensor data.") or an internal variable, during the execution of the command, of the control program related to the command. (Paragraphs 0028-0029, "Events can be associated with one or more toy robot input systems (e.g., sensors or input devices), associated with other input systems, associated with time intervals, associated with progress through actions associated with a state, or associated with other suitable conditions. Each event preferably represents a fixed set of input variable values (e.g., a specific input received at a specific location relative to the robot, a specific wavelength of sound, a specific input threshold value being exceeded, a sustained input received for more than a threshold period of time, etc.), but can alternatively represent a variable input (e.g., a range of inputs, etc.), represent an output, represent any other suitable robot parameter, or represent any other suitable parameter. In a specific example, examples of events can include timers, robot tilt (e.g., relative to a gravity vector, relative to a previous robot orientation, etc.), robot acceleration (e.g., along or against a gravity vector, along a robot axis, along a vector at an angle relative to a gravity vector and/or robot axis, etc.), button depression, button release, sound detection, sound direction detection, secondary robot direction detection (e.g., based on the beacon strength of a secondary robot, based on visual system feedback, etc.), secondary robot proximity detection, obstruction detection (e.g., based on motor feedback or visual systems), overcoming the obstruction, robot component actuation (e.g., head turning, tail wagging, wheels turning, etc.), component movement cessation, robot vibration or shaking, or include any other suitable event.
Each icon can illustrate the toy robot in the state (e.g., respective associated state) or undergoing the event (e.g., respective associated event); can include any other suitable illustration or representation representative of the respective state(s) or event(s) associated with the icon (e.g., animation, text, etc.); be otherwise graphically associated with the state(s) or event(s); or otherwise suitably associated with a program element (e.g., state, event, etc.). In some embodiments, a state icon can illustrate a toy robot subsystem operating in a state associated with that subsystem, and an event icon can illustrate a toy robot input system undergoing an event associated with that input system.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the workflow generation and robotic system as taught by Takeuchi with the methods to display current state of program execution during operation of the program as taught by Gupta. This would allow a user to efficiently follow the execution of the program and provide necessary input and/or debug the program as needed.
Regarding claim 16, where all the limitations of claim 1 are discussed above, Takeuchi further teaches:
16. (Currently Amended) The teaching device according to claim 1, wherein … of at least one of a force sensor, a visual sensor, an arc welding, a current or voltage sensor, and an I/O signal. (Paragraph 0148, "In the main view area MV, the temporal change of the force Fx in the X axis direction and the torque Tx around the X axis are displayed among the plurality of force measured by the force detector 130 at the time of executing the operation flow. In the main view area MV, it is possible to select and display any one or more temporal changes of force of the plurality of forces measured by the force detector 130. It is also possible to display the temporal change of the measured position of the TCP and the temporal change of the difference between the target position and the measured position of the TCP on the main view area MV. The period of displaying the result in the main view area MV can be an operation period of any one of the operation objects in the operation flow, or can be the entire period from the start to stop of the execution. For example, when any operation object is selected in the operation flow creation area FL, the execution result of the operation period of the operation object is displayed. When the sequence block SB1 is selected, the result of the entire period from the start to stop of the execution is displayed. The period of displaying result in the main view area MV may be an operation period over a plurality of continuing operation objects. The information of some execution results of the control program is also displayed in the result area RS. For example, for any operation object, it is possible to display the end state of the operation (success or failure), time required for the operation, force at the end of the operation, the position at the end of the operation, and the like in the result area RS. Other types of results than the one shown in FIG. 11A may be displayed in the main view area MV. For example, information related to the robot, such as the speed of the robot and the angle of each joint may be displayed.")
Takeuchi does not specifically teach displaying information on a sensor during execution of the program. However, Gupta, in the same field of endeavor of robotics, teaches:
… the information related to the execution state includes information related to a state, during the execution of the executed command, … (Paragraph 0062, "Sending sensor data at the robot S330 functions to communicate data potentially indicative of the occurrence of an event. The sensor data is preferably robot sensor data, but can alternatively be user device data, or be any other suitable data. The sensor data can be sampled by the robot (e.g., by the robot sensors) and/or recorded by the robot (e.g., by the robot processor and/or memory), or otherwise determined. The sensor data can include sensor measurements, sensor signals, data synthesized from one or more sensor measurements (e.g., ratios between different types of sensor measurements, measurement rate of change, etc.), or any other suitable sensor data. The sensor data can be sent by the robot continuously, or sent in batches (e.g., sent at a predetermined frequency, sent in response to receiving a request to send one or more batches), or sent in any other suitable manner. The robot preferably samples, records, and/or sends the sensor data during instruction execution (e.g., during robot operation in the first state), but can alternatively obtain and/or process sensor data in any other suitable manner at any other suitable time. The robot can begin sampling, recording, and/or sending the sensor data substantially concurrently with a communication or execution incident (e.g., connecting to the user device, receiving an indication that execution will begin, receiving an instruction, operating according to an instruction S320, completing operation according to an instruction, receiving a request for sensor data, etc.), after a time interval (e.g., a predetermined time interval, a time interval specified by an instruction) following a communication or execution incident, in response to entering a state (e.g., an “on” state), or at any other suitable time. The robot can stop sampling, recording, and/or sending the sensor data substantially concurrently with a communication or execution incident (e.g. disconnecting from the user device, receiving an indication that execution will end, receiving an instruction, operating according to an instruction S320, completing operation according to an instruction, receiving a request to stop sending sensor data, losing connection to the user device, etc.), after a time interval following a communication or execution incident, in response to entering an off state, or at any other suitable time. The robot can continue sampling, recording, and/or sending the sensor data (e.g., continuously, sporadically, at a predetermined frequency, etc.) during the time interval between beginning sampling, recording, and/or sending the sensor data and ending sampling, recording, and/or sending the sensor data." As well as Paragraph 0068, “Executing the program S300 can include tracking the current state of program execution (e.g., as related to the state machine), preferably at the user device, but alternately at the robot, at another robot or user device, or at any suitable device. Tracking the current state can include tracking the instructed state (e.g., a first state, wherein the instruction sent in the most recent performance of Block S310 is an instruction to operate in the first state), tracking the state in which the robot is operating (e.g., based on confirmation received from the robot, determined from data indicative of robot operation), or tracking any other state. An indication of the current state (e.g., indication that the robot has entered the current state) can be displayed during program execution, not be displayed at all, or otherwise presented. The indication is preferably displayed on the graphical representation of the state machine (e.g., in the graphical representation, highlighting the state icon associated with the current state), but alternately can be displayed as a graphical representation of the state independent of any graphical representation of the state machine, can be displayed as text, or can be otherwise suitably displayed. The indication can be displayed beginning substantially concurrent with sending the instruction to operate in a state (e.g., within five seconds of sending the instruction, within 500 ms of sending the instruction, within 30 ms of sending the instruction, etc.), substantially concurrent with determining that the robot is operating in the state or has entered the instructed state, after some preset time delay, in response to receiving a request to display the current state, or at any other suitable time.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the workflow generation and robotic system as taught by Takeuchi with the methods to display current state of program execution during operation of the program as taught by Gupta. This would enable a user to efficiently follow the execution of the program and provide necessary input and/or debug the program as needed.
Regarding claim 17, where all the limitations of claim 1 are discussed above, Takeuchi further teaches:
17. (Previously Presented) The teaching device according to claim 1, wherein the program creation screen includes a first region that displays a list of the plurality of commands, (Paragraph 0063-0065, "(1) Main view area MV is an area for displaying options of operation objects and conditional branch objects to be described later, execution results of a control program, and the like.
(2) Operation flow creation area FL is an area for displaying the operation flows in which a plurality of objects are graphically placed in an editable manner. The work represented by the operation flow is also called “sequence”.
(3) Sequence display area SQ is an area for displaying a tree structure of the sequence." As well as Paragraph 0082, “FIG. 7 shows an example of operation classifications and operation objects constituting an operation flow, and FIGS. 8A to 8D show outlines of operations of some operation objects. A plurality of operation objects can be categorized into the following four categories. All of these operations involve force control.”) and
a second region for creating the control program by disposing one or more commands selected from the first region, (Paragraph 0064, "Operation flow creation area FL is an area for displaying the operation flows in which a plurality of objects are graphically placed in an editable manner. The work represented by the operation flow is also called “sequence”.")
wherein the processor (Paragraph 0057, “The robot control device 200 controls the arm 110, the end effector 140, the transport device 50, and the camera 30. The functions of the robot control device 200 are realized, for example, by a computer with a processor and a memory executing a computer program.”) is further configured to cause the display to display the information … the program creation screen (Paragraph 0066, "Parameter setting area PR is an area for setting work parameters related to the entire work or operation parameters related to individual operations." and Paragraph 0081, " In FIG. 6D, when one of the blocks SB1 and OB1 to OB4 placed in the operation flow creation area FL is selected, the parameters corresponding to the selected block are displayed in the parameter setting area PR. For example, when the sequence block SB1 is selected, the work parameters related to the entire sequence are displayed. When one the objects OB1 to OB4 of the object is selected, the parameters related to the objects are displayed. In the example of FIG. 6D, the parameters related to the conditional branch block OB2 are displayed. These parameters are changed as necessary." Please also see Figures 6E and 7 which demonstrate that different operation objects may be selected to add to the program and information on these objects is displayed during selection.) such that … window does not overlap the first region and the second region. (Please see at least Figures 6A and 11A which demonstrate that the plurality of regions are displayed as separate windows none of which overlap.)
Takeuchi does not specifically teach the display being a pop-up display. Or displaying information on a control block when that block is executed during execution of the program. However, Gupta, in the same field of endeavor of robotics, teaches:
… related to the execution state (Paragraph 0062, "Sending sensor data at the robot S330 functions to communicate data potentially indicative of the occurrence of an event. The sensor data is preferably robot sensor data, but can alternatively be user device data, or be any other suitable data. The sensor data can be sampled by the robot (e.g., by the robot sensors) and/or recorded by the robot (e.g., by the robot processor and/or memory), or otherwise determined. The sensor data can include sensor measurements, sensor signals, data synthesized from one or more sensor measurements (e.g., ratios between different types of sensor measurements, measurement rate of change, etc.), or any other suitable sensor data. The sensor data can be sent by the robot continuously, or sent in batches (e.g., sent at a predetermined frequency, sent in response to receiving a request to send one or more batches), or sent in any other suitable manner. The robot preferably samples, records, and/or sends the sensor data during instruction execution (e.g., during robot operation in the first state), but can alternatively obtain and/or process sensor data in any other suitable manner at any other suitable time. The robot can begin sampling, recording, and/or sending the sensor data substantially concurrently with a communication or execution incident (e.g., connecting to the user device, receiving an indication that execution will begin, receiving an instruction, operating according to an instruction S320, completing operation according to an instruction, receiving a request for sensor data, etc.), after a time interval (e.g., a predetermined time interval, a time interval specified by an instruction) following a communication or execution incident, in response to entering a state (e.g., an “on” state), or at any other suitable time. The robot can stop sampling, recording, and/or sending the sensor data substantially concurrently with a communication or execution incident (e.g. disconnecting from the user device, receiving an indication that execution will end, receiving an instruction, operating according to an instruction S320, completing operation according to an instruction, receiving a request to stop sending sensor data, losing connection to the user device, etc.), after a time interval following a communication or execution incident, in response to entering an off state, or at any other suitable time. The robot can continue sampling, recording, and/or sending the sensor data (e.g., continuously, sporadically, at a predetermined frequency, etc.) during the time interval between beginning sampling, recording, and/or sending the sensor data and ending sampling, recording, and/or sending the sensor data." As well as Paragraph 0068, “Executing the program S300 can include tracking the current state of program execution (e.g., as related to the state machine), preferably at the user device, but alternately at the robot, at another robot or user device, or at any suitable device. Tracking the current state can include tracking the instructed state (e.g., a first state, wherein the instruction sent in the most recent performance of Block S310 is an instruction to operate in the first state), tracking the state in which the robot is operating (e.g., based on confirmation received from the robot, determined from data indicative of robot operation), or tracking any other state. An indication of the current state (e.g., indication that the robot has entered the current state) can be displayed during program execution, not be displayed at all, or otherwise presented. The indication is preferably displayed on the graphical representation of the state machine (e.g., in the graphical representation, highlighting the state icon associated with the current state), but alternately can be displayed as a graphical representation of the state independent of any graphical representation of the state machine, can be displayed as text, or can be otherwise suitably displayed. The indication can be displayed beginning substantially concurrent with sending the instruction to operate in a state (e.g., within five seconds of sending the instruction, within 500 ms of sending the instruction, within 30 ms of sending the instruction, etc.), substantially concurrent with determining that the robot is operating in the state or has entered the instructed state, after some preset time delay, in response to receiving a request to display the current state, or at any other suitable time.”) …
However, Jin, in the same field of endeavor of robotics, teaches:
… in a pop-up window that is displayed as a separate window from … the pop-up … (Paragraph 0017, "As another example, some embodiments include an apparatus (10) for automatic programming, comprising: a user interaction module (201), configured to receive a user's request on creating a first global parameter of a specific type by detecting the user's dragging a control corresponding to the specific type from a first user interface (701) and dropping the control on a second user interface (702); a displaying module (202), configured to display a first popup window (804) for editing the first global parameter on the second user interface (702); a processing module (203), configured to receive the user's editing operations on the first global parameter in the first popup window (804); and create the first global parameter according to the user's editing operations on the first global parameter; and the displaying module (202), further configured to display a new added first card (802) of the first global parameter in a list (802′) of global parameters on a third user interface (703).")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the workflow generation and robotic system as taught by Takeuchi with the methods to display current state of program execution during operation of the program as taught by Gupta and with ability to use a pop up display as taught by Jin. Incorporating the “pop-up” style window which may be triggered by a selection or execution into the interface as taught by Takeuchi where all windows are displayed in a non-overlapping manner would be obvious. This would allow for new windows to be triggered by different actions while maintaining the visibility of the previous windows which a user may still be interacting with. This would enable a user to continue their work without disruption while accessing focused information within the pop up window and would further allow a user to efficiently follow the execution of the operation program and debug the program as needed.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Takeuchi in view of Gupta and Jin and in further view of Inaba et al. (US 20210060772 A1), hereinafter Inaba.
Regarding claim 11, where all the limitations of claim 1 are discussed above, Takeuchi further teaches:
11. (Previously Presented) The teaching device according to claim 1, further comprising wherein the processor (Paragraph 0057, “The robot control device 200 controls the arm 110, the end effector 140, the transport device 50, and the camera 30. The functions of the robot control device 200 are realized, for example, by a computer with a processor and a memory executing a computer program.”) is further configured to … when the information related to the command is displayed in response to an execution instruction on the command. (Paragraph 0066, "Parameter setting area PR is an area for setting work parameters related to the entire work or operation parameters related to individual operations." and Paragraph 0081, " In FIG. 6D, when one of the blocks SB1 and OB1 to OB4 placed in the operation flow creation area FL is selected, the parameters corresponding to the selected block are displayed in the parameter setting area PR. For example, when the sequence block SB1 is selected, the work parameters related to the entire sequence are displayed. When one the objects OB1 to OB4 of the object is selected, the parameters related to the objects are displayed. In the example of FIG. 6D, the parameters related to the conditional branch block OB2 are displayed. These parameters are changed as necessary." Please also see Figure11A which shows the display and the information shown as a result of the generated program being executed.)
Takeuchi does not specifically teach defining the display content to be displayed. However, Inaba, in the same field of endeavor of robotic programming, teaches:
… set, for a command among the plurality of commands, display attribute information that defines a display content to be displayed as the information related to the command, and determine a display content according to the display attribute information set for the command … (Paragraph 0099, "In the robot programming device 100 described in (1) or (2), the display control unit 102 may accept an instruction to display setting content relating to any operation unit block, and display setting content relating to the operation unit block selected by the instruction in the advanced setting region 320.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic programming interface as taught by Takeuchi with the ability to instruct the system to display different content as taught by Inaba. This would allow a user to personalize the display for either a specific user or for a specific operation so as to provide more relevant data and allow for a more efficient workflow.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Takeuchi in view of Gupta, Jin and Inaba and in further view of Ogawa et al. (WO 2017149667 A1), hereinafter Ogawa.
Regarding claim 12, where all the limitations of claim 11 are discussed above, Takeuchi further teaches:
12. (Previously Presented) The teaching device according to claim 11, wherein the processor (Paragraph 0057, “The robot control device 200 controls the arm 110, the end effector 140, the transport device 50, and the camera 30. The functions of the robot control device 200 are realized, for example, by a computer with a processor and a memory executing a computer program.”) is further configured to …
determine a display content, … when the information related to the command is displayed in response to an execution instruction on the command. (Paragraph 0066, "Parameter setting area PR is an area for setting work parameters related to the entire work or operation parameters related to individual operations." and Paragraph 0081, " In FIG. 6D, when one of the blocks SB1 and OB1 to OB4 placed in the operation flow creation area FL is selected, the parameters corresponding to the selected block are displayed in the parameter setting area PR. For example, when the sequence block SB1 is selected, the work parameters related to the entire sequence are displayed. When one the objects OB1 to OB4 of the object is selected, the parameters related to the objects are displayed. In the example of FIG. 6D, the parameters related to the conditional branch block OB2 are displayed. These parameters are changed as necessary." Please also see Figure11A which shows the display and the information shown as a result of the generated program being executed.)
Takeuchi does not specifically teach displaying the progress status of the system on the display. However, Ogawa, in the same field of endeavor of interfaces for robotic systems, teaches:
… set the execution state attribute being information related to an execution state of the command, and … based on the execution state attribute … (Page 4, Paragraphs 2-4, "In this machine tool management apparatus, as shown in FIGS. 4, 5, and 6, a status bar 96 including a plurality of status display icons 90, 92, and 94 is displayed on any of the processing screens. The status bar 96 is displayed in a state extending in the left-right direction at the upper end on the processing screen, and a plurality of status display icons 90, 92, and 94 are arranged side by side. The plurality of status display icons 90, 92, and 94 indicate the state of the lathe body 40 that is a machine tool for a certain matter, and are roughly classified into three types.
First, the first type of status display icon 90 indicates the status of the lathe body 40 for matters that can be changed by the operator, and corresponds to the four status display icons 90a, 90b, 90c, and 90d from the left. . These four status display icons 90a, 90b, 90c, and 90d respectively indicate the part number of the selected workpiece, the part that indicates the currently executed program number, and the turret that is indexed for the selected tool. This indicates the number and the control mode of the NC control currently selected.
In this machine tool management apparatus, when each of these status display icons 90a, 90b, 90c, and 90d is touch-operated by an operator, a dialog box (a kind of window) for changing the items indicated by the respective icons. Is displayed on the currently displayed processing screen. FIG. 7 is a display screen of the display 30 when the state display icon 90d is touch-operated by the operator on the processing screen 50 shown in FIG. The status display icon 90d is an NC mode icon indicating the currently selected NC machining control mode. In this machine tool management apparatus, there are seven control modes for NC machining control for machining a workpiece, and the operating state of the lathe body 40 for machining the workpiece is selected. The operating state is set in accordance with a control mode selected alternatively from the seven control modes. As shown in FIG. 7, a dialog box 100 is displayed on the processing screen 50 to allow the operator to select an NC machining control mode from seven modes. In the dialog box 100, as shown in FIG. 7, seven icons 102 corresponding to seven control modes are arranged, and when one of these seven icons 102 is selected by the operator, the lathe body The 40 operating states are set as operating states corresponding to the control mode corresponding to the selected icon 102. When any of the seven icons 102 is selected, the NC mode icon 90d displayed on the processing screen is changed to an illustration icon representing the selected control mode.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system interface as taught by Takeuchi with the ability to display the progress state of the system as taught by Ogawa. This would allow the user to closely monitor an operation so that they may manage the system status and interact as necessary.
Conclusion
The Examiner has cited particular paragraphs or columns and line numbers in the referencesapplied to the claims above for the convenience of the Applicant. Although the specified citations arerepresentative of the teachings of the art and are applied to specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested of the Applicant in preparing responses, to fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner. See MPEP 2141.02 [R-07.2015] VI. A prior art reference must be considered in its entirety, i.e., as a whole, including portions that would lead away from the claimed Invention. W.L. Gore & Associates, Inc. v. Garlock, Inc., 721 F.2d 1540, 220 USPQ 303 (Fed. Cir. 1983), cert, denied, 469 U.S. 851 (1984). See also MPEP §2123.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HEATHER KENIRY whose telephone number is (571)270-5468. The examiner can normally be reached M-F 7:30-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Mott can be reached at (571) 270-5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.J.K./Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights