DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/09/2026 has been entered.
Response to Arguments
Applicant's arguments, filed 03/09/2026, regarding the rejection of claims 1-11 under 35 USC 103, have been fully considered but they are not persuasive. Applicant’s arguments generally assert that the Oumi, Kuwahara, and Madvil do not teach the newly amended features. Applicant's arguments do not comply with 37 CFR 1.111(c) because they do not clearly point out the patentable novelty which he or she thinks the claims present in view of the state of the art disclosed by the references cited or the objections made. Further, they do not show how the amendments avoid such references or objections. Therefore, Applicant’s arguments will not be addressed any further.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-3, 5-8, 10-11, & 13-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Oumi (US 20180354124 A1) in view of Kuwahara et al. (US 20180236657 A1, hereinafter Kuwahara), and further in view of Madvil et al. (US 20150277398 A1, hereinafter Madvil).
Regarding claim 1, Oumi discloses:
A control device comprising:
at least one memory configured to store instructions; and
at least one processor configured to execute the instructions (at least as in paragraph 0017, wherein “The robot controller 3 is composed of an arithmetic processing unit (calculator) including a CPU (Central Processing Unit) that acts as a processor, and a RAM (Random Access Memory) and a ROM (Read-Only Memory) that are connected to the CPU via a bus. An operation program for performing the operation of the robot 1 is inputted to the robot controller 3. The robot controller 3 includes a storage unit 3b that stores the operation program or the like”) to:
determine a first operation plan of a robot which executes a task in which an object is used (at least as in paragraph 0024, wherein “The motion image generating unit 11 displays an image 34 of the vehicle body 8. At this time, the image of the robot 1 is not displayed. The operator specifies a position 71 of the closed door 9 and a position 72 of the opened door 9. The position 71 is the position in which the motion of the workpiece is started. The position 72 is the position in which the motion of the workpiece is ended”);
display trajectory information regarding a trajectory of the object moved by the robot based on the first operation plan, (at least as in paragraph 0025, wherein “in step 81, the motion image generating unit 11 of the teaching device 10 generates a three-dimensional motion image in which the workpiece moves. The motion image generating unit 11 generates a motion image (animation) in which the door 9 of the vehicle body 8 is moved based on the three-dimensional model 21 of the vehicle body 8);
But Oumi does not explicitly teach “wherein the trajectory information indicates virtual objects which represent positions and postures of the objects at predetermined intervals during movement of the object;
display, as the trajectory information, a trajectory of the robot together with the trajectory of the object, such that the trajectory of the robot is associated with the virtual objects representing the positions and postures of the object at the predetermined intervals;
correct the trajectory information by receiving a correction of a position and a posture of a virtual object selected from the virtual objects based on an external input, wherein the trajectory information reflects the correction of the virtual object while maintaining an association between the trajectory of the robot and the virtual objects;
determine a second operation plan of the robot based on the correction; and
control the robot based on the second operation plan.”
However, Kuwahara, in the same field of endeavor of a robot control system utilizing a simulator wherein the joba job is first specified and a path is generated according to the conditions associated with the specified jobs, specifically teaches “display, as the trajectory information, a trajectory of the robot (at least as in paragraph 0114, “In step S22, the start-end position setting module 132 obtains the information related to the above-described specified job from the job storage module 112, and determines the start position and the end position of the section as a path generation target based on the obtained information”; at least as in paragraph 0114, “In step S23, the path generation module 133 obtains the information related to the start position and the end position from the start-end position setting module 132, and obtains model information from the model storage module 111”; at least as in paragraph 0120-0123 & Fig. 11, wherein the monitor displays the initial path generated according to the specified job in steps S28-S29”)…
correct the trajectory information by receiving a correction of a position and a posture … based on an external input, wherein the trajectory information reflects the correction (at least as in paragraph 0127, wherein “In step S31, the correction input acquisition module 123 confirms the presence or absence of an input of selecting an index of any of the via-points”; at least as in paragraph 0130, wherein “step S33, the correction input acquisition module 123 outputs a command of adding an edit mark EM (refer to FIG. 13) to the selected index to the image data output module 118, and the image data output module 118 outputs the image data corresponding to the command to the monitor 192… The active index can be moved by operation input (for example, input by dragging) by the input apparatus 191”; at least as in paragraph 0132, wherein “In a case where it is determined in step S34 that the operation input of moving the active index has been started… the information processing module 130 executes processing of correcting the path in accordance with the movement of the active index”; at least as in paragraph 0065, “In generation of a path, the information processing module 130 may be configured to generate the path such that the posture of the distal end portion 16 satisfies the above-described posture condition defined by the sixth input information, and may be configured to generate the path such that the operation angles of the joints J1 to J6 are within movable ranges defined by the seventh input information”; see also [0053] & [0164]; at least as in paragraph 0077, “The via-point correction module 135 changes the position of the via-point in the three-dimensional space in accordance with the position change of the above-described index”)…
determine a second operation plan of the robot based on the correction (at least as in paragraph 0140-0142, wherein “In step S41, the correction input acquisition module 123 obtains input information for moving the active index… In step S42, the correction input acquisition module 123 sets an amount of movement of the active index in accordance with the information obtained in step S41… In step S43, the via-point correction module 135 changes the position of the via-point in accordance with the amount of movement of the active index”; at least as in paragraph 0143-0144, wherein “In step S44, the path generation propriety determination module 136 determines the propriety of the generation of a path that passes through the changed via-point (the via-point whose position has been changed in step S43)”); and
control the robot based on the second operation plan (at least as in paragraph 0144, “In step S45, the corrected path generation module 137 regenerates the path so as to pass through the changed via-point, and applies the path to the above-described modified job stored in the job storage module 112”; at least as in paragraph 0038, “The robot controller 200 controls the robot 10 so as to move the distal end portion 16 along the above-described path (to operate the joints J1 to J6 in accordance with the above-described path)”).”
Madvil, in the same field of endeavor of robot control and simulation methods, specifically teaches “wherein the trajectory information indicates virtual objects which represent positions and postures of the objects at predetermined intervals during movement of the object (at least as in paragraph 0029, “The virtual work objects 210 can be virtual representations of actual work objects that can be worked on by actual robots and are positioned within the virtual workspace 224. Each virtual work object 210 includes or can be associated with one or more work locations 212 that correspond to work locations on an actual work object… Each virtual work object 210 includes or can be associated with a work object position 214 that identifies the position of the virtual work object with respect to the virtual workspace 224”; at least as in paragraph 0030, “The simulation 216 can be an RRS simulation that can simulate movements of the robots along the created paths and/or manipulation of one or more of the virtual work objects 210 via one or more of the virtual robots 202”)…
together with the trajectory of the object, such that the trajectory of the robot is associated with the virtual objects representing the positions and postures of the object at the predetermined intervals (at least as in paragraph 0029, “Each virtual work object 210 has a position that can be independent from its locations, where the position refers to the position of the work object and the location refers to the location on the work object to which the TCPs of the virtual robots can be attached”; at least as in paragraph 0030, “The paths 218 include paths of the virtual robots 202 and paths of the virtual work objects 210”; at least as in paragraph 0031, “The virtual workspace 224 can include the virtual robots 202 and the virtual work objects 210”; at least as in paragraph 0040, “The system can create paths for the virtual robots that move the work object through the workspace, simulate movements of the robots along the paths, and resolve any collisions to form collision-free paths (step 312)”; at least as in paragraph 0040, “The system can create paths for the virtual robots that move the work object through the workspace, simulate movements of the robots along the paths, and resolve any collisions to form collision-free paths (step 312)”)…
of a position and a posture of a virtual object selected from the virtual objects (at least as in paragraph 0030, “A path of a virtual robot can be based on and derived from the path of a virtual work object. The path of a virtual robot can be the path the TCP of the robot follows and can be a list of target locations that the TCPF of the robot should reach. The path of the work object can be the path the work object position follows. The paths of the virtual robots can be determined based on the path of a virtual work object”; at least as in paragraph 0033, “The parameters 226 can be provided by the simulation program in relation to the virtual robots 202. The parameters 226 can include positions, locations, speeds, accelerations, motions, rotations, and so on from which instructions and commands of the programs 222 can be generated to control the actual robots”)…
of the virtual object while maintaining an association between the trajectory of the robot and the virtual objects (at least as in paragraph 0044, “The system determines the paths for each of the virtual robots based on the virtual work object and the virtual workspace (step 404). The path of the virtual work object can be created from the virtual workspace and/or one or more work object positions. The paths for the virtual robots can be created via work locations to which TCPs of the virtual robots can be attached. The work locations can be transformed by the work object positions so that the TCPs of the virtual robots follow the work locations along paths that allow actual robots corresponding to the virtual robots to manipulate an actual work object corresponding to the virtual work object in an actual workspace that corresponds to the virtual workspace"; at least as in paragraph 0045, “The system removes any collisions in the paths for the virtual robots (step 406). Collisions related to a path of a virtual robot that can be detected via the simulation program can be removed by one or more of changing a configuration of a virtual robot, rotating a location of a TCP about a normal axis of the TCP, and adding one or more flyby positions through which the TCP of a virtual robot or the work object passes. All collisions can be removed so that each virtual robot has a collision-free path for manipulating the work object within the workspace”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Oumi, to include Kuwahara's teaching of the correction input acquisition module, via-point correction module, path generation module, and corrected path generation module and Madvil’s teaching of robot simulation systems with virtual robots and virtual work objects, since Kuwahara teaches wherein the robot system generates a path with higher quality leading to higher robot operation efficiency by allowing the user to confirm the relationship between the obstacle and the generated path and Madvil teaches wherein system improves safety by generating programs that can be collision-free.
Regarding claim 2, in view of the above combination of Oumi, Kuwahara, and Madvil, Oumi further discloses:
The control device according to claim 1,
wherein the at least one processor is further configured to execute the instructions to determine the second operation plan obtained by modifying the first operation plan to realize a state of the object specified by the correction (at least as in paragraph 0028, wherein “FIG. 5 shows an enlarged perspective view for explaining the state of the door when the position detecting unit detects the grip position… in step 83, the teaching device 10 starts the setting of a teaching point… The detection of the grip position 42a includes the detection of the orientation of the workpiece… The orientation of the workpiece may be, for example, the direction that relates to the surface of the grip region 41”; at least as in paragraph 0046, wherein “The operator can optionally set the movement path of the box 6 and the orientation of the box 6 on the screen, while the position and orientation of the robot 1 is not taken into account. A method of moving the box 6 can be specified on the screen by the operator. For example, the operator moves the box 6 on the screen with the mouse so as to set the state of the box 6 that moves from a motion start position on the pedestal 7a to a motion end position on the pedestal 7b. Alternatively, the operator may specify the movement path of the box 6 and the orientation of the box 6 according to, for example, the coordinate values of the predetermined coordinate system”).
Regarding claim 3, in view of the above combination of Oumi, Kuwahara, and Madvil, Oumi further discloses:
The control device according to claim 1,
wherein the at least one processor is further configured to execute the instructions to receive the correction relating to at least one of a position and/or posture of the object on the trajectory (at least as in paragraph 0046, wherein “The operator can optionally set the movement path of the box 6 and the orientation of the box 6 on the screen, while the position and orientation of the robot 1 is not taken into account. A method of moving the box 6 can be specified on the screen by the operator. For example, the operator moves the box 6 on the screen with the mouse so as to set the state of the box 6 that moves from a motion start position on the pedestal 7a to a motion end position on the pedestal 7b. Alternatively, the operator may specify the movement path of the box 6 and the orientation of the box 6 according to, for example, the coordinate values of the predetermined coordinate system”).
Regarding claim 5, in view of the above combination of Oumi, Kuwahara, and Madvil, Oumi further discloses:
The control device according to claim 1,
wherein the at least one processor is further configured to execute the instructions to display, as the trajectory information, information relating to a position at which the robot grasps the object, a grasping direction, or a posture of an end effector of the robot (at least as in paragraph 0038, wherein “The motion image generating unit 11 of the teaching device 10 is formed so as to display the motion of the robot 1 as a motion image in addition to the motion of the door 9. In the motion image including the robot 1, the hand 2, and the vehicle body 8, the image 31 in FIG. 2 is displayed so as to continuously change to the image 32 of FIG. 6. Furthermore, the operator can confirm whether or not the motion of the robot 1 and the motion in which the door 9 is opened are normal”), and
wherein the at least one processor is further configured to execute the instructions to receive the correction relating to the position at which the robot grasps the object, the grasping direction, or the posture of the end effector (at least as in paragraph 0026, wherein “in step 82, the operator specifies the grip position of the hand 2 on the door 9. While the door 9 is closed, the operator specifies the grip position 42 in the image 34 by the operation of a mouse. Furthermore, the operator specifies a grip region 41 that is a region for gripping the door 9 by the hand 2. The orientation of the grip region 41 corresponds to the orientation of the hand 2. The grip position 42 can be set in the grip region 41”; at least as in paragraph 0027, wherein “The three-dimensional model 21 may include information on the grip position 42 and information on the grip region 41”; see also Fig. 3, steps 82, 83, and 85).
Regarding claim 6, in view of the above combination of Oumi, Kuwahara, and Madvil, Oumi further discloses:
The control device according to claim 1,
wherein the at least one processor is further configured to execute the instructions to receive the correction indicating addition of an operation of the robot to change a grasping portion of the object (at least as in paragraph 0026, wherein “the operator specifies a grip region 41 that is a region for gripping the door 9 by the hand 2. The orientation of the grip region 41 corresponds to the orientation of the hand 2. The grip position 42 can be set in the grip region 41”; at least as in paragraph 0028, wherein “The orientation of the workpiece may be, for example, the direction that relates to the surface of the grip region 41”; at least as in paragraph 0029, wherein “The teaching-point setting unit. 13 can set the teaching point of the initial state in which the door 9 is closed based on the grip position 42a and the orientation of the grip region 41”), and
wherein the at least one processor is further configured to execute the instructions to determine the second operation plan in which the operation of the robot to change the grasping portion of the object is included (at least as in paragraph 0036, wherein “the teaching points are set at predetermined intervals, but the embodiment is not limited to this. While the motion image generating unit 11 moves the workpiece, the position detecting unit. 12 can detect the grip positions on the workpiece at the predetermined intervals and store the grip positions in the storage unit 17. After the completion of the motion of the workpiece in the motion image, the teaching-point setting unit 13 may set a plurality of the teaching points based on the information on a plurality of the grip positions that are stored in the storage unit 17”; at least as in paragraph 0037, wherein “Thereafter, an operation program creating unit 14 of the teaching device 10 creates the operation program based on the set teaching points”).
Regarding claim 7, the above combination of Oumi, Kuwahara, and Madvil teaches the control device according to claim 1, but does not explicitly teach wherein the at least one processor is further configured to execute the instructions to display, as the trajectory information, a trajectory of the robot together with the trajectory of the object.
However, Kuwahara, in the same field of endeavor of a robot control system utilizing a simulator, specifically teaches “wherein the at least one processor is further configured to execute the instructions to display, as the trajectory information, a trajectory of the robot together with the trajectory of the object” (at least as in paragraph 0130, wherein “In step S33, the correction input acquisition module 123 outputs a command of adding an edit mark EM (refer to FIG. 13) to the selected index to the image data output module 118, and the image data output module 118 outputs the image data corresponding to the command to the monitor 192”; at least as in paragraph 0148, wherein “FIG. 15 is a schematic diagram illustrating an image displayed on the monitor 192 in step S48, illustrating a case where the position of the index P13 in FIG. 13 has been changed and the line RT1 of the path has been changed accordingly”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Oumi, to include Kuwahara's teaching of the correction input acquisition module and image data output module, since Kuwahara teaches wherein the robot system generates a path with higher quality leading to higher robot operation efficiency by allowing the user to confirm the relationship between the obstacle and the generated path.
Regarding claim 8, the above combination of Oumi, Kuwahara, and Madvil teaches the control device according to claim 1, but does not explicitly teach wherein the at least one processor is further configured to further execute the instructions to control the robot based on the second operation plan if the second operation plan satisfies a constraint condition set in the first operation plan.
However, Kuwahara, in the same field of endeavor of a robot control system utilizing a simulator, specifically teaches “wherein the at least one processor is further configured to further execute the instructions to control the robot based on the second operation plan if the second operation plan satisfies a constraint condition set in the first operation plan” (at least as in paragraph 0114, wherein “the path generation module 133 generates the path to satisfy the above-described conditions i) to iii). For example, the path generation module 133 generates a via-point that satisfies the above-described conditions i) to iii), and repeats path correction over the entire path so as to set the path to pass through the via-point until the conditions i) to iii) are satisfied”; at least as in paragraph 0117, wherein “In step S25, the path generation module 133 selects one path from among the above-described plurality of candidate paths in accordance with the above-described evaluation criterion stored in the condition storage module 113, and applies the selected path to the specified job stored in the job storage module 112. That is, the via-point of the relevant path is inserted into the section as the path generation target among the specified jobs. As a result, the above-described initial path in the section is modified to the path that satisfies the above-described conditions i) to iii)”; at least as in paragraph 0143, wherein “In step S44, the path generation propriety determination module 136 determines the propriety of the generation of a path that passes through the changed via-point (the via-point whose position has been changed in step S43). For example, the path generation propriety determination module 136 determines whether the changed via-point satisfies the above-described conditions i) to iii)”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Oumi, to include Kuwahara's teaching of the information acquisition module, path generation module, and condition storage module, since Kuwahara teaches wherein the robot system generates a path with higher quality leading to higher robot operation efficiency by allowing the user to confirm the relationship between the obstacle and the generated path.
Regarding claim 10, Oumi discloses:
A control method executed by a computer, the control method comprising:
determining a first operation plan of a robot which executes a task in which an object is used (at least as in paragraph 0024, wherein “The motion image generating unit 11 displays an image 34 of the vehicle body 8. At this time, the image of the robot 1 is not displayed. The operator specifies a position 71 of the closed door 9 and a position 72 of the opened door 9. The position 71 is the position in which the motion of the workpiece is started. The position 72 is the position in which the motion of the workpiece is ended”);
displaying trajectory information regarding a trajectory of the object moved by the robot based on the first operation plan, (at least as in paragraph 0025, wherein “in step 81, the motion image generating unit 11 of the teaching device 10 generates a three-dimensional motion image in which the workpiece moves. The motion image generating unit 11 generates a motion image (animation) in which the door 9 of the vehicle body 8 is moved based on the three-dimensional model 21 of the vehicle body 8);
But Oumi does not explicitly teach “wherein the trajectory information indicates virtual objects which represent positions and postures of the objects at predetermined intervals during movement of the object;
displaying, as the trajectory information, a trajectory of the robot together with the trajectory of the object, such that the trajectory of the robot is associated with the virtual objects representing the positions and postures of the object at the predetermined intervals;
correcting the trajectory information by receiving a correction of a position and a posture of a virtual object selected from the virtual objects based on an external input, wherein the trajectory information reflects the correction of the virtual object while maintaining an association between the trajectory of the robot and the virtual objects;
determining a second operation plan of the robot based on the correction; and
control the robot based on the second operation plan.”
However, Kuwahara, in the same field of endeavor of a robot control system utilizing a simulator wherein the joba job is first specified and a path is generated according to the conditions associated with the specified jobs, specifically teaches “displaying, as the trajectory information, a trajectory of the robot (at least as in paragraph 0114, “In step S22, the start-end position setting module 132 obtains the information related to the above-described specified job from the job storage module 112, and determines the start position and the end position of the section as a path generation target based on the obtained information”; at least as in paragraph 0114, “In step S23, the path generation module 133 obtains the information related to the start position and the end position from the start-end position setting module 132, and obtains model information from the model storage module 111”; at least as in paragraph 0120-0123 & Fig. 11, wherein the monitor displays the initial path generated according to the specified job in steps S28-S29”)…
correcting the trajectory information by receiving a correction of a position and a posture … based on an external input, wherein the trajectory information reflects the correction (at least as in paragraph 0127, wherein “In step S31, the correction input acquisition module 123 confirms the presence or absence of an input of selecting an index of any of the via-points”; at least as in paragraph 0130, wherein “step S33, the correction input acquisition module 123 outputs a command of adding an edit mark EM (refer to FIG. 13) to the selected index to the image data output module 118, and the image data output module 118 outputs the image data corresponding to the command to the monitor 192… The active index can be moved by operation input (for example, input by dragging) by the input apparatus 191”; at least as in paragraph 0132, wherein “In a case where it is determined in step S34 that the operation input of moving the active index has been started… the information processing module 130 executes processing of correcting the path in accordance with the movement of the active index”; at least as in paragraph 0065, “In generation of a path, the information processing module 130 may be configured to generate the path such that the posture of the distal end portion 16 satisfies the above-described posture condition defined by the sixth input information, and may be configured to generate the path such that the operation angles of the joints J1 to J6 are within movable ranges defined by the seventh input information”; see also [0053] & [0164]; at least as in paragraph 0077, “The via-point correction module 135 changes the position of the via-point in the three-dimensional space in accordance with the position change of the above-described index”)…
determining a second operation plan of the robot based on the correction (at least as in paragraph 0140-0142, wherein “In step S41, the correction input acquisition module 123 obtains input information for moving the active index… In step S42, the correction input acquisition module 123 sets an amount of movement of the active index in accordance with the information obtained in step S41… In step S43, the via-point correction module 135 changes the position of the via-point in accordance with the amount of movement of the active index”; at least as in paragraph 0143-0144, wherein “In step S44, the path generation propriety determination module 136 determines the propriety of the generation of a path that passes through the changed via-point (the via-point whose position has been changed in step S43)”); and
control the robot based on the second operation plan (at least as in paragraph 0144, “In step S45, the corrected path generation module 137 regenerates the path so as to pass through the changed via-point, and applies the path to the above-described modified job stored in the job storage module 112”; at least as in paragraph 0038, “The robot controller 200 controls the robot 10 so as to move the distal end portion 16 along the above-described path (to operate the joints J1 to J6 in accordance with the above-described path)”).”
Madvil, in the same field of endeavor of robot control and simulation methods, specifically teaches “wherein the trajectory information indicates virtual objects which represent positions and postures of the objects at predetermined intervals during movement of the object (at least as in paragraph 0029, “The virtual work objects 210 can be virtual representations of actual work objects that can be worked on by actual robots and are positioned within the virtual workspace 224. Each virtual work object 210 includes or can be associated with one or more work locations 212 that correspond to work locations on an actual work object… Each virtual work object 210 includes or can be associated with a work object position 214 that identifies the position of the virtual work object with respect to the virtual workspace 224”; at least as in paragraph 0030, “The simulation 216 can be an RRS simulation that can simulate movements of the robots along the created paths and/or manipulation of one or more of the virtual work objects 210 via one or more of the virtual robots 202”)…
together with the trajectory of the object, such that the trajectory of the robot is associated with the virtual objects representing the positions and postures of the object at the predetermined intervals (at least as in paragraph 0029, “Each virtual work object 210 has a position that can be independent from its locations, where the position refers to the position of the work object and the location refers to the location on the work object to which the TCPs of the virtual robots can be attached”; at least as in paragraph 0030, “The paths 218 include paths of the virtual robots 202 and paths of the virtual work objects 210”; at least as in paragraph 0031, “The virtual workspace 224 can include the virtual robots 202 and the virtual work objects 210”; at least as in paragraph 0040, “The system can create paths for the virtual robots that move the work object through the workspace, simulate movements of the robots along the paths, and resolve any collisions to form collision-free paths (step 312)”; at least as in paragraph 0040, “The system can create paths for the virtual robots that move the work object through the workspace, simulate movements of the robots along the paths, and resolve any collisions to form collision-free paths (step 312)”)…
of a position and a posture of a virtual object selected from the virtual objects (at least as in paragraph 0030, “A path of a virtual robot can be based on and derived from the path of a virtual work object. The path of a virtual robot can be the path the TCP of the robot follows and can be a list of target locations that the TCPF of the robot should reach. The path of the work object can be the path the work object position follows. The paths of the virtual robots can be determined based on the path of a virtual work object”; at least as in paragraph 0033, “The parameters 226 can be provided by the simulation program in relation to the virtual robots 202. The parameters 226 can include positions, locations, speeds, accelerations, motions, rotations, and so on from which instructions and commands of the programs 222 can be generated to control the actual robots”)…
of the virtual object while maintaining an association between the trajectory of the robot and the virtual objects (at least as in paragraph 0044, “The system determines the paths for each of the virtual robots based on the virtual work object and the virtual workspace (step 404). The path of the virtual work object can be created from the virtual workspace and/or one or more work object positions. The paths for the virtual robots can be created via work locations to which TCPs of the virtual robots can be attached. The work locations can be transformed by the work object positions so that the TCPs of the virtual robots follow the work locations along paths that allow actual robots corresponding to the virtual robots to manipulate an actual work object corresponding to the virtual work object in an actual workspace that corresponds to the virtual workspace"; at least as in paragraph 0045, “The system removes any collisions in the paths for the virtual robots (step 406). Collisions related to a path of a virtual robot that can be detected via the simulation program can be removed by one or more of changing a configuration of a virtual robot, rotating a location of a TCP about a normal axis of the TCP, and adding one or more flyby positions through which the TCP of a virtual robot or the work object passes. All collisions can be removed so that each virtual robot has a collision-free path for manipulating the work object within the workspace”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Oumi, to include Kuwahara's teaching of the correction input acquisition module, via-point correction module, path generation module, and corrected path generation module and Madvil’s teaching of robot simulation systems with virtual robots and virtual work objects, since Kuwahara teaches wherein the robot system generates a path with higher quality leading to higher robot operation efficiency by allowing the user to confirm the relationship between the obstacle and the generated path and Madvil teaches wherein system improves safety by generating programs that can be collision-free.
Regarding claim 11, Oumi discloses:
A non-transitory computer readable storage medium storing a program executed by a computer (at least as in paragraph 0017, wherein “The robot controller 3 includes a storage unit 3b that stores the operation program or the like”), the program causing the computer to:
determine a first operation plan of a robot which executes a task in which an object is used (at least as in paragraph 0024, wherein “The motion image generating unit 11 displays an image 34 of the vehicle body 8. At this time, the image of the robot 1 is not displayed. The operator specifies a position 71 of the closed door 9 and a position 72 of the opened door 9. The position 71 is the position in which the motion of the workpiece is started. The position 72 is the position in which the motion of the workpiece is ended”);
display trajectory information regarding a trajectory of the object moved by the robot based on the first operation plan, (at least as in paragraph 0025, wherein “in step 81, the motion image generating unit 11 of the teaching device 10 generates a three-dimensional motion image in which the workpiece moves. The motion image generating unit 11 generates a motion image (animation) in which the door 9 of the vehicle body 8 is moved based on the three-dimensional model 21 of the vehicle body 8);
But Oumi does not explicitly teach “wherein the trajectory information indicates virtual objects which represent positions and postures of the objects at predetermined intervals during movement of the object;
display, as the trajectory information, a trajectory of the robot together with the trajectory of the object, such that the trajectory of the robot is associated with the virtual objects representing the positions and postures of the object at the predetermined intervals;
correct the trajectory information by receiving a correction of a position and a posture of a virtual object selected from the virtual objects based on an external input, wherein the trajectory information reflects the correction of the virtual object while maintaining an association between the trajectory of the robot and the virtual objects;
determine a second operation plan of the robot based on the correction; and
control the robot based on the second operation plan.”
However, Kuwahara, in the same field of endeavor of a robot control system utilizing a simulator wherein the job is first specified and a path is generated according to the conditions associated with the specified jobs, specifically teaches “display, as the trajectory information, a trajectory of the robot (at least as in paragraph 0114, “In step S22, the start-end position setting module 132 obtains the information related to the above-described specified job from the job storage module 112, and determines the start position and the end position of the section as a path generation target based on the obtained information”; at least as in paragraph 0114, “In step S23, the path generation module 133 obtains the information related to the start position and the end position from the start-end position setting module 132, and obtains model information from the model storage module 111”; at least as in paragraph 0120-0123 & Fig. 11, wherein the monitor displays the initial path generated according to the specified job in steps S28-S29”)…
correct the trajectory information by receiving a correction of a position and a posture … based on an external input, wherein the trajectory information reflects the correction (at least as in paragraph 0127, wherein “In step S31, the correction input acquisition module 123 confirms the presence or absence of an input of selecting an index of any of the via-points”; at least as in paragraph 0130, wherein “step S33, the correction input acquisition module 123 outputs a command of adding an edit mark EM (refer to FIG. 13) to the selected index to the image data output module 118, and the image data output module 118 outputs the image data corresponding to the command to the monitor 192… The active index can be moved by operation input (for example, input by dragging) by the input apparatus 191”; at least as in paragraph 0132, wherein “In a case where it is determined in step S34 that the operation input of moving the active index has been started… the information processing module 130 executes processing of correcting the path in accordance with the movement of the active index”; at least as in paragraph 0065, “In generation of a path, the information processing module 130 may be configured to generate the path such that the posture of the distal end portion 16 satisfies the above-described posture condition defined by the sixth input information, and may be configured to generate the path such that the operation angles of the joints J1 to J6 are within movable ranges defined by the seventh input information”; see also [0053] & [0164]; at least as in paragraph 0077, “The via-point correction module 135 changes the position of the via-point in the three-dimensional space in accordance with the position change of the above-described index”)…
determine a second operation plan of the robot based on the correction (at least as in paragraph 0140-0142, wherein “In step S41, the correction input acquisition module 123 obtains input information for moving the active index… In step S42, the correction input acquisition module 123 sets an amount of movement of the active index in accordance with the information obtained in step S41… In step S43, the via-point correction module 135 changes the position of the via-point in accordance with the amount of movement of the active index”; at least as in paragraph 0143-0144, wherein “In step S44, the path generation propriety determination module 136 determines the propriety of the generation of a path that passes through the changed via-point (the via-point whose position has been changed in step S43)”); and
control the robot based on the second operation plan (at least as in paragraph 0144, “In step S45, the corrected path generation module 137 regenerates the path so as to pass through the changed via-point, and applies the path to the above-described modified job stored in the job storage module 112”; at least as in paragraph 0038, “The robot controller 200 controls the robot 10 so as to move the distal end portion 16 along the above-described path (to operate the joints J1 to J6 in accordance with the above-described path)”).”
Madvil, in the same field of endeavor of robot control and simulation methods, specifically teaches “wherein the trajectory information indicates virtual objects which represent positions and postures of the objects at predetermined intervals during movement of the object (at least as in paragraph 0029, “The virtual work objects 210 can be virtual representations of actual work objects that can be worked on by actual robots and are positioned within the virtual workspace 224. Each virtual work object 210 includes or can be associated with one or more work locations 212 that correspond to work locations on an actual work object… Each virtual work object 210 includes or can be associated with a work object position 214 that identifies the position of the virtual work object with respect to the virtual workspace 224”; at least as in paragraph 0030, “The simulation 216 can be an RRS simulation that can simulate movements of the robots along the created paths and/or manipulation of one or more of the virtual work objects 210 via one or more of the virtual robots 202”)…
together with the trajectory of the object, such that the trajectory of the robot is associated with the virtual objects representing the positions and postures of the object at the predetermined intervals (at least as in paragraph 0029, “Each virtual work object 210 has a position that can be independent from its locations, where the position refers to the position of the work object and the location refers to the location on the work object to which the TCPs of the virtual robots can be attached”; at least as in paragraph 0030, “The paths 218 include paths of the virtual robots 202 and paths of the virtual work objects 210”; at least as in paragraph 0031, “The virtual workspace 224 can include the virtual robots 202 and the virtual work objects 210”; at least as in paragraph 0040, “The system can create paths for the virtual robots that move the work object through the workspace, simulate movements of the robots along the paths, and resolve any collisions to form collision-free paths (step 312)”; at least as in paragraph 0040, “The system can create paths for the virtual robots that move the work object through the workspace, simulate movements of the robots along the paths, and resolve any collisions to form collision-free paths (step 312)”)…
of a position and a posture of a virtual object selected from the virtual objects (at least as in paragraph 0030, “A path of a virtual robot can be based on and derived from the path of a virtual work object. The path of a virtual robot can be the path the TCP of the robot follows and can be a list of target locations that the TCPF of the robot should reach. The path of the work object can be the path the work object position follows. The paths of the virtual robots can be determined based on the path of a virtual work object”; at least as in paragraph 0033, “The parameters 226 can be provided by the simulation program in relation to the virtual robots 202. The parameters 226 can include positions, locations, speeds, accelerations, motions, rotations, and so on from which instructions and commands of the programs 222 can be generated to control the actual robots”)…
of the virtual object while maintaining an association between the trajectory of the robot and the virtual objects (at least as in paragraph 0044, “The system determines the paths for each of the virtual robots based on the virtual work object and the virtual workspace (step 404). The path of the virtual work object can be created from the virtual workspace and/or one or more work object positions. The paths for the virtual robots can be created via work locations to which TCPs of the virtual robots can be attached. The work locations can be transformed by the work object positions so that the TCPs of the virtual robots follow the work locations along paths that allow actual robots corresponding to the virtual robots to manipulate an actual work object corresponding to the virtual work object in an actual workspace that corresponds to the virtual workspace"; at least as in paragraph 0045, “The system removes any collisions in the paths for the virtual robots (step 406). Collisions related to a path of a virtual robot that can be detected via the simulation program can be removed by one or more of changing a configuration of a virtual robot, rotating a location of a TCP about a normal axis of the TCP, and adding one or more flyby positions through which the TCP of a virtual robot or the work object passes. All collisions can be removed so that each virtual robot has a collision-free path for manipulating the work object within the workspace”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Oumi, to include Kuwahara's teaching of the correction input acquisition module, via-point correction module, path generation module, and corrected path generation module and Madvil’s teaching of robot simulation systems with virtual robots and virtual work objects, since Kuwahara teaches wherein the robot system generates a path with higher quality leading to higher robot operation efficiency by allowing the user to confirm the relationship between the obstacle and the generated path and Madvil teaches wherein system improves safety by generating programs that can be collision-free.
Regarding claim 13, the above combination of Oumi, Kuwahara, and Madvil teaches the control device according to claim 1, but does not explicitly teach wherein the at least one processor is further configured to execute the instructions to:
determine whether or not the second operation plan satisfies a constraint condition; and
receive a re-correction relating to the trajectory information upon determining that the second operation plan does not satisfy the constraint condition.
However, Kuwahara, in the same field of endeavor of a robot control system utilizing a simulator, specifically teaches:
determine whether or not the second operation plan satisfies a constraint condition (at least as in paragraph 0115, “the robot simulator 100 executes step S24. In step S24, the path generation module 133 confirms whether the above-described repetition condition stored in the condition storage module 113 is satisfied”); and
receive a re-correction relating to the trajectory information upon determining that the second operation plan does not satisfy the constraint condition (at least as in paragraph 0116, “In a case where it is determined in step S24, that the above-described repetition condition is not satisfied, the robot simulator 100 returns the processing to step S23. Thereafter, generation and writing of the candidate path are repeated until the above-described repetition condition is satisfied, accumulating a plurality of candidate paths in the path storage module 114”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Oumi, to include Kuwahara's teaching of the information acquisition module, path generation module, and condition storage module, since Kuwahara teaches wherein the robot system generates a path with higher quality leading to higher robot operation efficiency by allowing the user to confirm the relationship between the obstacle and the generated path.
Regarding claim 14, in view of the above combination of Oumi, Kuwahara, and Madvil, Oumi further discloses:
The control device according to claim 1, wherein displaying the trajectory information comprises displaying, at a same time, each of the positions and postures of the object at each of the predetermined intervals, ((at least as in paragraph 0025, wherein “in step 81, the motion image generating unit 11 of the teaching device 10 generates a three-dimensional motion image in which the workpiece moves. The motion image generating unit 11 generates a motion image (animation) in which the door 9 of the vehicle body 8 is moved based on the three-dimensional model 21 of the vehicle body 8).
However, Madvil, in the same field of endeavor of robot control and simulation methods, specifically teaches the predetermined intervals are intervals of the first operation plan and the second operation plan (at least as in paragraph 0029, “The virtual work objects 210 can be virtual representations of actual work objects that can be worked on by actual robots and are positioned within the virtual workspace 224. Each virtual work object 210 includes or can be associated with one or more work locations 212 that correspond to work locations on an actual work object… Each virtual work object 210 includes or can be associated with a work object position 214 that identifies the position of the virtual work object with respect to the virtual workspace 224”; at least as in paragraph 0030, “The simulation 216 can be an RRS simulation that can simulate movements of the robots along the created paths and/or manipulation of one or more of the virtual work objects 210 via one or more of the virtual robots 202”; at least as in paragraph 0029, “Each virtual work object 210 has a position that can be independent from its locations, where the position refers to the position of the work object and the location refers to the location on the work object to which the TCPs of the virtual robots can be attached”; at least as in paragraph 0030, “The paths 218 include paths of the virtual robots 202 and paths of the virtual work objects 210”; at least as in paragraph 0031, “The virtual workspace 224 can include the virtual robots 202 and the virtual work objects 210”; at least as in paragraph 0040, “The system can create paths for the virtual robots that move the work object through the workspace, simulate movements of the robots along the paths, and resolve any collisions to form collision-free paths (step 312)”; at least as in paragraph 0040, “The system can create paths for the virtual robots that move the work object through the workspace, simulate movements of the robots along the paths, and resolve any collisions to form collision-free paths (step 312)”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Oumi, to include Madvil’s teaching of robot simulation systems with virtual robots and virtual work objects, since Madvil teaches wherein system improves safety by generating programs that can be collision-free.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Oumi (US 20180354124 A1) in view of Kuwahara et al. (US 20180236657 A1) and Madvil et al. (US 20150277398 A1, hereinafter Madvil), and further in view of Puranic et al. (US 12208521 B1, hereinafter Puranic).
Regarding claim 9, the above combination of Oumi, Kuwahara, and Madvil teaches the control device according to claim 1, but does not explicitly teach wherein the at least one processor is further configured to execute the instructions to:
convert the task to be executed by the robot into a logical formula based on a temporal logic;
generate a time step logical formula that is a logical formula indicating state for each time step to execute the task; and
generate, as an operation plan, a sequence of subtasks to be executed by the robot, each subtask corresponding to a respective time step, based on the time step logical formula.
Puranic discloses robotic systems for learning control policies through the use of temporal logic. Puranic specifically teaches “wherein the at least one processor is further configured to execute the instructions to:
convert the task to be executed by the robot into a logical formula based on a temporal logic (at least as in col. 9, ln. 21-32, wherein “the tasks to be accomplished are described in formal languages such as temporal logics”);
generate a time step logical formula that is a logical formula indicating state for each time step to execute the task (at least as in col. 20, ln. 25-32, wherein “For a policy or demonstration, the basic primitive in STL is a signal predicate u that is a formula of the form f(x(t))>0, where x(t) is the tuple (state, action) of the demonstration x at time t, and f is a function from the signal domain D=(S×A) to custom character. STL formulas are then defined recursively using Boolean combinations of sub-formulas, or by applying an interval-restricted temporal operator to a sub-formula”); and
generate, as an operation plan, a sequence of subtasks to be executed by the robot, each subtask corresponding to a respective time step, based on the time step logical formula (at least as in col. 21, ln. 47-50, wherein “Using the STL representations, complex tasks involving multiple goals, which cannot be easily encoded or represented in traditional IRL may be expressed”).”
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Oumi, to include Puranic’s teaching of utilizing temporal logic based formulas for robot tasks, since Puranic teaches wherein the robot system greatly improves speed and efficiency of learning while enabling generalizability to unknown scenarios.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Oumi (US 20180354124 A1) in view of Kuwahara et al. (US 20180236657 A1) and Madvil et al. (US 20150277398 A1, hereinafter Madvil), and further in view of Shimodaira et al. (US 20180250823 A1, hereinafter Shimodaira).
Regarding claim 12, the above combination of Oumi, Kuwahara, and Madvil teaches the control device according to claim 1 but does not explicitly teach wherein the at least one processor is further configured to execute the instructions to:
calculate a confidence degree regarding accuracy of the position or the posture of the object; and
display the virtual object in an emphasized manner upon determining that the confidence degree is less than a predetermined threshold value.
However, Shimodaira, in the same field of endeavor of robot control through modeling and simulation, specifically teaches:
calculate a confidence degree regarding accuracy of the position or the posture of the object (at least as in paragraph 0292, “a three-dimensional search result is scored on the basis of to what degree corresponding feature points are present in an input image (for example, on the basis of a proportion of the number of feature points corresponding to an error of a predetermined distance or less with respect to a search result, or a value obtained by subtracting an error amount of a feature point as a penalty according to a defined computation formula). In this method, a score lowers in a state in which there is a lot of invalid data (invalid pixel) which cannot be three-dimensionally measured. As mentioned above, a score may be used as an index indicating the reliability of a three-dimensional search result. For example, workpieces are set to be preferentially gripped in an order of higher scores. There is a high probability that a three-dimensional search result of a predetermined score or less is determined as being wrongly detected, and thus a workpiece may be set to be excluded from a grip target”); and
display the virtual object in an emphasized manner upon determining that the confidence degree is less than a predetermined threshold value (at least as in paragraph 0577, “A descending order or an ascending order of a correlation value, or a descending order or an ascending order of a Z direction height may be selected in the label order designation field 468. The correlation value mentioned here is the same as the above-described “score”. A target for determination of whether or not the target is located at a position or an attitude which is different from an expected position or attitude in a specific search result may be designated in the determination label designation field 469”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Oumi, to include Shimodaira’s teaching of determining a three-dimensional search result and displaying the search result score, since Shimodaira teaches wherein the robot setting apparatus with the three-dimensional search improves the accuracy of interference determination and reduces the work burden on the user.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RICARDO ICHIKAWA VISCARRA whose telephone number is (571)270-0154. The examiner can normally be reached M-F 9-12 & 2-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Mott can be reached on (571) 270-5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RICARDO I VISCARRA/Examiner, Art Unit 3657 /KHOI H TRAN/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights