DETAILED ACTION
This is a Final Office Action on the Merits in response to communications filed by applicant on February 13th, 2026. Claims 1-7 are currently pending and examined below
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendments to the claims filed on February 13th, 2026 have been entered. Claims 1, 6, and 7 are currently amended and pending, and claims 2-5 are as previously presented and pending. The amendments to the Abstract filed on February 13th, 2026 have been entered and have overcome each and every objection set forth in the previous Non-Final Office Action mailed November 13th, 2025.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, and 4-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 7181315 B2 ("Watanabe") in view of US 9802317 B2 ("Watts").
Regarding claim 1, Watanabe teaches a designation device comprising (Watanabe: Figure 1 display section 16, Abstract, “A manual-mode operating system for a robot provided with an end-effector. The manual-mode operating system includes a display section including a screen for displaying an image of an end-effector; a first input section for an input of coordinate system data on the screen of the display section, displaying the image of the end-effector, the coordinate system data designating a manual-mode coordinate system used for describing an orientation of the end-effector; a coordinate system setting section for setting the manual mode coordinate system at a spatial position corresponding to an input position on the screen of the display section, based on the coordinate system data input through the first input section; a second input section for an input of orientation data using the manual-mode coordinate system set through the coordinate system setting section, the orientation data instructing the orientation of the end-effector; and an orientation adjusting section for adjusting the orientation of the end-effector, based on the orientation data input through the second input section.”, Column 5 lines 52-67, “The robot controller 30 is further connected with a teaching unit 32 and peripheral equipment 34 through respective interfaces (not shown). As shown in FIG. 4, the teaching unit 32 includes electronic parts incorporated therein, such as a CPU 36, a memory (or a storage section) 38, etc., and is also provided with a display screen 14 and various operation keys 40 arranged on the outer surface of a casing.”. The cited passages clearly teaches a device used to designate inputs for a robot.):
a touch panel configured to receive an input for designating a predetermined surface of an object to be moved (Watanabe: Column 8 lines 1-10, “Alternatively, it is possible to provide the screen 14 with a touch panel function, and thus for the operator to shift the pointer toward the desired point by touching a pen-like touch tool to the desired point. In this manner, the origin coordinate data of the manual-mode coordinate system desired by the user can be input.”, Column 11 line 46 – Column 12 line 8, “Another example of the origin designation of the manual mode coordinate system will be described with reference to FIGS. 12 and 13. In the illustrated example, instead of directly designating the origin of the manual-mode coordinate system on the screen 14, a desired diagram (i.e., an area surrounded by a graphic line) or diagram part (i.e., a graphic line or point) in the displayed image is designated and the origin coordinate is determined in accordance with a predetermined rule. In FIG. 12, a circular arc 44 in the 3D image is illustrated to be designated on the screen 14. The area surrounded by the circular arc 44 in FIG. 12 represents, for example, a hole formed in the workpiece support 42 and, in such a case as to teach the task of inserting a rod-like workpiece (not shown) gripped by the hand into this hole, the circular arc 44 or the surrounded area thereof (i.e., the hole) is picked by the pointer. Then, the CPU 36 of the teaching unit 32 calculates the three-dimensional position of, for example, the center point of the circular arc 44, and identifies the calculated position with the origin position of the manual-mode coordinate system.”, Column 12 lines 35-51, “In the above procedure, the "face" of an object (or an image), surrounded by a closed curve, may be designated, instead of designating the closed curve such as the circular arc. When the "face" is designated, the position of the center of gravity of a contour diagram encircling the "face", for example, is identified with the origin of the manual-mode coordinate system. The determination of the orientation or axis-direction of the coordinate system may be performed in the same manner as in the above-described example of the circular arc.”. The cited passages teach that a user is able to designate either an area surrounded by a graphic line of an object or a face of an object. A person of ordinary skill in the art would recognize that an area surrounded by a graphic line is functionally similar to selecting a face of an object. Additionally, the input device can clearly be a touch panel where a pen-like touch tool to designate the desired points on the object.),
a memory configured to store instructions (Watanabe: Column 5 lines 34-51, “FIG. 3 typically shows a configuration of a manual-mode operating system according to one embodiment of the present invention. This embodiment relates to the manual mode operating system 10 for an actual robot, as shown in FIG. 2A, so that the corresponding components are denoted by common reference numerals and the descriptions thereof are not repeated.”, Column 5 lines 52-67, “The robot controller 30 is further connected with a teaching unit 32 and peripheral equipment 34 through respective interfaces (not shown). As shown in FIG. 4, the teaching unit 32 includes electronic parts incorporated therein, such as a CPU 36, a memory (or a storage section) 38, etc., and is also provided with a display screen 14 and various operation keys 40 arranged on the outer surface of a casing.”);
and a processor configured to execute the instructions to (Watanabe: Column 5 lines 34-51, “FIG. 3 typically shows a configuration of a manual-mode operating system according to one embodiment of the present invention. This embodiment relates to the manual mode operating system 10 for an actual robot, as shown in FIG. 2A, so that the corresponding components are denoted by common reference numerals and the descriptions thereof are not repeated.”, Column 5 lines 52-67, “The robot controller 30 is further connected with a teaching unit 32 and peripheral equipment 34 through respective interfaces (not shown). As shown in FIG. 4, the teaching unit 32 includes electronic parts incorporated therein, such as a CPU 36, a memory (or a storage section) 38, etc., and is also provided with a display screen 14 and various operation keys 40 arranged on the outer surface of a casing.”):
and make a display device display: an image including the object to be moved (Watanabe: Figures 7-13, Column 7 lies 25-37, “Also, as occasion demands, the existing conditions of the robot mechanism 26, the peripheral equipment 34, a work piece, etc. are displayed in the form of 3D graphics together with the hand 12.”, Column 7 lines 44-53, “Also, for example, if it is required that a condition where a workpiece is gripped by the hand 12 is displayed on the screen 14, the shape data of the workpiece in the form of an appendix or inclusion of the shape data of the hand 12 as well as the correlation in position and orientation between the hand 12 and the workpiece in the gripped condition should be previously stored in the memory 38 of the teaching unit 32. The CPU 36 uses these data and executes the operation for 3D-displaying the image of the workpiece gripped by the hand 12 (see an example described later).”, Column 13 lines 35-40, “Next, at step T2, the personal computer 46 creates a signal representing a 3D graphic image including at least the image of the hand 12, based on the shape data, layout data, etc. of the robot mechanism 26, hand 12, peripheral equipment 34, workpiece, etc., previously stored in the memory, as well as on the data received from the robot controller 30 at step Tl.”);
Watanabe does not teach the input comprising vertices of an external form of the object to be moved
and make a display device display: a two-dimensional image including the object to be moved,
and the surface designating the predetermined surface,
wherein the designation device is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal.
Watts, in the same field of endeavor, teaches the input comprising vertices of an external form of the object to be moved (Watts: Column 30 lines 31-48, “FIG. 7 illustrates another example interface of a remote assistor device, in accordance with at least some implementations described herein. In particular, FIG. 7 illustrates a remote assistor device 700 that has received a request the distinguish boxes in region 702 of a model 704 of objects, similar to region 602 of model 604 shown in FIG. 6. As shown, the GUI 706 of the remote assistor device includes two corner virtual boundary lines 708, 710 identified by the control system that correspond to a top right corner of the left box and a bottom left comer of the right box. Similar to virtual boundary line 608 in FIG. 6, a human user may accept, reject, or adjust one or both of the virtual corners 708, 710 so as to enable the control system to instruct the robotic manipulator to manipulate each of the two boxes. However, by identifying and providing comers to the remote assistor device, the control system can enable the human user adjust two perpendicular virtual boundary lines simultaneously.”. The cited passages teach that the system takes, as an input, corners of an object or objects that are used by the system to determine the object to manipulate. The corners can be designated by a user using an input device. One of ordinary skill in the art would recognize that the corners of an object are the vertices of said object.)
and make a display device display: a two-dimensional image including the object to be moved (Watts: Figure 6, Column 28 lines 20-34, “At block 502, in response to receiving the request, the remote assistor device provides, on its display, a graphical user interface (GUI) representative of the model and the virtual boundary line. Generally, the remote assistor device may provide visual data of the objects in the environment, and the model is one such example. Additionally or alternatively to providing a model of the objects, the remote assistor device may provide (after receiving it from the control system) a camera image of the objects in the environment. In other examples, the remote assistor device may provide a synthesized visual representation combining a camera image of the objects and a depth map of the objects. Other visual data associated with the objects in the environment is possible as well to enable the human user to provide useful feedback.”, Column 28 lines 35-51, “The GUI may include one or more of the graphical elements described above. The remote assistor device may provide a GUI including a 2D view of the model and/or a 3D view of the model, each of which may include a depth map of the objects. Within examples, the remote assistor device may provide one or more 2D images of a facade of the objects from respective viewpoints, a static 3D model, and/or a rotatable ( or otherwise adjustable and/or navigable) 3D model, such as a 3D model that a human user can zoom in on or zoom out from and can rotate about one or more axes.”, Column 29 lines 36-67, “The GUI 606 includes an indication of the task for which the control system requested remote assistance ( e.g., "Box Detection"), an indication of a user identifier of the human user operating the remote assistor device 600 e.g., ("REMOTE ASSISTOR #1"), as well as an indication of which robotic device in the workplace will be performing at least a portion of the task (e.g., "ROBOT #1"). In this case, ROBOT #1 may manipulate at least the two boxes at issue in region 602, based on human user feedback. The GUI also includes a visual representation of region 602, box hypotheses that the control system determined for region 602, a virtual boundary line 608 separating two adjacent virtual boxes, and a confidence score associated with the virtual boundary line (e.g., "CONFIDENCE: HIGH").”, The cited passages clearly show that the system is configured to display a 2D image on a display device.),
and the surface designating the predetermined surface (Watts: Column 30 lines 49-64, “FIG. 8 illustrates another example interface of a remote assistor device, in accordance with at least some implementations described herein. In particular, FIG. 8 illustrates a remote assistor device 800 that has received a request to distinguish boxes in region 802 of a model 804 of objects, similar to region 602 of model 604 shown in FIG. 6 and region 702 of model 704 shown in FIG. 7. As shown, the GUI 806 of the remote assistor device includes a virtual boundary line 808 identified by the control system that does not correctly correspond to an actual boundary between the two adjacent boxes. In this scenario, a human user may select the option to "Adjust" the virtual boundary like 808 and then proceed to drag the virtual boundary line to the right until it corresponds to the actual boundary, so as to enable the control system to instruct the robotic manipulator to manipulate each of the two boxes.”. The cited passages teach a method by which a user can adjust the boundaries defining the surfaces of two boxes. These surfaces are shown by a thick black line designating the outer edges of the surfaces of the boxes and a dashed black line representing the virtual boundary (i.e. where the two boxes meet). The system is further configured to allow a user to change this virtual boundary. One of ordinary skill in the art would have recognized that that the virtual boundary and blacklines represent the surface designating a predetermined surface. Therefore, the cited passages clearly teach that the system is configured to display a surface designating a predetermined surface.),
wherein the designation device is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal (Watts: Column 6 lines 34-45, “After receiving the response from the remote assistor device, the control system may control the robotic manipulator to perform the given task in accordance with the response. For instance, if the response modifies the virtual boundary line to correctly correspond to an actual boundary between two objects in the environment, the robotic manipulator may pick up and move each of the two objects.”, Column 17 lines 50-67, Column 24 lines 16-47, “And at block 306, the control system causes the robotic manipulator to perform the task based on the modification to the virtual boundary line. In particular, the control system may cause the robotic manipulator to perform the task by sending instructions to the robotic manipulator that includes information indicative of how to perform the task using the modified virtual boundary line.”, Column 24 line 48 – Column 25 line 3, “Then, the control system may instruct the robotic manipulator to perform the manipulation of each of the two objects based on the confirmed actual boundary between the two objects. In particular, the control system may instruct the robotic manipulator to manipulate one of the objects at a first time without manipulating the other object, and then proceed to manipulate the other object at a second time. Alternatively, the control system may instruct the robotic manipulator to manipulate both of the two objects at the same time (e.g., pick up two objects at once). Within other examples, the task may not necessarily involve a manipulation of both objects on either side of the confirmed actual boundary, as noted above. Accordingly, the control system may instruct the robotic manipulator to perform the manipulation of one of the two objects without performing a manipulation on the other of the two objects.”, Column 29 lines 36-67, “In this case, ROBOT #1 may manipulate at least the two boxes at issue in region 602, based on human user feedback.”, Column 30 lines 49-64, “In this scenario, a human user may select the option to "Adjust" the virtual boundary like 808 and then proceed to drag the virtual boundary line to the right until it corresponds to the actual boundary, so as to enable the control system to instruct the robotic manipulator to manipulate each of the two boxes.”. The cited passages clearly show that the robot is configured to move the objects based on a predetermined algorithm.).
Watanabe teaches a designation device comprising: a touch panel configured to receive an input for designating a predetermined surface of an object to be moved, a memory configured to store instructions; and a processor configured to execute the instructions to: and make a display device display: an image including the object to be moved. Watanabe does not teach the input comprising vertices of an external form of the object to be moved; and make a display device display: a two-dimensional image including the object to be moved; and the surface designating the predetermined surface, wherein the designation device is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal. Watts teaches the input comprising vertices of an external form of the object to be moved; and make a display device display: a two-dimensional image including the object to be moved; and the surface designating the predetermined surface, wherein the designation device is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal. A person of ordinary skill in the art would have had the technological capabilities required to have modified the device taught in Watanabe with the input comprising vertices of an external form of the object to be moved; and make a display device display: a two-dimensional image including the object to be moved; and the surface designating the predetermined surface, wherein the designation device is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal taught in Watts. Furthermore, Watanabe already teaches displaying an image of the object. Modifying Watanabe to display a two-dimensional image of the object instead of a three-dimensional image as taught in Watts is a simple matter of substituting a two-dimensional image for a three-dimensional image. Additionally, Watanabe teaches allowing a user to designate a surface of an object. Modifying Watanabe to display the surface designating the predefined surface as taught in Watts would only require the addition of the display method taught in Watts. While Watanabe does not explicitly teach designating the vertices of the object as the input, Watanabe clearly teaches that the vertices of the object can be used as the points to designate the object to be manipulated (Watanabe: Figure 9 desired point C, Figure 10 desired point D, Column 11, lines 28-55, “In FIG. 9, the image is displayed in the 3D way as viewed from a +Z axis direction. Then, the desired point C is designated on the image shown in FIG. 9. As a result of this, the X and Y coordinates of the position of the origin of the manual-mode coordinate system are determined. Next, the image as displayed is changed to the 3D image as viewed from the +X axis direction, as shown in FIG. 10, and the desired point D is designated on the image in FIG. 10. As a result of this, the Z coordinate of the position of the origin of the manual-mode coordinate system is determined.”). As such, one of ordinary skill in the art would have been easily able to modify Watanabe such that the system specifically uses the vertices as the input as taught in Watts. The device taught in Watanabe is used for teaching a robot a task, so modifying the robot to perform a task according to predetermined algorithm as taught in Watts would be well within the technological capabilities of one of ordinary skill in the art. Such modifications as described would not change or introduce new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a designation device comprising: the input comprising vertices of an external form of the object to be moved; and make a display device display: a two-dimensional image including the object to be moved; and the surface designating the predetermined surface, wherein the designation device is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the designation device taught in Watanabe with the input comprising vertices of an external form of the object to be moved; and make a display device display: a two-dimensional image including the object to be moved; and the surface designating the predetermined surface, wherein the designation device is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal taught in Watts with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Regarding claim 2, Watanabe in view of Watts teaches wherein the processor is configured to execute the instructions to display an axis having a predetermined angle with respect to the surface designating the predetermined surface (Watanabe: Column 10 line 53 – Column 11 line 4, “The set manual-mode coordinate system is displayed on the screen 14 in the form of, for example, an image representing a rectangular three axes, as shown in FIG. 8.”, Column 12 lines 9-34, “In the above example, the orientation or axis-direction of the manual-mode coordinate system is determined in such a manner that, for example, the +Z axis direction conforms to a direction perpendicular to a three-dimensional plane defined by the circular arc 44 and coincides with a direction of a +Z axis of one basic or common coordinate system predetermined in a layout space (hereinafter referred to as a layout coordinate system). However, if a normal line on the plane defined by the circular arc 44 is parallel to the XY plane of the layout coordinate system, the axis-direction of the manual-mode coordinate system is determined so that the +Z axis direction conforms to a direction of a + X axis of the layout coordinate system. If the orientation of the manual-mode coordinate system still cannot be determined, then the axis-direction is determined so that the +Z axis direction conforms to a direction of a + Y axis of the layout coordinate system. … . FIG. 13 shows a display example of the manual-mode coordinate system as set in this procedure.”. The cited passages clearly show that the axes are displayed and have a predetermined angle relative to the surface of the object.).
Regarding claim 4, Watanabe in view of Watts teaches wherein the predetermined angle is 90 degrees (Watanabe: Column 12 lines 9-34, “In the above example, the orientation or axis-direction of the manual-mode coordinate system is determined in such a manner that, for example, the +Z axis direction conforms to a direction perpendicular to a three-dimensional plane defined by the circular arc 44 and coincides with a direction of a +Z axis of one basic or common coordinate system predetermined in a layout space (hereinafter referred to as a layout coordinate system).”. One of ordinary skill in the art would recognize that the Z+ axis being perpendicular to the plane defined by the surface means that the z+ axis is at a 90° angle relative to said surface. Therefore, the cited passage clearly teaches wherein the predetermined angle is 90 degrees.).
Regarding claim 5, Watanabe in view of Watts teaches a robot system comprising: the designation device according to claim 1 (Watanabe: Figure 3, Column 5 lines 34-51, “In this embodiment, a robot controller 30 controls the operations of respective control axes provided in the robot mechanism 26. A movable hand 12 is attached, as the end-effector 12, to the distal end of an arm of the robot mechanism 26. The opening or closing action of the hand 12 is executed through the operation of an additional control axis (e.g., the seventh axis in a six-axis robot). In this arrangement, the robot controller 30 controls the operation of the additional control axis so as to make the hand 12 open or close, and recognizes the existing condition of the hand 12 (i.e., the existing position of the additional control axis) as occasion demands.”, Column 5 lines 52-67, “The robot controller 30 is further connected with a teaching unit 32 and peripheral equipment 34 through respective interfaces (not shown). As shown in FIG. 4, the teaching unit 32 includes electronic parts incorporated therein, such as a CPU 36, a memory (or a storage section) 38, etc., and is also provided with a display screen 14 and various operation keys 40 arranged on the outer surface of a casing.”. The cited passages clearly teach a robot system including the designation device.);
a robot configured to be capable of grasping an object to be moved (Watanabe: Figure 3, Column 5 lines 34-51, “In this embodiment, a robot controller 30 controls the operations of respective control axes provided in the robot mechanism 26. A movable hand 12 is attached, as the end-effector 12, to the distal end of an arm of the robot mechanism 26. The opening or closing action of the hand 12 is executed through the operation of an additional control axis (e.g., the seventh axis in a six-axis robot). In this arrangement, the robot controller 30 controls the operation of the additional control axis so as to make the hand 12 open or close, and recognizes the existing condition of the hand 12 (i.e., the existing position of the additional control axis) as occasion demands.”. The cited passage clearly shows that the robot is capable of gripping an object. This is also shown in Figures 7-11.);
and a control device configured to make the robot grasp the object to be moved based on an external form of the object to be moved, received by the designation device (Watts: Column 17 lines 50-67, Column 24 lines 16-47, “And at block 306, the control system causes the robotic manipulator to perform the task based on the modification to the virtual boundary line. In particular, the control system may cause the robotic manipulator to perform the task by sending instructions to the robotic manipulator that includes information indicative of how to perform the task using the modified virtual boundary line.”, Column 30 lines 49-64, “FIG. 8 illustrates another example interface of a remote assistor device, in accordance with at least some implementations described herein. In particular, FIG. 8 illustrates a remote assistor device 800 that has received a request to distinguish boxes in region 802 of a model 804 of objects, similar to region 602 of model 604 shown in FIG. 6 and region 702 of model 704 shown in FIG. 7. As shown, the GUI 806 of the remote assistor device includes a virtual boundary line 808 identified by the control system that does not correctly correspond to an actual boundary between the two adjacent boxes. In this scenario, a human user may select the option to "Adjust" the virtual boundary like 808 and then proceed to drag the virtual boundary line to the right until it corresponds to the actual boundary, so as to enable the control system to instruct the robotic manipulator to manipulate each of the two boxes.”. The cited passages clearly show that the system is configured to grasp an object based on the form of the object received from the designation device.).
Regarding claim 6, Watanabe teaches a designation method executed by a designation device, comprising (Watanabe: Figure 1 display section 16, Abstract, “A manual-mode operating system for a robot provided with an end-effector. The manual-mode operating system includes a display section including a screen for displaying an image of an end-effector; a first input section for an input of coordinate system data on the screen of the display section, displaying the image of the end-effector, the coordinate system data designating a manual-mode coordinate system used for describing an orientation of the end-effector; a coordinate system setting section for setting the manual mode coordinate system at a spatial position corresponding to an input position on the screen of the display section, based on the coordinate system data input through the first input section; a second input section for an input of orientation data using the manual-mode coordinate system set through the coordinate system setting section, the orientation data instructing the orientation of the end-effector; and an orientation adjusting section for adjusting the orientation of the end-effector, based on the orientation data input through the second input section.”, Column 5 lines 52-67, “The robot controller 30 is further connected with a teaching unit 32 and peripheral equipment 34 through respective interfaces (not shown). As shown in FIG. 4, the teaching unit 32 includes electronic parts incorporated therein, such as a CPU 36, a memory (or a storage section) 38, etc., and is also provided with a display screen 14 and various operation keys 40 arranged on the outer surface of a casing.”. The cited passages clearly teaches a device used to designate inputs for a robot.):
receiving an input for designating a predetermined surface of an object to be moved (Watanabe: Column 11 line 46 – Column 12 line 8, “Another example of the origin designation of the manual mode coordinate system will be described with reference to FIGS. 12 and 13. In the illustrated example, instead of directly designating the origin of the manual-mode coordinate system on the screen 14, a desired diagram (i.e., an area surrounded by a graphic line) or diagram part (i.e., a graphic line or point) in the displayed image is designated and the origin coordinate is determined in accordance with a predetermined rule. In FIG. 12, a circular arc 44 in the 3D image is illustrated to be designated on the screen 14. The area surrounded by the circular arc 44 in FIG. 12 represents, for example, a hole formed in the workpiece support 42 and, in such a case as to teach the task of inserting a rod-like workpiece (not shown) gripped by the hand into this hole, the circular arc 44 or the surrounded area thereof (i.e., the hole) is picked by the pointer. Then, the CPU 36 of the teaching unit 32 calculates the three-dimensional position of, for example, the center point of the circular arc 44, and identifies the calculated position with the origin position of the manual-mode coordinate system.”, Column 12 lines 35-51, “In the above procedure, the "face" of an object (or an image), surrounded by a closed curve, may be designated, instead of designating the closed curve such as the circular arc. When the "face" is designated, the position of the center of gravity of a contour diagram encircling the "face", for example, is identified with the origin of the manual-mode coordinate system. The determination of the orientation or axis-direction of the coordinate system may be performed in the same manner as in the above-described example of the circular arc.”. The cited passages teach that a user is able to designate either an area surrounded by a graphic line of an object or a face of an object. A person of ordinary skill in the art would recognize that an area surrounded by a graphic line is functionally similar to selecting a face of an object.);
and making a display device display: an image including the object to be moved (Watanabe: Figures 7-13, Column 7 lies 25-37, “Also, as occasion demands, the existing conditions of the robot mechanism 26, the peripheral equipment 34, a work piece, etc. are displayed in the form of 3D graphics together with the hand 12.”, Column 7 lines 44-53, “Also, for example, if it is required that a condition where a workpiece is gripped by the hand 12 is displayed on the screen 14, the shape data of the workpiece in the form of an appendix or inclusion of the shape data of the hand 12 as well as the correlation in position and orientation between the hand 12 and the workpiece in the gripped condition should be previously stored in the memory 38 of the teaching unit 32. The CPU 36 uses these data and executes the operation for 3D-displaying the image of the workpiece gripped by the hand 12 (see an example described later).”, Column 13 lines 35-40, “Next, at step T2, the personal computer 46 creates a signal representing a 3D graphic image including at least the image of the hand 12, based on the shape data, layout data, etc. of the robot mechanism 26, hand 12, peripheral equipment 34, workpiece, etc., previously stored in the memory, as well as on the data received from the robot controller 30 at step Tl.”);
Watanabe does not teach the input comprising vertices of an external form of the object to be moved
and making a display device display: a two-dimensional image including the object to be moved,
and the received surface designating the predetermined surface,
wherein the designation device is included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal.
Watts, in the same field of endeavor, teaches the input comprising vertices of an external form of the object to be moved (Watts: Column 30 lines 31-48, “FIG. 7 illustrates another example interface of a remote assistor device, in accordance with at least some implementations described herein. In particular, FIG. 7 illustrates a remote assistor device 700 that has received a request the distinguish boxes in region 702 of a model 704 of objects, similar to region 602 of model 604 shown in FIG. 6. As shown, the GUI 706 of the remote assistor device includes two corner virtual boundary lines 708, 710 identified by the control system that correspond to a top right corner of the left box and a bottom left comer of the right box. Similar to virtual boundary line 608 in FIG. 6, a human user may accept, reject, or adjust one or both of the virtual corners 708, 710 so as to enable the control system to instruct the robotic manipulator to manipulate each of the two boxes. However, by identifying and providing comers to the remote assistor device, the control system can enable the human user adjust two perpendicular virtual boundary lines simultaneously.”. The cited passages teach that the system takes, as an input, corners of an object or objects that are used by the system to determine the object to manipulate. The corners can be designated by a user using an input device. One of ordinary skill in the art would recognize that the corners of an object are the vertices of said object.)
and making a display device display: a two-dimensional image including the object to be moved (Watts: Figure 6, Column 28 lines 20-34, “At block 502, in response to receiving the request, the remote assistor device provides, on its display, a graphical user interface (GUI) representative of the model and the virtual boundary line. Generally, the remote assistor device may provide visual data of the objects in the environment, and the model is one such example. Additionally or alternatively to providing a model of the objects, the remote assistor device may provide (after receiving it from the control system) a camera image of the objects in the environment. In other examples, the remote assistor device may provide a synthesized visual representation combining a camera image of the objects and a depth map of the objects. Other visual data associated with the objects in the environment is possible as well to enable the human user to provide useful feedback.”, Column 28 lines 35-51, “The GUI may include one or more of the graphical elements described above. The remote assistor device may provide a GUI including a 2D view of the model and/or a 3D view of the model, each of which may include a depth map of the objects. Within examples, the remote assistor device may provide one or more 2D images of a facade of the objects from respective viewpoints, a static 3D model, and/or a rotatable ( or otherwise adjustable and/or navigable) 3D model, such as a 3D model that a human user can zoom in on or zoom out from and can rotate about one or more axes.”, Column 29 lines 36-67, “The GUI 606 includes an indication of the task for which the control system requested remote assistance ( e.g., "Box Detection"), an indication of a user identifier of the human user operating the remote assistor device 600 e.g., ("REMOTE ASSISTOR #1"), as well as an indication of which robotic device in the workplace will be performing at least a portion of the task (e.g., "ROBOT #1"). In this case, ROBOT #1 may manipulate at least the two boxes at issue in region 602, based on human user feedback. The GUI also includes a visual representation of region 602, box hypotheses that the control system determined for region 602, a virtual boundary line 608 separating two adjacent virtual boxes, and a confidence score associated with the virtual boundary line (e.g., "CONFIDENCE: HIGH").”, The cited passages clearly show that the system is configured to display a 2D image on a display device.),
and the received surface designating the predetermined surface (Watts: Column 30 lines 49-64, “FIG. 8 illustrates another example interface of a remote assistor device, in accordance with at least some implementations described herein. In particular, FIG. 8 illustrates a remote assistor device 800 that has received a request to distinguish boxes in region 802 of a model 804 of objects, similar to region 602 of model 604 shown in FIG. 6 and region 702 of model 704 shown in FIG. 7. As shown, the GUI 806 of the remote assistor device includes a virtual boundary line 808 identified by the control system that does not correctly correspond to an actual boundary between the two adjacent boxes. In this scenario, a human user may select the option to "Adjust" the virtual boundary like 808 and then proceed to drag the virtual boundary line to the right until it corresponds to the actual boundary, so as to enable the control system to instruct the robotic manipulator to manipulate each of the two boxes.”. The cited passages teach a method by which a user can adjust the boundaries defining the surfaces of two boxes. These surfaces are shown by a thick black line designating the outer edges of the surfaces of the boxes and a dashed black line representing the virtual boundary (i.e. where the two boxes meet). The system is further configured to allow a user to change this virtual boundary. One of ordinary skill in the art would have recognized that that the virtual boundary and blacklines represent the surface designating a predetermined surface. Therefore, the cited passages clearly teach that the system is configured to display a surface designating a predetermined surface.),
wherein the designation device is included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal (Watts: Column 6 lines 34-45, “After receiving the response from the remote assistor device, the control system may control the robotic manipulator to perform the given task in accordance with the response. For instance, if the response modifies the virtual boundary line to correctly correspond to an actual boundary between two objects in the environment, the robotic manipulator may pick up and move each of the two objects.”, Column 17 lines 50-67, Column 24 lines 16-47, “And at block 306, the control system causes the robotic manipulator to perform the task based on the modification to the virtual boundary line. In particular, the control system may cause the robotic manipulator to perform the task by sending instructions to the robotic manipulator that includes information indicative of how to perform the task using the modified virtual boundary line.”, Column 24 line 48 – Column 25 line 3, “Then, the control system may instruct the robotic manipulator to perform the manipulation of each of the two objects based on the confirmed actual boundary between the two objects. In particular, the control system may instruct the robotic manipulator to manipulate one of the objects at a first time without manipulating the other object, and then proceed to manipulate the other object at a second time. Alternatively, the control system may instruct the robotic manipulator to manipulate both of the two objects at the same time (e.g., pick up two objects at once). Within other examples, the task may not necessarily involve a manipulation of both objects on either side of the confirmed actual boundary, as noted above. Accordingly, the control system may instruct the robotic manipulator to perform the manipulation of one of the two objects without performing a manipulation on the other of the two objects.”, Column 29 lines 36-67, “In this case, ROBOT #1 may manipulate at least the two boxes at issue in region 602, based on human user feedback.”, Column 30 lines 49-64, “In this scenario, a human user may select the option to "Adjust" the virtual boundary like 808 and then proceed to drag the virtual boundary line to the right until it corresponds to the actual boundary, so as to enable the control system to instruct the robotic manipulator to manipulate each of the two boxes.”. The cited passages clearly show that the robot is configured to move the objects based on a predetermined algorithm.).
Watanabe teaches a designation method executed by a designation device, comprising: receiving an input for designating a predetermined surface of an object to be moved; and making a display device display: an image including the object to be moved. Watanabe does not teach the input comprising vertices of an external form of the object to be moved; and making a display device display: a two-dimensional image including the object to be moved, and the received surface designating the predetermined surface, wherein the designation device is included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal. Watts teaches and making a display device display: a two-dimensional image including the object to be moved, and the received surface designating the predetermined surface, wherein the designation device is included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal. A person of ordinary skill in the art would have had the technological capabilities required to have modified the method taught in Watanabe with the input comprising vertices of an external form of the object to be moved; and making a display device display: a two-dimensional image including the object to be moved, and the received surface designating the predetermined surface, wherein the designation device is included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal taught in Watts. Furthermore, Watanabe already teaches displaying an image of the object. Modifying Watanabe to display a two-dimensional image of the object instead of a three-dimensional image as taught in Watts is a simple matter of substituting a two-dimensional image for a three-dimensional image. Additionally, Watanabe teaches allowing a user to designate a surface of an object. Modifying Watanabe to display the surface designating the predefined surface as taught in Watts would only require the addition of the display method taught in Watts. While Watanabe does not explicitly teach designating the vertices of the object as the input, Watanabe clearly teaches that the vertices of the object can be used as the points to designate the object to be manipulated (Watanabe: Figure 9 desired point C, Figure 10 desired point D, Column 11, lines 28-55, “In FIG. 9, the image is displayed in the 3D way as viewed from a +Z axis direction. Then, the desired point C is designated on the image shown in FIG. 9. As a result of this, the X and Y coordinates of the position of the origin of the manual-mode coordinate system are determined. Next, the image as displayed is changed to the 3D image as viewed from the +X axis direction, as shown in FIG. 10, and the desired point D is designated on the image in FIG. 10. As a result of this, the Z coordinate of the position of the origin of the manual-mode coordinate system is determined.”). As such, one of ordinary skill in the art would have been easily able to modify Watanabe such that the system specifically uses the vertices as the input as taught in Watts. The method taught in Watanabe is used for teaching a robot a task, so modifying the robot to perform a task according to predetermined algorithm as taught in Watts would be well within the technological capabilities of one of ordinary skill in the art. Such modifications as described would not change or introduce new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a designation method executed by a designation device, comprising: the input comprising vertices of an external form of the object to be moved; and making a display device display: a two-dimensional image including the object to be moved, and the received surface designating the predetermined surface, wherein the designation device is included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the designation method taught in Watanabe with the input comprising vertices of an external form of the object to be moved; and making a display device display: a two-dimensional image including the object to be moved, and the received surface designating the predetermined surface, wherein the designation device is included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal taught in Watts with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Regarding claim 7, Watanabe teaches a non-transitory recording medium storing a program for causing a computer to (Watanabe: Column 5 lines 34-51, “FIG. 3 typically shows a configuration of a manual-mode operating system according to one embodiment of the present invention. This embodiment relates to the manual mode operating system 10 for an actual robot, as shown in FIG. 2A, so that the corresponding components are denoted by common reference numerals and the descriptions thereof are not repeated.”, Column 5 lines 52-67, “The robot controller 30 is further connected with a teaching unit 32 and peripheral equipment 34 through respective interfaces (not shown). As shown in FIG. 4, the teaching unit 32 includes electronic parts incorporated therein, such as a CPU 36, a memory (or a storage section) 38, etc., and is also provided with a display screen 14 and various operation keys 40 arranged on the outer surface of a casing.”):
receive an input for designating a predetermined surface of an object to be moved (Watanabe: Column 11 line 46 – Column 12 line 8, “Another example of the origin designation of the manual mode coordinate system will be described with reference to FIGS. 12 and 13. In the illustrated example, instead of directly designating the origin of the manual-mode coordinate system on the screen 14, a desired diagram (i.e., an area surrounded by a graphic line) or diagram part (i.e., a graphic line or point) in the displayed image is designated and the origin coordinate is determined in accordance with a predetermined rule. In FIG. 12, a circular arc 44 in the 3D image is illustrated to be designated on the screen 14. The area surrounded by the circular arc 44 in FIG. 12 represents, for example, a hole formed in the workpiece support 42 and, in such a case as to teach the task of inserting a rod-like workpiece (not shown) gripped by the hand into this hole, the circular arc 44 or the surrounded area thereof (i.e., the hole) is picked by the pointer. Then, the CPU 36 of the teaching unit 32 calculates the three-dimensional position of, for example, the center point of the circular arc 44, and identifies the calculated position with the origin position of the manual-mode coordinate system.”, Column 12 lines 35-51, “In the above procedure, the "face" of an object (or an image), surrounded by a closed curve, may be designated, instead of designating the closed curve such as the circular arc. When the "face" is designated, the position of the center of gravity of a contour diagram encircling the "face", for example, is identified with the origin of the manual-mode coordinate system. The determination of the orientation or axis-direction of the coordinate system may be performed in the same manner as in the above-described example of the circular arc.”. The cited passages teach that a user is able to designate either an area surrounded by a graphic line of an object or a face of an object. A person of ordinary skill in the art would recognize that an area surrounded by a graphic line is functionally similar to selecting a face of an object.);
and make a display device display an image including the object to be moved (Watanabe: Figures 7-13, Column 7 lies 25-37, “Also, as occasion demands, the existing conditions of the robot mechanism 26, the peripheral equipment 34, a work piece, etc. are displayed in the form of 3D graphics together with the hand 12.”, Column 7 lines 44-53, “Also, for example, if it is required that a condition where a workpiece is gripped by the hand 12 is displayed on the screen 14, the shape data of the workpiece in the form of an appendix or inclusion of the shape data of the hand 12 as well as the correlation in position and orientation between the hand 12 and the workpiece in the gripped condition should be previously stored in the memory 38 of the teaching unit 32. The CPU 36 uses these data and executes the operation for 3D-displaying the image of the workpiece gripped by the hand 12 (see an example described later).”, Column 13 lines 35-40, “Next, at step T2, the personal computer 46 creates a signal representing a 3D graphic image including at least the image of the hand 12, based on the shape data, layout data, etc. of the robot mechanism 26, hand 12, peripheral equipment 34, workpiece, etc., previously stored in the memory, as well as on the data received from the robot controller 30 at step Tl.”);
Watanabe does not teach the input comprising vertices of an external form of the object to be moved
and make a display device display a two-dimensional image including the object to be moved,
and the received surface designating the predetermined surface,
wherein the computer is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal.
Watts, in the same field of endeavor, teaches the input comprising vertices of an external form of the object to be moved (Watts: Column 30 lines 31-48, “FIG. 7 illustrates another example interface of a remote assistor device, in accordance with at least some implementations described herein. In particular, FIG. 7 illustrates a remote assistor device 700 that has received a request the distinguish boxes in region 702 of a model 704 of objects, similar to region 602 of model 604 shown in FIG. 6. As shown, the GUI 706 of the remote assistor device includes two corner virtual boundary lines 708, 710 identified by the control system that correspond to a top right corner of the left box and a bottom left comer of the right box. Similar to virtual boundary line 608 in FIG. 6, a human user may accept, reject, or adjust one or both of the virtual corners 708, 710 so as to enable the control system to instruct the robotic manipulator to manipulate each of the two boxes. However, by identifying and providing comers to the remote assistor device, the control system can enable the human user adjust two perpendicular virtual boundary lines simultaneously.”. The cited passages teach that the system takes, as an input, corners of an object or objects that are used by the system to determine the object to manipulate. The corners can be designated by a user using an input device. One of ordinary skill in the art would recognize that the corners of an object are the vertices of said object.)
and make a display device display a two-dimensional image including the object to be moved (Watts: Figure 6, Column 28 lines 20-34, “At block 502, in response to receiving the request, the remote assistor device provides, on its display, a graphical user interface (GUI) representative of the model and the virtual boundary line. Generally, the remote assistor device may provide visual data of the objects in the environment, and the model is one such example. Additionally or alternatively to providing a model of the objects, the remote assistor device may provide (after receiving it from the control system) a camera image of the objects in the environment. In other examples, the remote assistor device may provide a synthesized visual representation combining a camera image of the objects and a depth map of the objects. Other visual data associated with the objects in the environment is possible as well to enable the human user to provide useful feedback.”, Column 28 lines 35-51, “The GUI may include one or more of the graphical elements described above. The remote assistor device may provide a GUI including a 2D view of the model and/or a 3D view of the model, each of which may include a depth map of the objects. Within examples, the remote assistor device may provide one or more 2D images of a facade of the objects from respective viewpoints, a static 3D model, and/or a rotatable ( or otherwise adjustable and/or navigable) 3D model, such as a 3D model that a human user can zoom in on or zoom out from and can rotate about one or more axes.”, Column 29 lines 36-67, “The GUI 606 includes an indication of the task for which the control system requested remote assistance ( e.g., "Box Detection"), an indication of a user identifier of the human user operating the remote assistor device 600 e.g., ("REMOTE ASSISTOR #1"), as well as an indication of which robotic device in the workplace will be performing at least a portion of the task (e.g., "ROBOT #1"). In this case, ROBOT #1 may manipulate at least the two boxes at issue in region 602, based on human user feedback. The GUI also includes a visual representation of region 602, box hypotheses that the control system determined for region 602, a virtual boundary line 608 separating two adjacent virtual boxes, and a confidence score associated with the virtual boundary line (e.g., "CONFIDENCE: HIGH").”, The cited passages clearly show that the system is configured to display a 2D image on a display device.),
and the received surface designating the predetermined surface (Watts: Column 30 lines 49-64, “FIG. 8 illustrates another example interface of a remote assistor device, in accordance with at least some implementations described herein. In particular, FIG. 8 illustrates a remote assistor device 800 that has received a request to distinguish boxes in region 802 of a model 804 of objects, similar to region 602 of model 604 shown in FIG. 6 and region 702 of model 704 shown in FIG. 7. As shown, the GUI 806 of the remote assistor device includes a virtual boundary line 808 identified by the control system that does not correctly correspond to an actual boundary between the two adjacent boxes. In this scenario, a human user may select the option to "Adjust" the virtual boundary like 808 and then proceed to drag the virtual boundary line to the right until it corresponds to the actual boundary, so as to enable the control system to instruct the robotic manipulator to manipulate each of the two boxes.”. The cited passages teach a method by which a user can adjust the boundaries defining the surfaces of two boxes. These surfaces are shown by a thick black line designating the outer edges of the surfaces of the boxes and a dashed black line representing the virtual boundary (i.e. where the two boxes meet). The system is further configured to allow a user to change this virtual boundary. One of ordinary skill in the art would have recognized that that the virtual boundary and blacklines represent the surface designating a predetermined surface. Therefore, the cited passages clearly teach that the system is configured to display a surface designating a predetermined surface.),
wherein the computer is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal (Watts: Column 6 lines 34-45, “After receiving the response from the remote assistor device, the control system may control the robotic manipulator to perform the given task in accordance with the response. For instance, if the response modifies the virtual boundary line to correctly correspond to an actual boundary between two objects in the environment, the robotic manipulator may pick up and move each of the two objects.”, Column 17 lines 50-67, Column 24 lines 16-47, “And at block 306, the control system causes the robotic manipulator to perform the task based on the modification to the virtual boundary line. In particular, the control system may cause the robotic manipulator to perform the task by sending instructions to the robotic manipulator that includes information indicative of how to perform the task using the modified virtual boundary line.”, Column 24 line 48 – Column 25 line 3, “Then, the control system may instruct the robotic manipulator to perform the manipulation of each of the two objects based on the confirmed actual boundary between the two objects. In particular, the control system may instruct the robotic manipulator to manipulate one of the objects at a first time without manipulating the other object, and then proceed to manipulate the other object at a second time. Alternatively, the control system may instruct the robotic manipulator to manipulate both of the two objects at the same time (e.g., pick up two objects at once). Within other examples, the task may not necessarily involve a manipulation of both objects on either side of the confirmed actual boundary, as noted above. Accordingly, the control system may instruct the robotic manipulator to perform the manipulation of one of the two objects without performing a manipulation on the other of the two objects.”, Column 29 lines 36-67, “In this case, ROBOT #1 may manipulate at least the two boxes at issue in region 602, based on human user feedback.”, Column 30 lines 49-64, “In this scenario, a human user may select the option to "Adjust" the virtual boundary like 808 and then proceed to drag the virtual boundary line to the right until it corresponds to the actual boundary, so as to enable the control system to instruct the robotic manipulator to manipulate each of the two boxes.”. The cited passages clearly show that the robot is configured to move the objects based on a predetermined algorithm.).
Watanabe teaches a non-transitory recording medium storing a program for causing a computer to: receive an input for designating a predetermined surface of an object to be moved; and make a display device display an image including the object to be moved. Watanabe does not teach the input comprising vertices of an external form of the object to be moved; and make a display device display a two-dimensional image including the object to be moved; and the received surface designating the predetermined surface, wherein the computer is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal. Watts teaches the input comprising vertices of an external form of the object to be moved; and make a display device display a two-dimensional image including the object to be moved; and the received surface designating the predetermined surface, wherein the computer is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal. A person of ordinary skill in the art would have had the technological capabilities required to have modified the non-transitory recording medium taught in Watanabe with the input comprising vertices of an external form of the object to be moved; and make a display device display a two-dimensional image including the object to be moved; and the received surface designating the predetermined surface, wherein the computer is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal taught in Watts. Furthermore, Watanabe already teaches displaying an image of the object. Modifying Watanabe to display a two-dimensional image of the object instead of a three-dimensional image as taught in Watts is a simple matter of substituting a two-dimensional image for a three-dimensional image. Additionally, Watanabe teaches allowing a user to designate a surface of an object. Modifying Watanabe to display the surface designating the predefined surface as taught in Watts would only require the addition of the display method taught in Watts. While Watanabe does not explicitly teach designating the vertices of the object as the input, Watanabe clearly teaches that the vertices of the object can be used as the points to designate the object to be manipulated (Watanabe: Figure 9 desired point C, Figure 10 desired point D, Column 11, lines 28-55, “In FIG. 9, the image is displayed in the 3D way as viewed from a +Z axis direction. Then, the desired point C is designated on the image shown in FIG. 9. As a result of this, the X and Y coordinates of the position of the origin of the manual-mode coordinate system are determined. Next, the image as displayed is changed to the 3D image as viewed from the +X axis direction, as shown in FIG. 10, and the desired point D is designated on the image in FIG. 10. As a result of this, the Z coordinate of the position of the origin of the manual-mode coordinate system is determined.”). As such, one of ordinary skill in the art would have been easily able to modify Watanabe such that the system specifically uses the vertices as the input as taught in Watts. The non-transitory recording medium taught in Watanabe is used for teaching a robot a task, so modifying the robot to perform a task according to predetermined algorithm as taught in Watts would be well within the technological capabilities of one of ordinary skill in the art. Such modifications as described would not change or introduce new functionality. No inventive effort would have been required. The combination would have yielded the predictable result a non-transitory recording medium storing a program for causing a computer to: the input comprising vertices of an external form of the object to be moved; and make a display device display a two-dimensional image including the object to be moved; and the received surface designating the predetermined surface, wherein the computer is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the non-transitory recording medium taught in Watanabe with the input comprising vertices of an external form of the object to be moved; and make a display device display a two-dimensional image including the object to be moved; and the received surface designating the predetermined surface, wherein the computer is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal taught in Watts with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 7181315 B2 ("Watanabe") in view of US 9802317 B2 ("Watts") in further view of US 20190329405 A1 ("Atohira").
Regarding claim 3, Watanabe in view of Watts does not teach wherein the processor is configured to execute the instructions to input for adjusting a distance between the axis and the predetermined surface after a position and an orientation of the axis have been adjusted.
Atohira, in the same field of endeavor, teaches wherein the processor is configured to execute the instructions to input for adjusting a distance between the axis and the predetermined surface after a position and an orientation of the axis have been adjusted (Atohira: ¶ 0031, “An operation program display unit 112 displays information on the operation program of the robot 30 by superimposing it on the image of the real space displayed in the head-mounted display 80, using the operation program acquired. FIG. 7 illustrates an example of information on the operation program displayed in the head-mounted display 80. In the display example of FIG. 7, a three-dimensional image graphically showing nodes 201 to 204 of respective teaching points included in the operation program, and a path 210 connecting between the corresponding teaching points, are displayed by being superimposed on the image of the real space. In addition, a three-dimensional panel 220 showing a list of teaching points is displayed in the display example of FIG. 7. In this screen, when an operator moves his/her hand to touch and select a desired teaching point in the panel 220, or touches a node 202 of the teaching point with his/her hand, for example, a panel 230 showing a three-dimensional position of the selected teaching point is displayed by being superimposed on the image of the real space. In the example of FIG. 7, a teaching point 202 (a teaching point P2 in the panel 220) is selected. In description be low, display contents such as various operation panels and operation menus are also displayed as a three-dimensional image.”, ¶ 0032, “An operation-program changing unit 113 receives operation for changing teaching points performed by an operator. As an example of the operation for changing teaching points, the operator may move a node of each teaching point superimposed on the image of the real space by performing a drag operation with his/her hand. FIG. 8 illustrates a state where an operator moves the node 202 of the teaching point by performing a drag operation. Another example of operation for changing teaching points may be configured such that when an operator touches one of positional information items in the panel 230, indicating a three-dimensional position of a teaching point, with his/her hand, a numeric keypad 240 is displayed in the real space to allow the operator to operate the numeric keypad 240 to input a numeric value of the touched positional information. The example of FIG. 8 illustrates a state where a Z-coordinate in the panel 230, indicating a three-dimensional position of a teaching point, is selected by performing a touch operation and thereby the numeric keypad 240 is displayed.”. The cited passages teach that the system allows a user to change the three-dimensional position of teaching points used to control the robot.).
Watanabe in view of Watts teaches a designation device. Watanabe in view of Watts does not teach wherein the processor is configured to execute the instructions to input for adjusting a distance between the axis and the predetermined surface after a position and an orientation of the axis have been adjusted. Atohira teaches wherein the processor is configured to execute the instructions to input for adjusting a distance between the axis and the predetermined surface after a position and an orientation of the axis have been adjusted. A person of ordinary skill in the art would have had the technological capabilities required to have modified the device taught in Watanabe in view of Watts with wherein the processor is configured to execute the instructions to input for adjusting a distance between the axis and the predetermined surface after a position and an orientation of the axis have been adjusted taught in Atohira. Furthermore, even though Atohira only teaches changing the three-dimensional position of a control point, one of ordinary skill in the art would recognize that this would allow a user to also change the distance between the point and the object. Additionally, the device in Watanabe in view of Watts is already configured to allow a user to change a location of the axes with respect to the object. A person of ordinary skill in the art would have been able to have modified the device taught in Watanabe in view of Watts to be able to fully manipulate the thee-dimensional location of the axes using the method taught in Atohira. Such modifications would not have changed or introduced new functionality.
Response to Arguments
Applicant's arguments filed February 13th, 2026 have been fully considered but they are not persuasive.
Regarding Applicant’s arguments on Pages 6-8, Applicant argues that the prior art on record fails to teach the limitations of the amended independent claims 1, 6, and 7.
Specifically on Page 7-8, Applicant argues that the secondary reference Watts does not teach the limitation “a touch panel configured to receive an input for designating a predetermined surface of an object to be moved, the input comprising vertices of an external form of the object to be moved”. The Examiner respectfully disagrees. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As is stated above in the 35 U.S.C. § 103 rejection section of this Action, the primary reference Watanabe teaches a designation device comprising (Watanabe: Figure 1 display section 16, Abstract, Column 5 lines 52-67):a touch panel configured to receive an input for designating a predetermined surface of an object to be moved (Watanabe: Column 8 lines 1-10, Column 11 line 46 – Column 12 line 8, Column 12 lines 35-51), a memory configured to store instructions (Watanabe: Column 5 lines 34-51, Column 5 lines 52-67); and a processor configured to execute the instructions to (Watanabe: Column 5 lines 34-51, Column 5 lines 52-67): and make a display device display: an image including the object to be moved (Watanabe: Figures 7-13, Column 7 lies 25-37, Column 7 lines 44-53, Column 13 lines 35-40). As can clearly be seen, Watanabe can be configured such that the input device is a touch panel where a pen-like touch tool to designate the desired points on the object. The secondary reference Watts teaches the input comprising vertices of an external form of the object to be moved (Watts: Column 30 lines 31-48, “FIG. 7 illustrates another example interface of a remote assistor device, in accordance with at least some implementations described herein. In particular, FIG. 7 illustrates a remote assistor device 700 that has received a request the distinguish boxes in region 702 of a model 704 of objects, similar to region 602 of model 604 shown in FIG. 6. As shown, the GUI 706 of the remote assistor device includes two corner virtual boundary lines 708, 710 identified by the control system that correspond to a top right corner of the left box and a bottom left comer of the right box. Similar to virtual boundary line 608 in FIG. 6, a human user may accept, reject, or adjust one or both of the virtual corners 708, 710 so as to enable the control system to instruct the robotic manipulator to manipulate each of the two boxes. However, by identifying and providing comers to the remote assistor device, the control system can enable the human user adjust two perpendicular virtual boundary lines simultaneously.”) and make a display device display: a two-dimensional image including the object to be moved (Watts: Figure 6, Column 28 lines 20-34, Column 28 lines 35-51, Column 29 lines 36-67), and the surface designating the predetermined surface (Watts: Column 30 lines 49-64), wherein the designation device is configured to be included in a robot system that moves the object to be moved by following a predetermined algorithm in accordance with a work goal (Watts: Column 6 lines 34-45, Column 24 line 48 – Column 25 line 3, Column 29 lines 36-67). The cited passages of Watts clearly teaches that the system takes, as an input, corners of an object or objects that are used by the system to determine the object to manipulate. The corners can be designated by a user using an input device. One of ordinary skill in the art would recognize that the corners of an object are the vertices of said object. Additionally, while Watanabe does not explicitly teach designating the vertices of the object as the input, Watanabe clearly teaches that the vertices of the object can be used as the points to designate the object to be manipulated (Watanabe: Figure 9 desired point C, Figure 10 desired point D, Column 11, lines 28-55, “In FIG. 9, the image is displayed in the 3D way as viewed from a +Z axis direction. Then, the desired point C is designated on the image shown in FIG. 9. As a result of this, the X and Y coordinates of the position of the origin of the manual-mode coordinate system are determined. Next, the image as displayed is changed to the 3D image as viewed from the +X axis direction, as shown in FIG. 10, and the desired point D is designated on the image in FIG. 10. As a result of this, the Z coordinate of the position of the origin of the manual-mode coordinate system is determined.”). As such, one of ordinary skill in the art would have been easily able to modify Watanabe such that the system specifically uses the vertices as the input as taught in Watts. As such, Watanabe in view of Watts clearly teaches the limitation “a touch panel configured to receive an input for designating a predetermined surface of an object to be moved, the input comprising vertices of an external form of the object to be moved”.
Therefore, for the reasons stated above and in the in the 35 U.S.C. § 103 rejection section, the 35 U.S.C. § 103 rejection of the amended independent claims 1, 6, and 7 are maintained.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Noah W Stiebritz whose telephone number is (571)272-3414. The examiner can normally be reached Monday thru Friday 7-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached at (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.W.S./ Examiner, Art Unit 3658
/Ramon A. Mercado/Supervisory Patent Examiner, Art Unit 3658
Prosecution Insights