DETAILED ACTION
This is a Final Office Action on the merits in response to communications filed by Applicant on April 28th, 2026. Claims 1-23 are currently pending and examined below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendments to the Claims filed on April 28th, 2026, have been entered. Claims 1, 3, 5, 14-17 are currently amended and pending, claims 2, 4, 6-13, and 18-21 are as previously presented and pending, and claims 22 and 23 are new and pending. The amendments to the Claims filed on May 21st, 2026, have overcome each and every objection set forth in the previous Non-Final Office Action mailed April 16th, 2025.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 22 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2018/0029232 A1 ("Ouchi") in view of JP 2013180380 A (“Hashimoto”).
Regarding claim 22, Ouchi teaches a robot system comprising (Ouchi: Figure 1 robot system 1, ¶ 0044, “The robot system 1 is provided with a robot 20 and a control apparatus 30. In addition, the control apparatus 30 is provided with a robot control device 40 and an information processing device 50.”):
a robot (Ouchi: Figure 1 robot 20, ¶ 0044, “The robot system 1 is provided with a robot 20 and a control apparatus 30. In addition, the control apparatus 30 is provided with a robot control device 40 and an information processing device 50.”);
a display portion (Ouchi: ¶ 0073, “When receiving a parameter according to a task that the robot control device 40 causes the robot 20 to perform from the user, the information processing device 50 displays an operational screen, which is a screen on which task information indicating each of a plurality of tasks performed by the robot 20 provided with the force detecting unit 21 is displayed, receives a parameter according to a task of the robot 20 indicated by selected task information once task information is selected (in a case where certain task information is selected by the user), and displays information indicating operation of the robot 20 based on the received parameter onto the operational screen.”. The cited passage clearly shows that the system includes a display portion.),
wherein the controller is configured to display at least one of a type of the virtual dynamical characteristics and a parameter of the virtual dynamical characteristics on the display portion (Ouchi: Figure 9 region JG5, ¶ 0166, “The tab TB4 is a tab to display a GUI that receives a parameter of force control in the region TBR in the task indicated by task information selected by the user on the operational screen P2 illustrated in FIG. 5. Herein, the parameter is an impedance parameter since the force control is impedance control in this example. That is, the parameter is each of a virtual inertia coefficient, a virtual viscosity coefficient, and a virtual elasticity coefficient. The movement of the robot 20 through the force control is determined by these impedance parameters.”, ¶ 0169, “In the task indicated by the task information selected by the user on the operational screen P2 illustrated in FIG. 5, the region JG5 is a region to display a plurality of input fields, into which each of parameters according to the task (in this example, parameters of force control) is input by the user.”. The controller is clearly configured to display both the type and parameter of the virtual dynamical characteristics, wherein such quantities include a virtual inertia coefficient, a virtual viscosity coefficient, and a virtual elasticity coefficient. ).
Ouchi does not teach an image pickup unit configured to capture a current image; and
a controller configured to obtain information of virtual dynamical characteristics acting between a feature portion in a goal image and a feature portion in the current image and control the robot based on the information of the virtual dynamical characteristics.
Hashimoto, in the same field of endeavor, teaches an image pickup unit configured to capture a current image (Hashimoto: Figure 1 imaging device 30, ¶ 0021, “The robot 20 includes a support base 20a fixed to the ground, a manipulator part (manipulator) 20b connected to the support base 20a so as to be rotatable and rotatable, a gripping part 20c connected to the manipulator part 20b, a force sensor 20d, and an imaging device 30.”); and
a controller configured to obtain information of virtual dynamical characteristics acting between a feature portion in a goal image and a feature portion in the current image and control the robot based on the information of the virtual dynamical characteristics (Hashimoto: ¶ 0043, “Further, the visual control unit 110 detects the positional relationship between the component (object) 200 and the assembly component 210 based on the captured image. Specifically, for example, the visual control unit 110 uses a component (object) based on a current image obtained by imaging the current object and a goal image obtained by moving the object to a target position. The positional relationship between 200 and the part to be assembled 210 is detected. Then, the visual control unit 110 outputs a positional relationship signal indicating the detected positional relationship to a load displacement conversion unit 120 described later of the compliant motion control unit 150. Thereby, the load displacement conversion part 120 mentioned later determines the axis which prescribes | regulates the operation | movement of impedance control based on a positional relationship signal.”, ¶ 0045, “The load displacement conversion unit 120 acquires load information (for example, force applied to the component 200) input from the load calculation unit 40 at a predetermined sampling rate (1 kHz as an example in the present embodiment). Then, based on the positional relationship signal input from the visual control unit 110 and the load information acquired from the load calculation unit 40, the load displacement conversion unit 120 moves the relative position to which the robot 20 is moved according to the force.”, ¶ 0077, “Next, a goal image after the assembly of the component 200 is generated from the detected position and orientation of the component 210 to be assembled (step S104). The visual control unit 110 generates a first operation instruction signal by visual servoing based on the difference between the goal image and the current image (step S105). Next, the load displacement conversion unit 120 generates a second operation instruction signal by impedance control from the positions and postures of the component 200 and the assembly component 210 (step S106).”. The cited passages show that the force is determined based on the positional relationship of the object in the captured image and the assembled component in the goal image and the load. Additionally, impedance control is used to control the robot and said impedance control used the positional relationship and determined load. The impedance control therefore acts as a virtual attractive force. Additionally, the cited passages clearly shows that the robot is controlled such that the virtual attractive force (i.e. the impedance control) causes feature in the current image approaches the feature in the goal image.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot apparatus taught in Ouchi with an image pickup unit configured to capture a current image; and a controller configured to obtain information of virtual dynamical characteristics acting between a feature portion in a goal image and a feature portion in the current image and control the robot based on the information of the virtual dynamical characteristics taught in Hashimoto with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because such a method of control allows the robot to continue to operate even if it comes into contact with an obstacle and prevents damage to the object the robot is gripping (Hashimoto: ¶ 0082, “Thereby, even if the component (object) 200 contacts the obstacle, the control device 10 generates a control signal so that the component (object) 200 follows the obstacle. Therefore, the object follows the obstacle. Thus, the robot 20 can be controlled. Thereby, the robot 20 can assemble the component (object) 200 by moving the component (object) 200 to the final target position without destroying the component (object) 200.”).
Claim(s) 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2018/0029232 A1 ("Ouchi") in view of JP 2013180380 A (“Hashimoto”) in further view of US 2013/30238131 A1 (“Kondo”).
Regarding claim 23, Ouchi teaches a robot system comprising (Ouchi: Figure 1 robot system 1, ¶ 0044, “The robot system 1 is provided with a robot 20 and a control apparatus 30. In addition, the control apparatus 30 is provided with a robot control device 40 and an information processing device 50.”):
a robot (Ouchi: Figure 1 robot 20, ¶ 0044, “The robot system 1 is provided with a robot 20 and a control apparatus 30. In addition, the control apparatus 30 is provided with a robot control device 40 and an information processing device 50.”);
a display portion (Ouchi: ¶ 0073, “When receiving a parameter according to a task that the robot control device 40 causes the robot 20 to perform from the user, the information processing device 50 displays an operational screen, which is a screen on which task information indicating each of a plurality of tasks performed by the robot 20 provided with the force detecting unit 21 is displayed, receives a parameter according to a task of the robot 20 indicated by selected task information once task information is selected (in a case where certain task information is selected by the user), and displays information indicating operation of the robot 20 based on the received parameter onto the operational screen.”. The cited passage clearly shows that the system includes a display portion.),
Ouchi does not teach an image pickup unit configured to capture a current image; and
a controller configured to obtain information of virtual dynamical characteristics acting between a feature portion in a goal image and a feature portion in the current image and control the robot based on the information of the virtual dynamical characteristics,
wherein the controller is configured to allow the user to set whether or not to display the current image and the goal image in a superimposed manner on the display portion.
Hashimoto, in the same field of endeavor, teaches an image pickup unit configured to capture a current image (Hashimoto: Figure 1 imaging device 30, ¶ 0021, “The robot 20 includes a support base 20a fixed to the ground, a manipulator part (manipulator) 20b connected to the support base 20a so as to be rotatable and rotatable, a gripping part 20c connected to the manipulator part 20b, a force sensor 20d, and an imaging device 30.”); and
a controller configured to obtain information of virtual dynamical characteristics acting between a feature portion in a goal image and a feature portion in the current image and control the robot based on the information of the virtual dynamical characteristics (Hashimoto: ¶ 0043, “Further, the visual control unit 110 detects the positional relationship between the component (object) 200 and the assembly component 210 based on the captured image. Specifically, for example, the visual control unit 110 uses a component (object) based on a current image obtained by imaging the current object and a goal image obtained by moving the object to a target position. The positional relationship between 200 and the part to be assembled 210 is detected. Then, the visual control unit 110 outputs a positional relationship signal indicating the detected positional relationship to a load displacement conversion unit 120 described later of the compliant motion control unit 150. Thereby, the load displacement conversion part 120 mentioned later determines the axis which prescribes | regulates the operation | movement of impedance control based on a positional relationship signal.”, ¶ 0045, “The load displacement conversion unit 120 acquires load information (for example, force applied to the component 200) input from the load calculation unit 40 at a predetermined sampling rate (1 kHz as an example in the present embodiment). Then, based on the positional relationship signal input from the visual control unit 110 and the load information acquired from the load calculation unit 40, the load displacement conversion unit 120 moves the relative position to which the robot 20 is moved according to the force.”, ¶ 0077, “Next, a goal image after the assembly of the component 200 is generated from the detected position and orientation of the component 210 to be assembled (step S104). The visual control unit 110 generates a first operation instruction signal by visual servoing based on the difference between the goal image and the current image (step S105). Next, the load displacement conversion unit 120 generates a second operation instruction signal by impedance control from the positions and postures of the component 200 and the assembly component 210 (step S106).”. The cited passages show that the force is determined based on the positional relationship of the object in the captured image and the assembled component in the goal image and the load. Additionally, impedance control is used to control the robot and said impedance control used the positional relationship and determined load. The impedance control therefore acts as a virtual attractive force. Additionally, the cited passages clearly shows that the robot is controlled such that the virtual attractive force (i.e. the impedance control) causes feature in the current image approaches the feature in the goal image.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot apparatus taught in Ouchi with an image pickup unit configured to capture a current image; and a controller configured to obtain information of virtual dynamical characteristics acting between a feature portion in a goal image and a feature portion in the current image and control the robot based on the information of the virtual dynamical characteristics taught in Hashimoto with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because such a method of control allows the robot to continue to operate even if it comes into contact with an obstacle and prevents damage to the object the robot is gripping (Hashimoto: ¶ 0082, “Thereby, even if the component (object) 200 contacts the obstacle, the control device 10 generates a control signal so that the component (object) 200 follows the obstacle. Therefore, the object follows the obstacle. Thus, the robot 20 can be controlled. Thereby, the robot 20 can assemble the component (object) 200 by moving the component (object) 200 to the final target position without destroying the component (object) 200.”).
Ouchi in view of Hashimoto does not teach wherein the controller is configured to allow the user to set whether or not to display the current image and the goal image in a superimposed manner on the display portion.
Kondo, in the same field of endeavor, teaches wherein the controller is configured to allow the user to set whether or not to display the current image and the goal image in a superimposed manner on the display portion (Kondo: ¶ 0088, “FIG. 8A illustrates a state in which the detailed contour of the object to be held extracted in the region specified by the user is superimposed upon the captured image displayed on the display screen.”. The cited paragraph teaches overlaying two images, where one image is a contour and the other image is of the environment with the object. The cited paragraph also teaches displaying the overlaid image.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed, to have combine the robot apparatus taught in Ouchi in view of Hashimoto in further view of Matsuzaki in further view of NG with the method of overlaying two images and displaying the result taught in Kondo with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because it is a simple substation to the input of the algorithm. One of ordinary skill in the art would have the technological capabilities to see that the algorithm used to overlay two images does not change when applied to a different set of images. Furthermore, one of ordinary skill in the art would have the ability to modify the robot apparatus taught in Ouchi in view of Hashimoto in further view of Matsuzaki in further view of NG with the method of overlaying two images taught in Kondo without inventive effort.
Allowable Subject Matter
Claim(s) 1-21 is/are allowed.
The following is a statement of reasons for the indication of allowable subject matter:
Independent claim 1 teaches a robot apparatus comprising:
a robot;
an image pickup unit configured to capture a current image; and
a controller configured to control the robot,
wherein the controller obtains a goal image in which a first feature portion corresponding to a part of a control target object and a second feature portion corresponding to a part of an obstacle are included,
wherein the controller is configured to set a virtual attractive force toward the first feature portion and a virtual repulsive force away from the second feature portion,
wherein the controller obtains, by using the image pickup unit, the current image in which a third feature portion corresponding to the first feature portion in the goal image is included, and
wherein the controller controls the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image.
Independent claim 1 has since been amended to recite the limitation “wherein the controller controls the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. Said limitation is indicated as being allowable subject matter. The reasons for indicating allowable subject matter is as follows: The primary reference (US 2018/0029232 A1 (“Ouchi”)) teaches a system and method for controlling a robot. The system is configured to display to the user a plurality of operations for the robot to perform. Once an operation is selected by the user, the display allows the user to set the type and parameter of the virtual dynamical characteristics of the impedance control used to control the robot to perform the desired operation. While Ouchi does teach setting the type and parameters of the impedance control of the robot, Ouchi does not teach or suggest the setting or use of a virtual repulsive force. Therefore, Ouchi does not teach the limitation “wherein the controller controls the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (JP 2013180380 A (“Hashimoto”)) teaches a system and method for controlling a robot. The system is configured to capture a current image of the robot and the object grasped by the robot and a goal image of the completed assembly task. The force used to control the robot through impedance control is determined based on the positional relationship of the object in the captured image and the assembled component in the goal image and the load. Additionally, the impedance control used acts as a virtual attractive force. Furthermore, the robot is controlled such that the virtual attractive force (i.e. the impedance control) causes feature in the current image approaches the feature in the goal image. While Hashimoto does teach determining a virtual attractive force between a feature in a goal image and a feature in a current image and controlling the robot according to this virtual attractive force such that the feature in the current image moves toward the feature in the goal image, Hashimoto does not teach or suggest the setting or use of a virtual repulsive force. Therefore, Hashimoto does not teach the limitation “wherein the controller controls the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (JP 2012011498 A (“Matsuzaki”)) teaches a system and method for controlling a robot. The system is configured to set a virtual attractive fore between feature points on a robot and a desired object in the environment. The system is further configured to set a virtual repulsive force along a plurality of points on the obstacles in the environment, wherein the virtual repulsive force is set such that the force decrease as the distance between the obstacle and the robot increases. Both the virtual attractive and virtual repulsive force are set using a 3D model of the robot and its environment. While Matsuzaki does teach the use of a virtual repulsive and attractive force in the control of the robot, Matsuzaki does not teach that these force are set based on a feature point in a goal image and a corresponding feature point in a current image, or that the virtual repulsive force is determined based on the virtual attractive force and a distance between the feature point in a goal image and a corresponding feature point in a current image. Therefore, Matsuzaki does not teach the limitation “wherein the controller controls the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (US 9595095 B2 (“Aiso”)) teaches a system and method for controlling a robot. The system is configured to capture an image of a workspace and objects in said workspace using a camera. The system is the configured generate a model of the objects captured in the image and extract features of the objects based on said model. The system is additionally configured to, on a display, present the user with a button that, when pressed, causes the system to perform the feature extraction process. Aiso does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “wherein the controller controls the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (US 20210059762 A1 (“NG”)) teaches a system and method for allowing a user to select feature points in a medical image displayed on a display. NG does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “wherein the controller controls the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (US 20130238131 A1 (“Kondo”)) teaches a system and method for controlling a robot. The system is configured to display an image including an object to a user. The system is then configured to allow the user to elect a range on the display that includes the object. The system extracts the contours of the object in the range specified by the user, wherein the contours are then superimposed on the captured image. Kondo does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “wherein the controller controls the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference ((“Goswami”)) teaches a system and method for controlling a robot. The system is configured to present an image of the robot and its environment to the user. The system is configured to display a virtual tool of the system on the display such that the user can use this virtual tool to properly align the robot. Goswami does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “wherein the controller controls the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”.
Therefore, because the independent claim 1 has been amended with the above limitations, independent claim 1 is allowable for the reasons stated above.
Furthermore, the allowable features disclosed in independent claim 1, in combination with the additional elements of the independent claim, are patentably distinguishable over the prior art references.
Independent claim 14 teaches a method of controlling a robot apparatus, the method comprising:
obtaining a goal image in which a first feature portion corresponding to a part of a control target object and a second feature portion corresponding to a part of an obstacle are included;
setting a virtual attractive force toward the first feature portion and a virtual repulsive force away from the second feature portion,
obtaining, by using the image pickup unit, the current image in which a third feature portion corresponding to the first feature portion in the goal image is included, and
controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image.
Independent claim 14 has since been amended to recite the limitation “controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. Said limitation is indicated as being allowable subject matter. The reasons for indicating allowable subject matter is as follows: The primary reference (US 2018/0029232 A1 (“Ouchi”)) teaches a system and method for controlling a robot. The system is configured to display to the user a plurality of operations for the robot to perform. Once an operation is selected by the user, the display allows the user to set the type and parameter of the virtual dynamical characteristics of the impedance control used to control the robot to perform the desired operation. While Ouchi does teach setting the type and parameters of the impedance control of the robot, Ouchi does not teach or suggest the setting or use of a virtual repulsive force. Therefore, Ouchi does not teach the limitation “controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (JP 2013180380 A (“Hashimoto”)) teaches a system and method for controlling a robot. The system is configured to capture a current image of the robot and the object grasped by the robot and a goal image of the completed assembly task. The force used to control the robot through impedance control is determined based on the positional relationship of the object in the captured image and the assembled component in the goal image and the load. Additionally, the impedance control used acts as a virtual attractive force. Furthermore, the robot is controlled such that the virtual attractive force (i.e. the impedance control) causes feature in the current image approaches the feature in the goal image. While Hashimoto does teach determining a virtual attractive force between a feature in a goal image and a feature in a current image and controlling the robot according to this virtual attractive force such that the feature in the current image moves toward the feature in the goal image, Hashimoto does not teach or suggest the setting or use of a virtual repulsive force. Therefore, Hashimoto does not teach the limitation “controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (JP 2012011498 A (“Matsuzaki”)) teaches a system and method for controlling a robot. The system is configured to set a virtual attractive fore between feature points on a robot and a desired object in the environment. The system is further configured to set a virtual repulsive force along a plurality of points on the obstacles in the environment, wherein the virtual repulsive force is set such that the force decrease as the distance between the obstacle and the robot increases. Both the virtual attractive and virtual repulsive force are set using a 3D model of the robot and its environment. While Matsuzaki does teach the use of a virtual repulsive and attractive force in the control of the robot, Matsuzaki does not teach that these force are set based on a feature point in a goal image and a corresponding feature point in a current image, or that the virtual repulsive force is determined based on the virtual attractive force and a distance between the feature point in a goal image and a corresponding feature point in a current image. Therefore, Matsuzaki does not teach the limitation “controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (US 9595095 B2 (“Aiso”)) teaches a system and method for controlling a robot. The system is configured to capture an image of a workspace and objects in said workspace using a camera. The system is the configured generate a model of the objects captured in the image and extract features of the objects based on said model. The system is additionally configured to, on a display, present the user with a button that, when pressed, causes the system to perform the feature extraction process. Aiso does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (US 20210059762 A1 (“NG”)) teaches a system and method for allowing a user to select feature points in a medical image displayed on a display. NG does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (US 20130238131 A1 (“Kondo”)) teaches a system and method for controlling a robot. The system is configured to display an image including an object to a user. The system is then configured to allow the user to elect a range on the display that includes the object. The system extracts the contours of the object in the range specified by the user, wherein the contours are then superimposed on the captured image. Kondo does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference ((“Goswami”)) teaches a system and method for controlling a robot. The system is configured to present an image of the robot and its environment to the user. The system is configured to display a virtual tool of the system on the display such that the user can use this virtual tool to properly align the robot. Goswami does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”.
Therefore, because the independent claim 14 has been amended with the above limitations, independent claim 14 is allowable for the reasons stated above.
Furthermore, the allowable features disclosed in independent claim 14, in combination with the additional elements of the independent claim, are patentably distinguishable over the prior art references.
Independent claim 15 teaches an image processing apparatus configured to obtain information for performing force control of a robot, the image processing apparatus comprising:
a controller configured to:
obtain a goal image in which a first feature portion corresponding to a part of a control target object and a second feature portion corresponding to a part of an obstacle are included;
set a virtual attractive force toward the first feature portion and a virtual repulsive force away from the second feature portion,
obtain, by using the image pickup unit, the current image in which a third feature portion corresponding to the first feature portion in the goal image is included, and
output information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image.
Independent claim 15 has since been amended to recite the limitation “output information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. Said limitation is indicated as being allowable subject matter. The reasons for indicating allowable subject matter is as follows: The primary reference (US 2018/0029232 A1 (“Ouchi”)) teaches a system and method for controlling a robot. The system is configured to display to the user a plurality of operations for the robot to perform. Once an operation is selected by the user, the display allows the user to set the type and parameter of the virtual dynamical characteristics of the impedance control used to control the robot to perform the desired operation. While Ouchi does teach setting the type and parameters of the impedance control of the robot, Ouchi does not teach or suggest the setting or use of a virtual repulsive force. Therefore, Ouchi does not teach the limitation “output information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (JP 2013180380 A (“Hashimoto”)) teaches a system and method for controlling a robot. The system is configured to capture a current image of the robot and the object grasped by the robot and a goal image of the completed assembly task. The force used to control the robot through impedance control is determined based on the positional relationship of the object in the captured image and the assembled component in the goal image and the load. Additionally, the impedance control used acts as a virtual attractive force. Furthermore, the robot is controlled such that the virtual attractive force (i.e. the impedance control) causes feature in the current image approaches the feature in the goal image. While Hashimoto does teach determining a virtual attractive force between a feature in a goal image and a feature in a current image and controlling the robot according to this virtual attractive force such that the feature in the current image moves toward the feature in the goal image, Hashimoto does not teach or suggest the setting or use of a virtual repulsive force. Therefore, Hashimoto does not teach the limitation “output information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (JP 2012011498 A (“Matsuzaki”)) teaches a system and method for controlling a robot. The system is configured to set a virtual attractive fore between feature points on a robot and a desired object in the environment. The system is further configured to set a virtual repulsive force along a plurality of points on the obstacles in the environment, wherein the virtual repulsive force is set such that the force decrease as the distance between the obstacle and the robot increases. Both the virtual attractive and virtual repulsive force are set using a 3D model of the robot and its environment. While Matsuzaki does teach the use of a virtual repulsive and attractive force in the control of the robot, Matsuzaki does not teach that these force are set based on a feature point in a goal image and a corresponding feature point in a current image, or that the virtual repulsive force is determined based on the virtual attractive force and a distance between the feature point in a goal image and a corresponding feature point in a current image. Therefore, Matsuzaki does not teach the limitation “output information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (US 9595095 B2 (“Aiso”)) teaches a system and method for controlling a robot. The system is configured to capture an image of a workspace and objects in said workspace using a camera. The system is the configured generate a model of the objects captured in the image and extract features of the objects based on said model. The system is additionally configured to, on a display, present the user with a button that, when pressed, causes the system to perform the feature extraction process. Aiso does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “output information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (US 20210059762 A1 (“NG”)) teaches a system and method for allowing a user to select feature points in a medical image displayed on a display. NG does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “output information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (US 20130238131 A1 (“Kondo”)) teaches a system and method for controlling a robot. The system is configured to display an image including an object to a user. The system is then configured to allow the user to elect a range on the display that includes the object. The system extracts the contours of the object in the range specified by the user, wherein the contours are then superimposed on the captured image. Kondo does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “output information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference ((“Goswami”)) teaches a system and method for controlling a robot. The system is configured to present an image of the robot and its environment to the user. The system is configured to display a virtual tool of the system on the display such that the user can use this virtual tool to properly align the robot. Goswami does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “output information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”.
Therefore, because the independent claim 15 has been amended with the above limitations, independent claim 15 is allowable for the reasons stated above.
Furthermore, the allowable features disclosed in independent claim 15, in combination with the additional elements of the independent claim, are patentably distinguishable over the prior art references.
Independent claim 16 teaches an image processing method of obtaining information for performing force control of a robot, the image processing method comprising:
obtaining a goal image in which a first feature portion corresponding to a part of a control target object and a second feature portion corresponding to a part of an obstacle are included;
setting a virtual attractive force toward the first feature portion and a virtual repulsive force away from the second feature portion,
obtaining, by using the image pickup unit, the current image in which a third feature portion corresponding to the first feature portion in the goal image is included, and
outputting information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image.
Independent claim 16 has since been amended to recite the limitation “outputting information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. Said limitation is indicated as being allowable subject matter. The reasons for indicating allowable subject matter is as follows: The primary reference (US 2018/0029232 A1 (“Ouchi”)) teaches a system and method for controlling a robot. The system is configured to display to the user a plurality of operations for the robot to perform. Once an operation is selected by the user, the display allows the user to set the type and parameter of the virtual dynamical characteristics of the impedance control used to control the robot to perform the desired operation. While Ouchi does teach setting the type and parameters of the impedance control of the robot, Ouchi does not teach or suggest the setting or use of a virtual repulsive force. Therefore, Ouchi does not teach the limitation “outputting information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (JP 2013180380 A (“Hashimoto”)) teaches a system and method for controlling a robot. The system is configured to capture a current image of the robot and the object grasped by the robot and a goal image of the completed assembly task. The force used to control the robot through impedance control is determined based on the positional relationship of the object in the captured image and the assembled component in the goal image and the load. Additionally, the impedance control used acts as a virtual attractive force. Furthermore, the robot is controlled such that the virtual attractive force (i.e. the impedance control) causes feature in the current image approaches the feature in the goal image. While Hashimoto does teach determining a virtual attractive force between a feature in a goal image and a feature in a current image and controlling the robot according to this virtual attractive force such that the feature in the current image moves toward the feature in the goal image, Hashimoto does not teach or suggest the setting or use of a virtual repulsive force. Therefore, Hashimoto does not teach the limitation “outputting information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (JP 2012011498 A (“Matsuzaki”)) teaches a system and method for controlling a robot. The system is configured to set a virtual attractive fore between feature points on a robot and a desired object in the environment. The system is further configured to set a virtual repulsive force along a plurality of points on the obstacles in the environment, wherein the virtual repulsive force is set such that the force decrease as the distance between the obstacle and the robot increases. Both the virtual attractive and virtual repulsive force are set using a 3D model of the robot and its environment. While Matsuzaki does teach the use of a virtual repulsive and attractive force in the control of the robot, Matsuzaki does not teach that these force are set based on a feature point in a goal image and a corresponding feature point in a current image, or that the virtual repulsive force is determined based on the virtual attractive force and a distance between the feature point in a goal image and a corresponding feature point in a current image. Therefore, Matsuzaki does not teach the limitation “outputting information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (US 9595095 B2 (“Aiso”)) teaches a system and method for controlling a robot. The system is configured to capture an image of a workspace and objects in said workspace using a camera. The system is the configured generate a model of the objects captured in the image and extract features of the objects based on said model. The system is additionally configured to, on a display, present the user with a button that, when pressed, causes the system to perform the feature extraction process. Aiso does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “outputting information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (US 20210059762 A1 (“NG”)) teaches a system and method for allowing a user to select feature points in a medical image displayed on a display. NG does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “outputting information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference (US 20130238131 A1 (“Kondo”)) teaches a system and method for controlling a robot. The system is configured to display an image including an object to a user. The system is then configured to allow the user to elect a range on the display that includes the object. The system extracts the contours of the object in the range specified by the user, wherein the contours are then superimposed on the captured image. Kondo does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “outputting information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”. The secondary reference ((“Goswami”)) teaches a system and method for controlling a robot. The system is configured to present an image of the robot and its environment to the user. The system is configured to display a virtual tool of the system on the display such that the user can use this virtual tool to properly align the robot. Goswami does not teach or suggest the determination or use of a virtual attractive or repulsive force, and therefore does not teach the limitation “outputting information for controlling the robot, when the third feature portion in the current image approaches the first feature portion in the goal image in which the virtual attractive force is set, to avoid the second feature portion in the goal image in which the virtual repulsive force is set based on the virtual attractive force and the virtual repulsive force calculated from positional relationships between the third feature portion in the current image and the first and second feature portions in the goal image”.
Therefore, because the independent claim 16 has been amended with the above limitations, independent claim 16 is allowable for the reasons stated above.
Furthermore, the allowable features disclosed in independent claim 16, in combination with the additional elements of the independent claim, are patentably distinguishable over the prior art references.
Finally, the dependent claims 2-13 and 17-21 are allowable as being dependent on an allowable independent claim.
Response to Arguments
Applicant's arguments filed April 28th, 2026 have been fully considered but they are not persuasive.
Regarding Applicant’s Arguments on Pages 13-14, Applicant argues that the prior at on record fails to teach the limitations of the new independent claims 22 and 23.
Specifically on Pages 13-14, Applicant argues that the prior art on record fails to teach the limitation “wherein the controller displays a second user interface image for receiving setting of the first feature portion and the second feature portion on a display portion”. The Examiner respectfully disagrees. The new independent claims so not recite the limitation “wherein the controller displays a second user interface image for receiving setting of the first feature portion and the second feature portion on a display portion” or a limitation of similar phraseology. The new independent claim 22 recites the limitation “wherein the controller is configured to display at least one of a type of the virtual dynamical characteristics and a parameter of the virtual dynamical characteristics on the display portion”, however this limitation does not specify that the first and second feature portions are set by the user on the display. The new independent claim 23 recites the limitation “wherein the controller is configured to allow the user to set whether or not to display the current image and the goal image in a superimposed manner on the display portion”, however this limitation does not specify that the first and second feature portions are set by the user on the display. As such, because neither new independent claim recites the limitation “wherein the controller displays a second user interface image for receiving setting of the first feature portion and the second feature portion on a display portion” or a limitation of similar phraseology, Applicant’s arguments are not persuasive. Furthermore, assuming that the new independent claims 22 and 23 did recite the limitation “wherein the controller displays a second user interface image for receiving setting of the first feature portion and the second feature portion on a display portion” or a limitation of similar phraseology, the Examiner would still respectfully disagree. As was stated in the Non-Final Office Action mailed January 28th, 2026, the primary reference Ouchi teaches a robot system comprising (Ouchi: Figure 1 robot system 1, ¶ 0044): a robot (Ouchi: Figure 1 robot 20, ¶ 0044); a display portion (Ouchi: ¶ 0073), wherein the controller is configured to display at least one of a type of the virtual dynamical characteristics and a parameter of the virtual dynamical characteristics on the display portion (Ouchi: Figure 9 region JG5, ¶ 0166, ¶ 0169). Ouchi teaches a system and method for controlling a robot. The system is configured to display to the user a plurality of operations for the robot to perform. Once a operation is selected by the user, the display allows the user to set the type and parameter of the virtual dynamical characteristics of the impedance control used to control the robot to perform the desired operation. The secondary reference Hashimoto teaches an image pickup unit configured to capture a current image (Hashimoto: Figure 1 imaging device 30, ¶ 0021); and a controller configured to obtain information of virtual dynamical characteristics acting between a feature portion in a goal image and a feature portion in the current image and control the robot based on the information of the virtual dynamical characteristics (Hashimoto: ¶ 0043, ¶ 0045, ¶ 0077). Hashimoto teaches a system and method for controlling a robot. The system is configured to capture a current image of the robot and the object grasped by the robot and a goal image of the completed assembly task. The force used to control the robot through impedance control is determined based on the positional relationship of the object in the captured image and the assembled component in the goal image and the load. Additionally, the impedance control used acts as a virtual attractive force. Additionally, the robot is controlled such that the virtual attractive force (i.e. the impedance control) causes feature in the current image approaches the feature in the goal image. The secondary reference NG teaches wherein the controller displays a second user interface image for receiving setting of the first feature portion and the second feature portion on a display portion (NG: ¶ 0117). NG teaches a system and method for allowing a user to select feature points in a medical image displayed on a display. The system taught in Ouchi in view of Hashimoto is already configured to allow a user to select various control parameters for use in the impedance control of the robot and is configured to determine the virtual attractive force between corresponding features in a goal image and a current image such that the feature in the current image moves toward the feature in the goal image. As such one of ordinary skill in the art would have been able to modify the system taught in Ouchi in view of Hashimoto such that the system allows a user to select feature points as taught in NG according to methods known in the art. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded predictable results.
Therefore, for the reasons stated above, the 35 U.S.C. § 103 rejections of the new independent claims 22 and 23 are maintained.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Noah W Stiebritz whose telephone number is (571)272-3414. The examiner can normally be reached Monday thru Friday 7-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached at (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.W.S./ Examiner, Art Unit 3658
/MOHAMAD O EL SAYAH/ Examiner, Art Unit 3658
Prosecution Insights