Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This Office action is in response to the amendment filed on 07/02/2026. Claims 1-5, 8-9, 12-14, 16-21, 24-26 and 51 are currently pending with claims 1-4, 8, 12-14, 16-17, 19, 26, and 51 being amended, and claims 6-7, 10-11, 15, 22-23, 27-50, and 52-75 being cancelled.
Response to Amendment
The amendments to the claims submitted on 07/02/2026 overcome the claim objections set forth in the previous Office action except for those set forth in the claim objection section.
Response to Arguments
Applicant’s arguments, see remarks and amendments, filed 07/02/2026, with respect to the rejection of claim 13 under 35 U.S.C. 112(b) have been fully considered and are persuasive. The rejection of claim 13 under 35 U.S.C. 112(b) has been withdrawn. Examiner would note that the broadest reasonable interpretation of this claim would encompass a first joint coupled to a first member and a second joint coupled to both the first member and a second member. In this configuration, the first and second joints would implicitly be aligned in a straight line regardless of the position/orientation of the second member.
Examiner notes wherein Applicant argues the newly amended limitations, which have not been addressed by the prior art of record. As such, Examiner has augmented the below rejection(s) in view of the prior art of record to address the newly amended limitations.
Applicant’s arguments, see remarks and amendments, filed 07/02/2026, with respect to the rejection of claim 1 under 35 U.S.C. 102(a)(2) have been fully considered. The Applicant has asserted that Hanson fails to teach a first encoder associated with a first joint and configuration of said encoder upon determination that the actual joint position is different from the expected joint position. Hanson suggests performing calibration to configure the actuators/servomotors of each joint, they do not specifically disclose that these also include an encoder. While most servomotors include an encoder, Logins (WO 2023136743 A1) has been cited below to further demonstrate that this is known in the art.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 8-9, 12-14, 16, 26, and 51 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hanson et al. (US 20180350100 A1), hereinafter Hanson in view of Logins et al. (WO 2023136743 A1), hereinafter Logins and Alifragkis et al. (US 20150266183 A1), hereinafter Alifragkis.
Regarding claim 1, Hanson teaches:
1. (Currently Amended) A method comprising:
receiving first image data from at least one sensor when a robot is in a first pose, wherein a first joint of the robot and a second joint of the robot represented in the first image data has an expected joint position … when the robot is in the first pose;
determining, based on the first image data, an actual joint position of the first joint and the second joint when the robot is in the first pose; (Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again." Please also see Paragraph 0019 which demonstrates that each joint may be identified and the positions evaluated.) and
configuring … of a first actuator associated with the first joint based, (Paragraph 0018, "In some embodiments, robot 102 may include a mounting base 110 and a rotating base member 112. Rotating base member 112 may rotate around a base axis 113 using a base rotation actuator 114. A lower arm member 116 may movably connect to rotating base member 112 around a lower pivot joint 117 using a lower pivot actuator 118. An upper arm member 120 may movably connect to lower arm member 116 around an upper pivot joint 121 using an upper pivot actuator 122. Upper arm member 120 may movably connect to a rotating wrist member 124 that rotates around a roll axis 125 of upper arm member 120 at wrist roll joint 126 using wrist roll actuator 127. Rotating wrist member 124 may movably connect to a pivoting wrist member 128 that pivots around a wrist pivot joint 129 of rotating wrist member 124 using a wrist pivot actuator (not shown). Pivoting wrist member 128 may movably connect to an end effector mount 132 that rotates around an end effector roll axis 133 of pivoting wrist member 128 using end effector roll actuator 134 at end effector roll joint 135. Robot 102 is merely an example and many other configurations of robotic arms or other robot configurations are possible including at least one positioning member (such as lower arm member 116, upper arm member 120, rotating wrist member 124, pivoting wrist member 128, and end effector mount 132), at least one joint connected to the at least one positioning member (such as lower pivot joint 117, upper pivot joint 121, wrist roll joint 126, wrist pivot joint 129, end effector roll joint 135), and at least one actuator for moving the at least one positioning member around the at least one joint (such as base rotation actuator 114, lower pivot actuator 118, upper pivot actuator 122, wrist roll actuator 127, wrist pivot actuator, end effector roll actuator 134). In some embodiments, actuators 114, 118, 122, 127, 134 may each include a servo motor, such as an electric servo motor with positioning mechanics and a control loop for controllably positioning their respective members around the respective joints, and the servo motor may be control be a control signal from controller 104.") at least in part, on the actual joint position of the first joint and/or the second joint (Paragraphs 0024-0025, "In some embodiments, calibration module 170 may include an image processor 186 that extracts position information, such as the relative positions of one or more members and actuators of robot 102, from image data, such as reference images in reference images data source 172 and calibration images in calibration images data source 173. For example, a three dimensional image comparison algorithm may identify each reference mark present in the target image and identify each reference mark as either a robot reference mark or an environment reference mark. Comparison of the environment reference marks may enable the image comparison to correct for differences in camera position and then each robot reference mark from the reference image may be compared to each corresponding robot reference mark in the calibration image and deviations between the two positions (reference position in the reference image and actual position in the calibration image). Image processor 186 may convert the deviations between the images into quantified physical differences that can be compensated for by applying offset values to one or more actuators of robot 102. In some embodiments, image processor 186 may include one or more image comparison algorithms, such as a three dimensional image comparison application with one or more image processing algorithms, stored in and retrieved from image compare algorithms data source 175. In some embodiments, image processor 186 may include an actual position calculator 187 that calculates the actual position of robot 102 from the calibration image using some or all of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 with reference to environment reference marks 154, 155, 156, 157. Image processor 186 may include an image comparison calculator 188 that compares the calibration image to the reference image using at least three of environment reference marks 154, 155, 156, 157 to compensate for any variations in camera position and then determines any variations in the positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 between the compared images. In some embodiments, image processor 186 includes an actuator offset calculator 189 that calculates physical differences and/or actuator offsets from the variations in positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 determined by image comparison calculator 188. For example, actuator offset calculator 189 may include a scaling function based on physical parameters of robot 102 and/or operating environment 150 and/or their respective reference marks. In some embodiments, image processor 186 may determine calibration offsets to be used by controller 104 and store them in calibration offsets data source 174.
In some embodiments, calibration module 170 may include a positioning interface 192 that communicates calibration offsets to controller 104 for use by positioning controller 105. For example, positioning interface 192 may be invoked after a calibration is complete and send the most recent calibration offsets from calibration offsets data source 174 to controller 104 via I/O interface 168 to be stored in actuator offsets 108 and used by positioning controller 105 to control the actuators of robot 102. In some embodiments, positioning interface 192 enables controller 104 to query calibration module 170 and/or the stored actuator offsets in calibration offsets data source 174 to receive updated calibration values on demand." Please also see Paragraph 0019 which demonstrates that the system has a plurality of joints, each of which is identified and compared to a desired position.) when it is determined that the actual joint position of the first joint and the second joint is different from the expected joint position when the robot is in the first pose(Paragraph 0022, "In some embodiments, computing system 162 may include one or more application programs, data sources, and/or functional modules for calibrating robot 102 through controller 104. For example, computing system 162 may include a calibration module 170 and supporting data sources 172, 173, 174, 175 that support the functions, processes, and sub-modules of calibration module 170. Calibration module 170 may provide various processes for capturing a reference image and a calibration image, then processing those images to extract and compare position information for robot 102 to calculate one or more offsets for controller 104. Calibration module 170 may store, access, read, modify, or otherwise use data sources 172, 173, 174, 175. For example, reference images data source 172 may be used to store and retrieve one or more reference images related to robot 102 in one or more reference positions and captured at one or more reference times. Calibration images data source 173 may be used to store and retrieve one or more calibration images related to robot 102 in one or more calibration positions and captured at one or more calibration times. Calibration offsets data source 174 may be used to store and retrieve one or more calibration offsets for the actuators of robot 102 determined at one or more calibration times. Image compare algorithms 175 may include one or more image comparison algorithms, such as a three dimensional image comparison for calculating the relative positions of reference marks from image data, such as reference images and calibration images. One or more functions, processes, and sub-modules of calibration module 170 may interact with controller 104 and/or camera 163 via I/0 interface 168. Camera 163 may be selected from a variety of image data collectors, such as a digital camera or other image sensor, using visible light or another portion of the electromagnetic spectrum, such as infrared or ultraviolet imaging, to create an image that may be stored and processed as image data by calibration module 170." Please also see Paragraph 0019 which demonstrates that each joint may be identified and the positions evaluated.)
Hanson does not specifically disclose the desired pose being such that the joints are aligned or that each servomotor includes an encoder. However, Alifragkis, in the same field of endeavor of robotic calibration, teaches:
… in which the first joint and the second joint are aligned … (Paragraph 0028, “In the present patent application the term "location" comprises a position (e.g. x, y, z in a Cartesian coordinate system) as well as an orientation (e.g. a, b, c around the x-, y-, z-axes) of the distal end of the robot arm. Another term "pose" is used for describing a certain status of the robot arm with the chain links and the joints being in certain positions, orientations and angles. Due to the high degree of freedom regarding the movement of the distal end of the robot arm of an industrial robot, it is possible that one and the same location of the distal end can be achieved with different poses of the robot arm.” As well as Paragraph 0045, “Preferably, the calibration method described for only one calibration location is repeated for a plurality of calibration locations. A sensitivity matrix is determined for each of the calibration locations during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration locations are used for updating the kinematic model of the robot. Furthermore, the method is preferably repeated for a plurality of different calibration poses of the robot arm for each calibration location, each calibration pose corresponding to certain angle values of the articulated joints of the robot arm. A sensitivity matrix is determined for each of the calibration poses during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration poses are used for updating the kinematic model of the robot. The number of calibration poses needed for calibrating the robot depends on the complexity of the robot and the robot arm, respectively. In simple robot arm configurations or in situations where only part of the kinematic model of the robot is to be updated, even one calibration location and one or two corresponding calibration poses may be sufficient to calibrate the robot. In other cases more (e.g. at least five, preferably at least ten) different robot poses are used for determining all kinematic parameters of the robot in order to obtain a complete and precise updated kinematic model. Often, the kinematic model of a conventional industrial robot comprises at least 30 characteristic parameters. Each calibration pose provides for six equations and, hence, for the determination of at the most six robot calibration parameters. In order to compensate the influence of noise or other disturbances, it is suggested to choose the number of calibration poses such that the overall number of equations which can be formed in the various poses is much larger than the number of calibration parameters to be determined for the kinematic model of a certain type of robot.” This demonstrates that a robot may be controlled to reach a plurality of specific poses (each with corresponding joint angles) where the relative positioning of each element is known during the calibration process. This suggests alignment of the joints at each pose as well as in a straight line when joint angles are 0 degrees or 180 degrees.)
However, Logins, in the same field of endeavor of robotic calibration, teaches:
… a first encoder … (Page 3 Paragraphs 2-3, “Preferably, the calibration method described for only one calibration location is repeated for a plurality of calibration locations. A sensitivity matrix is determined for each of the calibration locations during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration locations are used for updating the kinematic model of the robot. Furthermore, the method is preferably repeated for a plurality of different calibration poses of the robot arm for each calibration location, each calibration pose corresponding to certain angle values of the articulated joints of the robot arm. A sensitivity matrix is determined for each of the calibration poses during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration poses are used for updating the kinematic model of the robot. The number of calibration poses needed for calibrating the robot depends on the complexity of the robot and the robot arm, respectively. In simple robot arm configurations or in situations where only part of the kinematic model of the robot is to be updated, even one calibration location and one or two corresponding calibration poses may be sufficient to calibrate the robot. In other cases more (e.g. at least five, preferably at least ten) different robot poses are used for determining all kinematic parameters of the robot in order to obtain a complete and precise updated kinematic model. Often, the kinematic model of a conventional industrial robot comprises at least 30 characteristic parameters. Each calibration pose provides for six equations and, hence, for the determination of at the most six robot calibration parameters. In order to compensate the influence of noise or other disturbances, it is suggested to choose the number of calibration poses such that the overall number of equations which can be formed in the various poses is much larger than the number of calibration parameters to be determined for the kinematic model of a certain type of robot.
Preferably, the calibration method described for only one calibration location is repeated for a plurality of calibration locations. A sensitivity matrix is determined for each of the calibration locations during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration locations are used for updating the kinematic model of the robot. Furthermore, the method is preferably repeated for a plurality of different calibration poses of the robot arm for each calibration location, each calibration pose corresponding to certain angle values of the articulated joints of the robot arm. A sensitivity matrix is determined for each of the calibration poses during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration poses are used for updating the kinematic model of the robot. The number of calibration poses needed for calibrating the robot depends on the complexity of the robot and the robot arm, respectively. In simple robot arm configurations or in situations where only part of the kinematic model of the robot is to be updated, even one calibration location and one or two corresponding calibration poses may be sufficient to calibrate the robot. In other cases more (e.g. at least five, preferably at least ten) different robot poses are used for determining all kinematic parameters of the robot in order to obtain a complete and precise updated kinematic model. Often, the kinematic model of a conventional industrial robot comprises at least 30 characteristic parameters. Each calibration pose provides for six equations and, hence, for the determination of at the most six robot calibration parameters. In order to compensate the influence of noise or other disturbances, it is suggested to choose the number of calibration poses such that the overall number of equations which can be formed in the various poses is much larger than the number of calibration parameters to be determined for the kinematic model of a certain type of robot.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robot and calibration methods as taught by Hanson with the ability to move the robot into poses with known joint angles as taught by Alifragkis as well as with the encoders taught by Logins. This would ensure highly effective calibration of the robotic system so that operation may be performed accurately despite changing elements and wear on components.
Regarding claim 2, where all the limitations of claim 1 are discussed above, Hanson further teaches:
2. (Currently Amended) The method of claim 1, further comprising:
receiving second image data from the at least one sensor when the robot is in a second pose, (Paragraphs 0022-0023, "In some embodiments, computing system 162 may include one or more application programs, data sources, and/or functional modules for calibrating robot 102 through controller 104. For example, computing system 162 may include a calibration module 170 and supporting data sources 172, 173, 174, 175 that support the functions, processes, and sub-modules of calibration module 170. Calibration module 170 may provide various processes for capturing a reference image and a calibration image, then processing those images to extract and compare position information for robot 102 to calculate one or more offsets for controller 104. Calibration module 170 may store, access, read, modify, or otherwise use data sources 172, 173, 174, 175. For example, reference images data source 172 may be used to store and retrieve one or more reference images related to robot 102 in one or more reference positions and captured at one or more reference times. Calibration images data source 173 may be used to store and retrieve one or more calibration images related to robot 102 in one or more calibration positions and captured at one or more calibration times. Calibration offsets data source 174 may be used to store and retrieve one or more calibration offsets for the actuators of robot 102 determined at one or more calibration times. Image compare algorithms 175 may include one or more image comparison algorithms, such as a three dimensional image comparison for calculating the relative positions of reference marks from image data, such as reference images and calibration images. One or more functions, processes, and sub-modules of calibration module 170 may interact with controller 104 and/or camera 163 via I/0 interface 168. Camera 163 may be selected from a variety of image data collectors, such as a digital camera or other image sensor, using visible light or another portion of the electromagnetic spectrum, such as infrared or ultraviolet imaging, to create an image that may be stored and processed as image data by calibration module 170.
In some embodiments, calibration module 170 may include a reference position process 178 that communicates a command to robot 102 and/or controller 104 to drive the actuators to a defined reference position. For example, reference position process 178 may issue a command to controller 104 to return robot 102 to the home position. In some embodiments, reference position process 178 may include a plurality of selectable reference positions for different calibration options. In some embodiments, calibration module 170 may include a reference image capture process 180 that captures or retrieves a reference image of robot 102 in a reference position for use in subsequent calibrations. For example, reference image capture process 180 may issue a command to camera 163 to capture an image of robot 102. In some embodiments, reference image capture process 180 may determine whether a previously captured reference image is available in reference images data source 172 and, if so, it may retrieve the stored reference image for use in the subsequent calibration, if not, it may use camera 163 to capture a new reference image of robot 102 and store it in reference images data source 172. In some embodiments, processes 178, 180 are executed proximate the time robot 102 is put into service and/or mechanically calibrated and validated to assure that the reference image or images accurately reflect a baseline image of the desired calibrated state of robot 102. In some embodiments, calibration module 170 may include a calibration position process 182 that communicates a command to robot 102 and/or controller 104 to drive the actuators to a defined calibration position. For example, calibration position process 182 may issue a command to controller 104 to return robot 102 to the home position. In some embodiments, calibration position process 128 may include a plurality of selectable calibration positions for different calibration options, each corresponding to an available reference position with a corresponding reference image in reference images data source 172. In some embodiments, calibration module 170 may include a calibration image capture process 184 that captures or retrieves a calibration image of robot 102 in a calibration position for use in the present calibration. For example, calibration image capture process 184 may issue a command to camera 163 to capture an image of robot 102. In some embodiments, calibration image capture process 180 may store each calibration image in calibration images data source 173. In some embodiments, processes 182, 184 are executed each time a new calibration is conducted.") wherein a third (Paragraph 0019, "Robot 102 may include a plurality of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 on its exterior surfaces that may be used for calibration operations. For example, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include visual indicia applied to the surface of robot 102 in desired locations that provide sufficient contrast or unique visual patterns that are easily detected within visual image data. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a circle or other shape with consistent dimensions and a high contrast color relative to the surface color of robot 102, such as a red circle on yellow or a black dot on white. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a distinctive pattern, such as a black dot in a white circle. Other visual indicia may be used as robot reference marks 140, 141, 142, 143, 144, 145, 146, 147. These indicia may be printed, painted, adhered, machined, or otherwise placed on robot 102 to mark the desired positions for visual reference. In the embodiment shown, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 are positioned on various positioning members of robot 102 relative to their respective joints to enable the position of each joint to be determined from visual image data. Each positioning member of interest may include one or more reference marks. For example, robot reference mark 140 on rotating base member 112 enables position information and calibration offsets to be calculated for base rotation actuator 114. Robot reference marks 141, 142 on lower arm member 116 enable position information and calibration offsets to be calculated for lower pivot actuator 118. Robot reference marks 143, 144 on upper arm member 120 enable position information and calibration offsets to be calculated for upper pivot actuator 122. Robot reference mark 145 on rotating wrist member 124 enables position information and calibration offsets to be calculated for wrist roll actuator 127. Robot reference mark 146 on pivoting wrist member 128 enables position information and calibration offsets to be calculated for the wrist pivot actuator. Robot reference mark 147 on end effector mount member 132 enables position information and calibration offsets to be calculated for end effector roll actuator 134. In some embodiments, the various members may include additional surfaces with different orientations and those additional surfaces may include additional robot reference marks to enable image data to be collected from additional orientations and fields of view. In some embodiments, two reference marks may be provided per joint or axis, but a single reference mark or greater than two reference marks are also possible where surface space and imaging and processing resources permit.") represented in the second image data has an expected joint position when the robot is in the second pose;
determining, based on the second image data, an actual joint position of the third (Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again.") and
configuring a second encoder (Paragraph 0018, "In some embodiments, robot 102 may include a mounting base 110 and a rotating base member 112. Rotating base member 112 may rotate around a base axis 113 using a base rotation actuator 114. A lower arm member 116 may movably connect to rotating base member 112 around a lower pivot joint 117 using a lower pivot actuator 118. An upper arm member 120 may movably connect to lower arm member 116 around an upper pivot joint 121 using an upper pivot actuator 122. Upper arm member 120 may movably connect to a rotating wrist member 124 that rotates around a roll axis 125 of upper arm member 120 at wrist roll joint 126 using wrist roll actuator 127. Rotating wrist member 124 may movably connect to a pivoting wrist member 128 that pivots around a wrist pivot joint 129 of rotating wrist member 124 using a wrist pivot actuator (not shown). Pivoting wrist member 128 may movably connect to an end effector mount 132 that rotates around an end effector roll axis 133 of pivoting wrist member 128 using end effector roll actuator 134 at end effector roll joint 135. Robot 102 is merely an example and many other configurations of robotic arms or other robot configurations are possible including at least one positioning member (such as lower arm member 116, upper arm member 120, rotating wrist member 124, pivoting wrist member 128, and end effector mount 132), at least one joint connected to the at least one positioning member (such as lower pivot joint 117, upper pivot joint 121, wrist roll joint 126, wrist pivot joint 129, end effector roll joint 135), and at least one actuator for moving the at least one positioning member around the at least one joint (such as base rotation actuator 114, lower pivot actuator 118, upper pivot actuator 122, wrist roll actuator 127, wrist pivot actuator, end effector roll actuator 134). In some embodiments, actuators 114, 118, 122, 127, 134 may each include a servo motor, such as an electric servo motor with positioning mechanics and a control loop for controllably positioning their respective members around the respective joints, and the servo motor may be control be a control signal from controller 104.") of a second actuator associated with the third third(Paragraphs 0024-0025, "In some embodiments, calibration module 170 may include an image processor 186 that extracts position information, such as the relative positions of one or more members and actuators of robot 102, from image data, such as reference images in reference images data source 172 and calibration images in calibration images data source 173. For example, a three dimensional image comparison algorithm may identify each reference mark present in the target image and identify each reference mark as either a robot reference mark or an environment reference mark. Comparison of the environment reference marks may enable the image comparison to correct for differences in camera position and then each robot reference mark from the reference image may be compared to each corresponding robot reference mark in the calibration image and deviations between the two positions (reference position in the reference image and actual position in the calibration image). Image processor 186 may convert the deviations between the images into quantified physical differences that can be compensated for by applying offset values to one or more actuators of robot 102. In some embodiments, image processor 186 may include one or more image comparison algorithms, such as a three dimensional image comparison application with one or more image processing algorithms, stored in and retrieved from image compare algorithms data source 175. In some embodiments, image processor 186 may include an actual position calculator 187 that calculates the actual position of robot 102 from the calibration image using some or all of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 with reference to environment reference marks 154, 155, 156, 157. Image processor 186 may include an image comparison calculator 188 that compares the calibration image to the reference image using at least three of environment reference marks 154, 155, 156, 157 to compensate for any variations in camera position and then determines any variations in the positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 between the compared images. In some embodiments, image processor 186 includes an actuator offset calculator 189 that calculates physical differences and/or actuator offsets from the variations in positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 determined by image comparison calculator 188. For example, actuator offset calculator 189 may include a scaling function based on physical parameters of robot 102 and/or operating environment 150 and/or their respective reference marks. In some embodiments, image processor 186 may determine calibration offsets to be used by controller 104 and store them in calibration offsets data source 174.
In some embodiments, calibration module 170 may include a positioning interface 192 that communicates calibration offsets to controller 104 for use by positioning controller 105. For example, positioning interface 192 may be invoked after a calibration is complete and send the most recent calibration offsets from calibration offsets data source 174 to controller 104 via I/O interface 168 to be stored in actuator offsets 108 and used by positioning controller 105 to control the actuators of robot 102. In some embodiments, positioning interface 192 enables controller 104 to query calibration module 170 and/or the stored actuator offsets in calibration offsets data source 174 to receive updated calibration values on demand.") when it is determined that the actual joint position of the third third (Paragraph 0022, "In some embodiments, computing system 162 may include one or more application programs, data sources, and/or functional modules for calibrating robot 102 through controller 104. For example, computing system 162 may include a calibration module 170 and supporting data sources 172, 173, 174, 175 that support the functions, processes, and sub-modules of calibration module 170. Calibration module 170 may provide various processes for capturing a reference image and a calibration image, then processing those images to extract and compare position information for robot 102 to calculate one or more offsets for controller 104. Calibration module 170 may store, access, read, modify, or otherwise use data sources 172, 173, 174, 175. For example, reference images data source 172 may be used to store and retrieve one or more reference images related to robot 102 in one or more reference positions and captured at one or more reference times. Calibration images data source 173 may be used to store and retrieve one or more calibration images related to robot 102 in one or more calibration positions and captured at one or more calibration times. Calibration offsets data source 174 may be used to store and retrieve one or more calibration offsets for the actuators of robot 102 determined at one or more calibration times. Image compare algorithms 175 may include one or more image comparison algorithms, such as a three dimensional image comparison for calculating the relative positions of reference marks from image data, such as reference images and calibration images. One or more functions, processes, and sub-modules of calibration module 170 may interact with controller 104 and/or camera 163 via I/0 interface 168. Camera 163 may be selected from a variety of image data collectors, such as a digital camera or other image sensor, using visible light or another portion of the electromagnetic spectrum, such as infrared or ultraviolet imaging, to create an image that may be stored and processed as image data by calibration module 170.")
Regarding claim 3, where all the limitations of claim 1 are discussed above, Hanson further teaches:
3. (Currently Amended) The method of claim 1, wherein configuring the first encoder based, at least in part, on the actual joint position of the first joint and/or the second joint comprises:
configuring the first encoder to scale an output of the first encoder by a scaling factor, the scaling factor being determined based on a difference between the actual joint position and the expected joint position. (Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again.")
Regarding claim 8, where all the limitations of claim 1 are discussed above, Hanson further teaches:
8. (Currently Amended) The method of claim 1, (Paragraph 0019, "Robot 102 may include a plurality of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 on its exterior surfaces that may be used for calibration operations. For example, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include visual indicia applied to the surface of robot 102 in desired locations that provide sufficient contrast or unique visual patterns that are easily detected within visual image data. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a circle or other shape with consistent dimensions and a high contrast color relative to the surface color of robot 102, such as a red circle on yellow or a black dot on white. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a distinctive pattern, such as a black dot in a white circle. Other visual indicia may be used as robot reference marks 140, 141, 142, 143, 144, 145, 146, 147. These indicia may be printed, painted, adhered, machined, or otherwise placed on robot 102 to mark the desired positions for visual reference. In the embodiment shown, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 are positioned on various positioning members of robot 102 relative to their respective joints to enable the position of each joint to be determined from visual image data. Each positioning member of interest may include one or more reference marks. For example, robot reference mark 140 on rotating base member 112 enables position information and calibration offsets to be calculated for base rotation actuator 114. Robot reference marks 141, 142 on lower arm member 116 enable position information and calibration offsets to be calculated for lower pivot actuator 118. Robot reference marks 143, 144 on upper arm member 120 enable position information and calibration offsets to be calculated for upper pivot actuator 122. Robot reference mark 145 on rotating wrist member 124 enables position information and calibration offsets to be calculated for wrist roll actuator 127. Robot reference mark 146 on pivoting wrist member 128 enables position information and calibration offsets to be calculated for the wrist pivot actuator. Robot reference mark 147 on end effector mount member 132 enables position information and calibration offsets to be calculated for end effector roll actuator 134. In some embodiments, the various members may include additional surfaces with different orientations and those additional surfaces may include additional robot reference marks to enable image data to be collected from additional orientations and fields of view. In some embodiments, two reference marks may be provided per joint or axis, but a single reference mark or greater than two reference marks are also possible where surface space and imaging and processing resources permit.") comprising
(Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again.")
configuring … of a second actuator associated with the second joint based, (Paragraph 0018, "In some embodiments, robot 102 may include a mounting base 110 and a rotating base member 112. Rotating base member 112 may rotate around a base axis 113 using a base rotation actuator 114. A lower arm member 116 may movably connect to rotating base member 112 around a lower pivot joint 117 using a lower pivot actuator 118. An upper arm member 120 may movably connect to lower arm member 116 around an upper pivot joint 121 using an upper pivot actuator 122. Upper arm member 120 may movably connect to a rotating wrist member 124 that rotates around a roll axis 125 of upper arm member 120 at wrist roll joint 126 using wrist roll actuator 127. Rotating wrist member 124 may movably connect to a pivoting wrist member 128 that pivots around a wrist pivot joint 129 of rotating wrist member 124 using a wrist pivot actuator (not shown). Pivoting wrist member 128 may movably connect to an end effector mount 132 that rotates around an end effector roll axis 133 of pivoting wrist member 128 using end effector roll actuator 134 at end effector roll joint 135. Robot 102 is merely an example and many other configurations of robotic arms or other robot configurations are possible including at least one positioning member (such as lower arm member 116, upper arm member 120, rotating wrist member 124, pivoting wrist member 128, and end effector mount 132), at least one joint connected to the at least one positioning member (such as lower pivot joint 117, upper pivot joint 121, wrist roll joint 126, wrist pivot joint 129, end effector roll joint 135), and at least one actuator for moving the at least one positioning member around the at least one joint (such as base rotation actuator 114, lower pivot actuator 118, upper pivot actuator 122, wrist roll actuator 127, wrist pivot actuator, end effector roll actuator 134). In some embodiments, actuators 114, 118, 122, 127, 134 may each include a servo motor, such as an electric servo motor with positioning mechanics and a control loop for controllably positioning their respective members around the respective joints, and the servo motor may be control be a control signal from controller 104.") at least in part, on the actual joint position of the second joint and/or the first joint (Paragraphs 0024-0025, "In some embodiments, calibration module 170 may include an image processor 186 that extracts position information, such as the relative positions of one or more members and actuators of robot 102, from image data, such as reference images in reference images data source 172 and calibration images in calibration images data source 173. For example, a three dimensional image comparison algorithm may identify each reference mark present in the target image and identify each reference mark as either a robot reference mark or an environment reference mark. Comparison of the environment reference marks may enable the image comparison to correct for differences in camera position and then each robot reference mark from the reference image may be compared to each corresponding robot reference mark in the calibration image and deviations between the two positions (reference position in the reference image and actual position in the calibration image). Image processor 186 may convert the deviations between the images into quantified physical differences that can be compensated for by applying offset values to one or more actuators of robot 102. In some embodiments, image processor 186 may include one or more image comparison algorithms, such as a three dimensional image comparison application with one or more image processing algorithms, stored in and retrieved from image compare algorithms data source 175. In some embodiments, image processor 186 may include an actual position calculator 187 that calculates the actual position of robot 102 from the calibration image using some or all of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 with reference to environment reference marks 154, 155, 156, 157. Image processor 186 may include an image comparison calculator 188 that compares the calibration image to the reference image using at least three of environment reference marks 154, 155, 156, 157 to compensate for any variations in camera position and then determines any variations in the positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 between the compared images. In some embodiments, image processor 186 includes an actuator offset calculator 189 that calculates physical differences and/or actuator offsets from the variations in positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 determined by image comparison calculator 188. For example, actuator offset calculator 189 may include a scaling function based on physical parameters of robot 102 and/or operating environment 150 and/or their respective reference marks. In some embodiments, image processor 186 may determine calibration offsets to be used by controller 104 and store them in calibration offsets data source 174.
In some embodiments, calibration module 170 may include a positioning interface 192 that communicates calibration offsets to controller 104 for use by positioning controller 105. For example, positioning interface 192 may be invoked after a calibration is complete and send the most recent calibration offsets from calibration offsets data source 174 to controller 104 via I/O interface 168 to be stored in actuator offsets 108 and used by positioning controller 105 to control the actuators of robot 102. In some embodiments, positioning interface 192 enables controller 104 to query calibration module 170 and/or the stored actuator offsets in calibration offsets data source 174 to receive updated calibration values on demand.") when it is determined that the actual joint position of the second joint and the first joint is different from the expected joint position when the robot is in the first pose (Paragraph 0022, "In some embodiments, computing system 162 may include one or more application programs, data sources, and/or functional modules for calibrating robot 102 through controller 104. For example, computing system 162 may include a calibration module 170 and supporting data sources 172, 173, 174, 175 that support the functions, processes, and sub-modules of calibration module 170. Calibration module 170 may provide various processes for capturing a reference image and a calibration image, then processing those images to extract and compare position information for robot 102 to calculate one or more offsets for controller 104. Calibration module 170 may store, access, read, modify, or otherwise use data sources 172, 173, 174, 175. For example, reference images data source 172 may be used to store and retrieve one or more reference images related to robot 102 in one or more reference positions and captured at one or more reference times. Calibration images data source 173 may be used to store and retrieve one or more calibration images related to robot 102 in one or more calibration positions and captured at one or more calibration times. Calibration offsets data source 174 may be used to store and retrieve one or more calibration offsets for the actuators of robot 102 determined at one or more calibration times. Image compare algorithms 175 may include one or more image comparison algorithms, such as a three dimensional image comparison for calculating the relative positions of reference marks from image data, such as reference images and calibration images. One or more functions, processes, and sub-modules of calibration module 170 may interact with controller 104 and/or camera 163 via I/0 interface 168. Camera 163 may be selected from a variety of image data collectors, such as a digital camera or other image sensor, using visible light or another portion of the electromagnetic spectrum, such as infrared or ultraviolet imaging, to create an image that may be stored and processed as image data by calibration module 170.")
Hanson does not specifically teach each joints servomotor including an encoder. However, Logins, in the same field of endeavor of robotic calibration, teaches:
… a second encoder … (Page 3 Paragraphs 2-3, “Preferably, the calibration method described for only one calibration location is repeated for a plurality of calibration locations. A sensitivity matrix is determined for each of the calibration locations during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration locations are used for updating the kinematic model of the robot. Furthermore, the method is preferably repeated for a plurality of different calibration poses of the robot arm for each calibration location, each calibration pose corresponding to certain angle values of the articulated joints of the robot arm. A sensitivity matrix is determined for each of the calibration poses during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration poses are used for updating the kinematic model of the robot. The number of calibration poses needed for calibrating the robot depends on the complexity of the robot and the robot arm, respectively. In simple robot arm configurations or in situations where only part of the kinematic model of the robot is to be updated, even one calibration location and one or two corresponding calibration poses may be sufficient to calibrate the robot. In other cases more (e.g. at least five, preferably at least ten) different robot poses are used for determining all kinematic parameters of the robot in order to obtain a complete and precise updated kinematic model. Often, the kinematic model of a conventional industrial robot comprises at least 30 characteristic parameters. Each calibration pose provides for six equations and, hence, for the determination of at the most six robot calibration parameters. In order to compensate the influence of noise or other disturbances, it is suggested to choose the number of calibration poses such that the overall number of equations which can be formed in the various poses is much larger than the number of calibration parameters to be determined for the kinematic model of a certain type of robot.
Preferably, the calibration method described for only one calibration location is repeated for a plurality of calibration locations. A sensitivity matrix is determined for each of the calibration locations during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration locations are used for updating the kinematic model of the robot. Furthermore, the method is preferably repeated for a plurality of different calibration poses of the robot arm for each calibration location, each calibration pose corresponding to certain angle values of the articulated joints of the robot arm. A sensitivity matrix is determined for each of the calibration poses during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration poses are used for updating the kinematic model of the robot. The number of calibration poses needed for calibrating the robot depends on the complexity of the robot and the robot arm, respectively. In simple robot arm configurations or in situations where only part of the kinematic model of the robot is to be updated, even one calibration location and one or two corresponding calibration poses may be sufficient to calibrate the robot. In other cases more (e.g. at least five, preferably at least ten) different robot poses are used for determining all kinematic parameters of the robot in order to obtain a complete and precise updated kinematic model. Often, the kinematic model of a conventional industrial robot comprises at least 30 characteristic parameters. Each calibration pose provides for six equations and, hence, for the determination of at the most six robot calibration parameters. In order to compensate the influence of noise or other disturbances, it is suggested to choose the number of calibration poses such that the overall number of equations which can be formed in the various poses is much larger than the number of calibration parameters to be determined for the kinematic model of a certain type of robot.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robot and calibration methods as taught by Hanson with the encoders taught by Logins. This would ensure highly effective control/calibration of the robotic system so that operation may be performed accurately despite changing elements and wear on components.
Regarding claim 9, where all the limitations of claim 8 are discussed above, Hanson further teaches:
9. (Original) The method of claim 8, wherein the first joint of the robot is coupled to a first member of the robot and the second joint of the robot is coupled to the first member of the robot. (Paragraph 0018, "In some embodiments, robot 102 may include a mounting base 110 and a rotating base member 112. Rotating base member 112 may rotate around a base axis 113 using a base rotation actuator 114. A lower arm member 116 may movably connect to rotating base member 112 around a lower pivot joint 117 using a lower pivot actuator 118. An upper arm member 120 may movably connect to lower arm member 116 around an upper pivot joint 121 using an upper pivot actuator 122. Upper arm member 120 may movably connect to a rotating wrist member 124 that rotates around a roll axis 125 of upper arm member 120 at wrist roll joint 126 using wrist roll actuator 127. Rotating wrist member 124 may movably connect to a pivoting wrist member 128 that pivots around a wrist pivot joint 129 of rotating wrist member 124 using a wrist pivot actuator (not shown). Pivoting wrist member 128 may movably connect to an end effector mount 132 that rotates around an end effector roll axis 133 of pivoting wrist member 128 using end effector roll actuator 134 at end effector roll joint 135. Robot 102 is merely an example and many other configurations of robotic arms or other robot configurations are possible including at least one positioning member (such as lower arm member 116, upper arm member 120, rotating wrist member 124, pivoting wrist member 128, and end effector mount 132), at least one joint connected to the at least one positioning member (such as lower pivot joint 117, upper pivot joint 121, wrist roll joint 126, wrist pivot joint 129, end effector roll joint 135), and at least one actuator for moving the at least one positioning member around the at least one joint (such as base rotation actuator 114, lower pivot actuator 118, upper pivot actuator 122, wrist roll actuator 127, wrist pivot actuator, end effector roll actuator 134). In some embodiments, actuators 114, 118, 122, 127, 134 may each include a servo motor, such as an electric servo motor with positioning mechanics and a control loop for controllably positioning their respective members around the respective joints, and the servo motor may be control be a control signal from controller 104.")
Regarding claim 12, where all the limitations of claim 8 are discussed above, Hanson further teaches:
12. (Currently Amended) The method of claim 8, wherein determining, based on the first image data, the actual joint position of the first joint and (Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again." This would occur when the offset amount is determined to be zero. The system would then proceed to determine relative positioning of the other joints to the first joint identified.) comprises:
determining, based on the first image data, that the first joint is aligned with the second joint
determining the actual joint position of the first joint and the second joint in response to determining that the first joint is aligned with the second joint. (Paragraph 0022, "In some embodiments, computing system 162 may include one or more application programs, data sources, and/or functional modules for calibrating robot 102 through controller 104. For example, computing system 162 may include a calibration module 170 and supporting data sources 172, 173, 174, 175 that support the functions, processes, and sub-modules of calibration module 170. Calibration module 170 may provide various processes for capturing a reference image and a calibration image, then processing those images to extract and compare position information for robot 102 to calculate one or more offsets for controller 104. Calibration module 170 may store, access, read, modify, or otherwise use data sources 172, 173, 174, 175. For example, reference images data source 172 may be used to store and retrieve one or more reference images related to robot 102 in one or more reference positions and captured at one or more reference times. Calibration images data source 173 may be used to store and retrieve one or more calibration images related to robot 102 in one or more calibration positions and captured at one or more calibration times. Calibration offsets data source 174 may be used to store and retrieve one or more calibration offsets for the actuators of robot 102 determined at one or more calibration times. Image compare algorithms 175 may include one or more image comparison algorithms, such as a three dimensional image comparison for calculating the relative positions of reference marks from image data, such as reference images and calibration images. One or more functions, processes, and sub-modules of calibration module 170 may interact with controller 104 and/or camera 163 via I/0 interface 168. Camera 163 may be selected from a variety of image data collectors, such as a digital camera or other image sensor, using visible light or another portion of the electromagnetic spectrum, such as infrared or ultraviolet imaging, to create an image that may be stored and processed as image data by calibration module 170.")
Regarding claim 13, where all the limitations of claim 12 are discussed above, Hanson further teaches:
13. (Currently Amended) The method of claim 12, wherein:
the first joint is coupled to a first member of the robot;
the second joint is coupled to a second member of the robot; (Paragraph 0018, “In some embodiments, robot 102 may include a mounting base 110 and a rotating base member 112. Rotating base member 112 may rotate around a base axis 113 using a base rotation actuator 114. A lower arm member 116 may movably connect to rotating base member 112 around a lower pivot joint 117 using a lower pivot actuator 118. An upper arm member 120 may movably connect to lower arm member 116 around an upper pivot joint 121 using an upper pivot actuator 122. Upper arm member 120 may movably connect to a rotating wrist member 124 that rotates around a roll axis 125 of upper arm member 120 at wrist roll joint 126 using wrist roll actuator 127. Rotating wrist member 124 may movably connect to a pivoting wrist member 128 that pivots around a wrist pivot joint 129 of rotating wrist member 124 using a wrist pivot actuator (not shown). Pivoting wrist member 128 may movably connect to an end effector mount 132 that rotates around an end effector roll axis 133 of pivoting wrist member 128 using end effector roll actuator 134 at end effector roll joint 135. Robot 102 is merely an example and many other configurations of robotic arms or other robot configurations are possible including at least one positioning member (such as lower arm member 116, upper arm member 120, rotating wrist member 124, pivoting wrist member 128, and end effector mount 132), at least one joint connected to the at least one positioning member (such as lower pivot joint 117, upper pivot joint 121, wrist roll joint 126, wrist pivot joint 129, end effector roll joint 135), and at least one actuator for moving the at least one positioning member around the at least one joint (such as base rotation actuator 114, lower pivot actuator 118, upper pivot actuator 122, wrist roll actuator 127, wrist pivot actuator, end effector roll actuator 134). In some embodiments, actuators 114, 118, 122, 127, 134 may each include a servo motor, such as an electric servo motor with positioning mechanics and a control loop for controllably positioning their respective members around the respective joints, and the servo motor may be control be a control signal from controller 104.” Please also see Figure 3.) and
…
Hanson does not specifically disclose the desired pose being such that the joints are aligned in a straight line. However, Alifragkis, in the same field of endeavor of robotic calibration, teaches:
… determining that the first joint is aligned with the second joint comprises determining that the first member member (Paragraph 0028, “In the present patent application the term "location" comprises a position (e.g. x, y, z in a Cartesian coordinate system) as well as an orientation (e.g. a, b, c around the x-, y-, z-axes) of the distal end of the robot arm. Another term "pose" is used for describing a certain status of the robot arm with the chain links and the joints being in certain positions, orientations and angles. Due to the high degree of freedom regarding the movement of the distal end of the robot arm of an industrial robot, it is possible that one and the same location of the distal end can be achieved with different poses of the robot arm.” As well as Paragraph 0045, “Preferably, the calibration method described for only one calibration location is repeated for a plurality of calibration locations. A sensitivity matrix is determined for each of the calibration locations during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration locations are used for updating the kinematic model of the robot. Furthermore, the method is preferably repeated for a plurality of different calibration poses of the robot arm for each calibration location, each calibration pose corresponding to certain angle values of the articulated joints of the robot arm. A sensitivity matrix is determined for each of the calibration poses during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration poses are used for updating the kinematic model of the robot. The number of calibration poses needed for calibrating the robot depends on the complexity of the robot and the robot arm, respectively. In simple robot arm configurations or in situations where only part of the kinematic model of the robot is to be updated, even one calibration location and one or two corresponding calibration poses may be sufficient to calibrate the robot. In other cases more (e.g. at least five, preferably at least ten) different robot poses are used for determining all kinematic parameters of the robot in order to obtain a complete and precise updated kinematic model. Often, the kinematic model of a conventional industrial robot comprises at least 30 characteristic parameters. Each calibration pose provides for six equations and, hence, for the determination of at the most six robot calibration parameters. In order to compensate the influence of noise or other disturbances, it is suggested to choose the number of calibration poses such that the overall number of equations which can be formed in the various poses is much larger than the number of calibration parameters to be determined for the kinematic model of a certain type of robot.” This demonstrates that a robot may be controlled to reach a plurality of specific poses (each with corresponding joint angles) where the relative positioning of each element is known during the calibration process. This suggests alignment of the joints at each pose as well as in a straight line when joint angles are 0 degrees or 180 degrees.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robot and calibration methods as taught by Hanson with the ability to move the robot into poses with known joint angles as taught by Alifragkis. This would ensure highly effective calibration of the robotic system so that operation may be performed accurately despite changing elements and wear on components.
Regarding claim 14, where all the limitations of claim 12 are discussed above, Hanson further teaches:
14. (Currently Amended) The method of claim 12, wherein determining, based on the first image data, that the first joint is aligned with the second joint
identifying a first visually identifiable feature represented in the first image data, (Paragraph 0024, "In some embodiments, calibration module 170 may include an image processor 186 that extracts position information, such as the relative positions of one or more members and actuators of robot 102, from image data, such as reference images in reference images data source 172 and calibration images in calibration images data source 173. For example, a three dimensional image comparison algorithm may identify each reference mark present in the target image and identify each reference mark as either a robot reference mark or an environment reference mark. Comparison of the environment reference marks may enable the image comparison to correct for differences in camera position and then each robot reference mark from the reference image may be compared to each corresponding robot reference mark in the calibration image and deviations between the two positions (reference position in the reference image and actual position in the calibration image). Image processor 186 may convert the deviations between the images into quantified physical differences that can be compensated for by applying offset values to one or more actuators of robot 102. In some embodiments, image processor 186 may include one or more image comparison algorithms, such as a three dimensional image comparison application with one or more image processing algorithms, stored in and retrieved from image compare algorithms data source 175. In some embodiments, image processor 186 may include an actual position calculator 187 that calculates the actual position of robot 102 from the calibration image using some or all of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 with reference to environment reference marks 154, 155, 156, 157. Image processor 186 may include an image comparison calculator 188 that compares the calibration image to the reference image using at least three of environment reference marks 154, 155, 156, 157 to compensate for any variations in camera position and then determines any variations in the positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 between the compared images. In some embodiments, image processor 186 includes an actuator offset calculator 189 that calculates physical differences and/or actuator offsets from the variations in positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 determined by image comparison calculator 188. For example, actuator offset calculator 189 may include a scaling function based on physical parameters of robot 102 and/or operating environment 150 and/or their respective reference marks. In some embodiments, image processor 186 may determine calibration offsets to be used by controller 104 and store them in calibration offsets data source 174.") wherein the first visually identifiable feature identifies the first joint; (Paragraph 0019, "Robot 102 may include a plurality of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 on its exterior surfaces that may be used for calibration operations. For example, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include visual indicia applied to the surface of robot 102 in desired locations that provide sufficient contrast or unique visual patterns that are easily detected within visual image data. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a circle or other shape with consistent dimensions and a high contrast color relative to the surface color of robot 102, such as a red circle on yellow or a black dot on white. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a distinctive pattern, such as a black dot in a white circle. Other visual indicia may be used as robot reference marks 140, 141, 142, 143, 144, 145, 146, 147. These indicia may be printed, painted, adhered, machined, or otherwise placed on robot 102 to mark the desired positions for visual reference. In the embodiment shown, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 are positioned on various positioning members of robot 102 relative to their respective joints to enable the position of each joint to be determined from visual image data. Each positioning member of interest may include one or more reference marks. For example, robot reference mark 140 on rotating base member 112 enables position information and calibration offsets to be calculated for base rotation actuator 114. Robot reference marks 141, 142 on lower arm member 116 enable position information and calibration offsets to be calculated for lower pivot actuator 118. Robot reference marks 143, 144 on upper arm member 120 enable position information and calibration offsets to be calculated for upper pivot actuator 122. Robot reference mark 145 on rotating wrist member 124 enables position information and calibration offsets to be calculated for wrist roll actuator 127. Robot reference mark 146 on pivoting wrist member 128 enables position information and calibration offsets to be calculated for the wrist pivot actuator. Robot reference mark 147 on end effector mount member 132 enables position information and calibration offsets to be calculated for end effector roll actuator 134. In some embodiments, the various members may include additional surfaces with different orientations and those additional surfaces may include additional robot reference marks to enable image data to be collected from additional orientations and fields of view. In some embodiments, two reference marks may be provided per joint or axis, but a single reference mark or greater than two reference marks are also possible where surface space and imaging and processing resources permit.")
identifying a second visually identifiable feature represented in the first image data, (Paragraph 0024, "In some embodiments, calibration module 170 may include an image processor 186 that extracts position information, such as the relative positions of one or more members and actuators of robot 102, from image data, such as reference images in reference images data source 172 and calibration images in calibration images data source 173. For example, a three dimensional image comparison algorithm may identify each reference mark present in the target image and identify each reference mark as either a robot reference mark or an environment reference mark. Comparison of the environment reference marks may enable the image comparison to correct for differences in camera position and then each robot reference mark from the reference image may be compared to each corresponding robot reference mark in the calibration image and deviations between the two positions (reference position in the reference image and actual position in the calibration image). Image processor 186 may convert the deviations between the images into quantified physical differences that can be compensated for by applying offset values to one or more actuators of robot 102. In some embodiments, image processor 186 may include one or more image comparison algorithms, such as a three dimensional image comparison application with one or more image processing algorithms, stored in and retrieved from image compare algorithms data source 175. In some embodiments, image processor 186 may include an actual position calculator 187 that calculates the actual position of robot 102 from the calibration image using some or all of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 with reference to environment reference marks 154, 155, 156, 157. Image processor 186 may include an image comparison calculator 188 that compares the calibration image to the reference image using at least three of environment reference marks 154, 155, 156, 157 to compensate for any variations in camera position and then determines any variations in the positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 between the compared images. In some embodiments, image processor 186 includes an actuator offset calculator 189 that calculates physical differences and/or actuator offsets from the variations in positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 determined by image comparison calculator 188. For example, actuator offset calculator 189 may include a scaling function based on physical parameters of robot 102 and/or operating environment 150 and/or their respective reference marks. In some embodiments, image processor 186 may determine calibration offsets to be used by controller 104 and store them in calibration offsets data source 174.") wherein the second visually identifiable feature identifies the second joint; (Paragraph 0019, "Robot 102 may include a plurality of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 on its exterior surfaces that may be used for calibration operations. For example, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include visual indicia applied to the surface of robot 102 in desired locations that provide sufficient contrast or unique visual patterns that are easily detected within visual image data. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a circle or other shape with consistent dimensions and a high contrast color relative to the surface color of robot 102, such as a red circle on yellow or a black dot on white. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a distinctive pattern, such as a black dot in a white circle. Other visual indicia may be used as robot reference marks 140, 141, 142, 143, 144, 145, 146, 147. These indicia may be printed, painted, adhered, machined, or otherwise placed on robot 102 to mark the desired positions for visual reference. In the embodiment shown, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 are positioned on various positioning members of robot 102 relative to their respective joints to enable the position of each joint to be determined from visual image data. Each positioning member of interest may include one or more reference marks. For example, robot reference mark 140 on rotating base member 112 enables position information and calibration offsets to be calculated for base rotation actuator 114. Robot reference marks 141, 142 on lower arm member 116 enable position information and calibration offsets to be calculated for lower pivot actuator 118. Robot reference marks 143, 144 on upper arm member 120 enable position information and calibration offsets to be calculated for upper pivot actuator 122. Robot reference mark 145 on rotating wrist member 124 enables position information and calibration offsets to be calculated for wrist roll actuator 127. Robot reference mark 146 on pivoting wrist member 128 enables position information and calibration offsets to be calculated for the wrist pivot actuator. Robot reference mark 147 on end effector mount member 132 enables position information and calibration offsets to be calculated for end effector roll actuator 134. In some embodiments, the various members may include additional surfaces with different orientations and those additional surfaces may include additional robot reference marks to enable image data to be collected from additional orientations and fields of view. In some embodiments, two reference marks may be provided per joint or axis, but a single reference mark or greater than two reference marks are also possible where surface space and imaging and processing resources permit.") and
determining that the first joint is aligned with the second joint when the first visually identifiable feature and the second visually identifiable feature have a particular spatial relationship. (Paragraph 0022, "In some embodiments, computing system 162 may include one or more application programs, data sources, and/or functional modules for calibrating robot 102 through controller 104. For example, computing system 162 may include a calibration module 170 and supporting data sources 172, 173, 174, 175 that support the functions, processes, and sub-modules of calibration module 170. Calibration module 170 may provide various processes for capturing a reference image and a calibration image, then processing those images to extract and compare position information for robot 102 to calculate one or more offsets for controller 104. Calibration module 170 may store, access, read, modify, or otherwise use data sources 172, 173, 174, 175. For example, reference images data source 172 may be used to store and retrieve one or more reference images related to robot 102 in one or more reference positions and captured at one or more reference times. Calibration images data source 173 may be used to store and retrieve one or more calibration images related to robot 102 in one or more calibration positions and captured at one or more calibration times. Calibration offsets data source 174 may be used to store and retrieve one or more calibration offsets for the actuators of robot 102 determined at one or more calibration times. Image compare algorithms 175 may include one or more image comparison algorithms, such as a three dimensional image comparison for calculating the relative positions of reference marks from image data, such as reference images and calibration images. One or more functions, processes, and sub-modules of calibration module 170 may interact with controller 104 and/or camera 163 via I/0 interface 168. Camera 163 may be selected from a variety of image data collectors, such as a digital camera or other image sensor, using visible light or another portion of the electromagnetic spectrum, such as infrared or ultraviolet imaging, to create an image that may be stored and processed as image data by calibration module 170.")
Regarding claim 16, where all the limitations of claim 1 are discussed above, Hanson further teaches:
16. (Currently Amended) The method of claim 1, wherein:
(Paragraph 0019, "Robot 102 may include a plurality of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 on its exterior surfaces that may be used for calibration operations. For example, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include visual indicia applied to the surface of robot 102 in desired locations that provide sufficient contrast or unique visual patterns that are easily detected within visual image data. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a circle or other shape with consistent dimensions and a high contrast color relative to the surface color of robot 102, such as a red circle on yellow or a black dot on white. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a distinctive pattern, such as a black dot in a white circle. Other visual indicia may be used as robot reference marks 140, 141, 142, 143, 144, 145, 146, 147. These indicia may be printed, painted, adhered, machined, or otherwise placed on robot 102 to mark the desired positions for visual reference. In the embodiment shown, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 are positioned on various positioning members of robot 102 relative to their respective joints to enable the position of each joint to be determined from visual image data. Each positioning member of interest may include one or more reference marks. For example, robot reference mark 140 on rotating base member 112 enables position information and calibration offsets to be calculated for base rotation actuator 114. Robot reference marks 141, 142 on lower arm member 116 enable position information and calibration offsets to be calculated for lower pivot actuator 118. Robot reference marks 143, 144 on upper arm member 120 enable position information and calibration offsets to be calculated for upper pivot actuator 122. Robot reference mark 145 on rotating wrist member 124 enables position information and calibration offsets to be calculated for wrist roll actuator 127. Robot reference mark 146 on pivoting wrist member 128 enables position information and calibration offsets to be calculated for the wrist pivot actuator. Robot reference mark 147 on end effector mount member 132 enables position information and calibration offsets to be calculated for end effector roll actuator 134. In some embodiments, the various members may include additional surfaces with different orientations and those additional surfaces may include additional robot reference marks to enable image data to be collected from additional orientations and fields of view. In some embodiments, two reference marks may be provided per joint or axis, but a single reference mark or greater than two reference marks are also possible where surface space and imaging and processing resources permit.")(Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again.")
at least one member of the robot couples the first joint to the second joint, (Paragraph 0018, "In some embodiments, robot 102 may include a mounting base 110 and a rotating base member 112. Rotating base member 112 may rotate around a base axis 113 using a base rotation actuator 114. A lower arm member 116 may movably connect to rotating base member 112 around a lower pivot joint 117 using a lower pivot actuator 118. An upper arm member 120 may movably connect to lower arm member 116 around an upper pivot joint 121 using an upper pivot actuator 122. Upper arm member 120 may movably connect to a rotating wrist member 124 that rotates around a roll axis 125 of upper arm member 120 at wrist roll joint 126 using wrist roll actuator 127. Rotating wrist member 124 may movably connect to a pivoting wrist member 128 that pivots around a wrist pivot joint 129 of rotating wrist member 124 using a wrist pivot actuator (not shown). Pivoting wrist member 128 may movably connect to an end effector mount 132 that rotates around an end effector roll axis 133 of pivoting wrist member 128 using end effector roll actuator 134 at end effector roll joint 135. Robot 102 is merely an example and many other configurations of robotic arms or other robot configurations are possible including at least one positioning member (such as lower arm member 116, upper arm member 120, rotating wrist member 124, pivoting wrist member 128, and end effector mount 132), at least one joint connected to the at least one positioning member (such as lower pivot joint 117, upper pivot joint 121, wrist roll joint 126, wrist pivot joint 129, end effector roll joint 135), and at least one actuator for moving the at least one positioning member around the at least one joint (such as base rotation actuator 114, lower pivot actuator 118, upper pivot actuator 122, wrist roll actuator 127, wrist pivot actuator, end effector roll actuator 134). In some embodiments, actuators 114, 118, 122, 127, 134 may each include a servo motor, such as an electric servo motor with positioning mechanics and a control loop for controllably positioning their respective members around the respective joints, and the servo motor may be control be a control signal from controller 104.") and
the method further comprises:
determining an actual joint position of the second joint using the actual joint position of the first joint. (Paragraph 0019, "Robot 102 may include a plurality of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 on its exterior surfaces that may be used for calibration operations. For example, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include visual indicia applied to the surface of robot 102 in desired locations that provide sufficient contrast or unique visual patterns that are easily detected within visual image data. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a circle or other shape with consistent dimensions and a high contrast color relative to the surface color of robot 102, such as a red circle on yellow or a black dot on white. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a distinctive pattern, such as a black dot in a white circle. Other visual indicia may be used as robot reference marks 140, 141, 142, 143, 144, 145, 146, 147. These indicia may be printed, painted, adhered, machined, or otherwise placed on robot 102 to mark the desired positions for visual reference. In the embodiment shown, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 are positioned on various positioning members of robot 102 relative to their respective joints to enable the position of each joint to be determined from visual image data. Each positioning member of interest may include one or more reference marks. For example, robot reference mark 140 on rotating base member 112 enables position information and calibration offsets to be calculated for base rotation actuator 114. Robot reference marks 141, 142 on lower arm member 116 enable position information and calibration offsets to be calculated for lower pivot actuator 118. Robot reference marks 143, 144 on upper arm member 120 enable position information and calibration offsets to be calculated for upper pivot actuator 122. Robot reference mark 145 on rotating wrist member 124 enables position information and calibration offsets to be calculated for wrist roll actuator 127. Robot reference mark 146 on pivoting wrist member 128 enables position information and calibration offsets to be calculated for the wrist pivot actuator. Robot reference mark 147 on end effector mount member 132 enables position information and calibration offsets to be calculated for end effector roll actuator 134. In some embodiments, the various members may include additional surfaces with different orientations and those additional surfaces may include additional robot reference marks to enable image data to be collected from additional orientations and fields of view. In some embodiments, two reference marks may be provided per joint or axis, but a single reference mark or greater than two reference marks are also possible where surface space and imaging and processing resources permit.")
Regarding claim 26, Hanson further teaches:
26. (Currently Amended) A robot comprising:
a set of members;
a set of joints coupling the set of members; (Paragraph 0018, “In some embodiments, robot 102 may include a mounting base 110 and a rotating base member 112. Rotating base member 112 may rotate around a base axis 113 using a base rotation actuator 114. A lower arm member 116 may movably connect to rotating base member 112 around a lower pivot joint 117 using a lower pivot actuator 118. An upper arm member 120 may movably connect to lower arm member 116 around an upper pivot joint 121 using an upper pivot actuator 122. Upper arm member 120 may movably connect to a rotating wrist member 124 that rotates around a roll axis 125 of upper arm member 120 at wrist roll joint 126 using wrist roll actuator 127. Rotating wrist member 124 may movably connect to a pivoting wrist member 128 that pivots around a wrist pivot joint 129 of rotating wrist member 124 using a wrist pivot actuator (not shown). Pivoting wrist member 128 may movably connect to an end effector mount 132 that rotates around an end effector roll axis 133 of pivoting wrist member 128 using end effector roll actuator 134 at end effector roll joint 135. Robot 102 is merely an example and many other configurations of robotic arms or other robot configurations are possible including at least one positioning member (such as lower arm member 116, upper arm member 120, rotating wrist member 124, pivoting wrist member 128, and end effector mount 132), at least one joint connected to the at least one positioning member (such as lower pivot joint 117, upper pivot joint 121, wrist roll joint 126, wrist pivot joint 129, end effector roll joint 135), and at least one actuator for moving the at least one positioning member around the at least one joint (such as base rotation actuator 114, lower pivot actuator 118, upper pivot actuator 122, wrist roll actuator 127, wrist pivot actuator, end effector roll actuator 134). In some embodiments, actuators 114, 118, 122, 127, 134 may each include a servo motor, such as an electric servo motor with positioning mechanics and a control loop for controllably positioning their respective members around the respective joints, and the servo motor may be control be a control signal from controller 104.”) and
a controller (Paragraph 0021, “In some embodiments, a calibration unit 160 may include a computing system 162. Computing system 162 may provide a system and user interface for calculating calibration offsets for robot 102 and controller 104 using visual data from a camera 163. In some embodiments, computing system 162 is a general purpose computing devices, such as a personal computer, work station, mobile device, or an embedded system in an industrial control system (using general purpose computing components and operating systems). In some embodiments, computing system 162 may be a specialized data processing system for the task of calibrating system 100. Computing system 162 may include at least one memory 164, processor 166, and input/output (I/O) interface 168 interconnected by a bus (not shown). Further, computing system 162 may include communication with external I/O device/resources and/or storage systems, including connected system, such controller 104, camera 163, and network resources. In general, processor 166 executes computer program code, such as calibration module 170, that is stored in memory 164 and/or a storage system. While executing computer program code, processor 166 can read and/or write data to/from memory 164, storage systems, and I/O devices (through I/O interface 168). The bus provides a communication link between each of the components within computing system 162. I/O devices may comprise any device that enables a user to interact with computing system 162 (e.g., keyboard, pointing device, display, etc.). Computing system 162 is only representative of various possible combinations of hardware and software. For example, the processor may comprise a single processing unit, or be distributed across one or more processing units in one or more locations, e.g., on a client and server. Similarly, memory and/or storage systems may reside at one or more physical locations. Memory and/or storage systems can comprise any combination of various types of non-transitory computer readable storage medium including magnetic media, optical media, random access memory (RAM), read only memory (ROM), etc. In some embodiments, computing system 162 is a laptop computer in communication with controller 104 via a wired (serial, USB, Ethernet, etc.) or wireless (802.11, Bluetooth, etc.) connection and running application software for calibrating system 100. In some embodiments, some or all of the functions of computing system 162 may be on board robot 102 using an integrated computing system, such as an on board control module, with or without wireless communication to one or more user interfaces and/or remote data storage.”) configured to:
receive first image data from at least one sensor when the robot is in a first pose,
wherein a first joint and a second joint of the set of joints represented in the first image data has an expected joint position … when the robot is in the first pose;
determine, based on the first image data, an actual joint position of the first joint and the second joint when the robot is in the first pose; (Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again.") and
configure … of a first actuator associated with the first joint (Paragraph 0018, "In some embodiments, robot 102 may include a mounting base 110 and a rotating base member 112. Rotating base member 112 may rotate around a base axis 113 using a base rotation actuator 114. A lower arm member 116 may movably connect to rotating base member 112 around a lower pivot joint 117 using a lower pivot actuator 118. An upper arm member 120 may movably connect to lower arm member 116 around an upper pivot joint 121 using an upper pivot actuator 122. Upper arm member 120 may movably connect to a rotating wrist member 124 that rotates around a roll axis 125 of upper arm member 120 at wrist roll joint 126 using wrist roll actuator 127. Rotating wrist member 124 may movably connect to a pivoting wrist member 128 that pivots around a wrist pivot joint 129 of rotating wrist member 124 using a wrist pivot actuator (not shown). Pivoting wrist member 128 may movably connect to an end effector mount 132 that rotates around an end effector roll axis 133 of pivoting wrist member 128 using end effector roll actuator 134 at end effector roll joint 135. Robot 102 is merely an example and many other configurations of robotic arms or other robot configurations are possible including at least one positioning member (such as lower arm member 116, upper arm member 120, rotating wrist member 124, pivoting wrist member 128, and end effector mount 132), at least one joint connected to the at least one positioning member (such as lower pivot joint 117, upper pivot joint 121, wrist roll joint 126, wrist pivot joint 129, end effector roll joint 135), and at least one actuator for moving the at least one positioning member around the at least one joint (such as base rotation actuator 114, lower pivot actuator 118, upper pivot actuator 122, wrist roll actuator 127, wrist pivot actuator, end effector roll actuator 134). In some embodiments, actuators 114, 118, 122, 127, 134 may each include a servo motor, such as an electric servo motor with positioning mechanics and a control loop for controllably positioning their respective members around the respective joints, and the servo motor may be control be a control signal from controller 104.") based, at least in part, on the actual joint position of the first joint and/or the second joint (Paragraphs 0024-0025, "In some embodiments, calibration module 170 may include an image processor 186 that extracts position information, such as the relative positions of one or more members and actuators of robot 102, from image data, such as reference images in reference images data source 172 and calibration images in calibration images data source 173. For example, a three dimensional image comparison algorithm may identify each reference mark present in the target image and identify each reference mark as either a robot reference mark or an environment reference mark. Comparison of the environment reference marks may enable the image comparison to correct for differences in camera position and then each robot reference mark from the reference image may be compared to each corresponding robot reference mark in the calibration image and deviations between the two positions (reference position in the reference image and actual position in the calibration image). Image processor 186 may convert the deviations between the images into quantified physical differences that can be compensated for by applying offset values to one or more actuators of robot 102. In some embodiments, image processor 186 may include one or more image comparison algorithms, such as a three dimensional image comparison application with one or more image processing algorithms, stored in and retrieved from image compare algorithms data source 175. In some embodiments, image processor 186 may include an actual position calculator 187 that calculates the actual position of robot 102 from the calibration image using some or all of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 with reference to environment reference marks 154, 155, 156, 157. Image processor 186 may include an image comparison calculator 188 that compares the calibration image to the reference image using at least three of environment reference marks 154, 155, 156, 157 to compensate for any variations in camera position and then determines any variations in the positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 between the compared images. In some embodiments, image processor 186 includes an actuator offset calculator 189 that calculates physical differences and/or actuator offsets from the variations in positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 determined by image comparison calculator 188. For example, actuator offset calculator 189 may include a scaling function based on physical parameters of robot 102 and/or operating environment 150 and/or their respective reference marks. In some embodiments, image processor 186 may determine calibration offsets to be used by controller 104 and store them in calibration offsets data source 174.
In some embodiments, calibration module 170 may include a positioning interface 192 that communicates calibration offsets to controller 104 for use by positioning controller 105. For example, positioning interface 192 may be invoked after a calibration is complete and send the most recent calibration offsets from calibration offsets data source 174 to controller 104 via I/O interface 168 to be stored in actuator offsets 108 and used by positioning controller 105 to control the actuators of robot 102. In some embodiments, positioning interface 192 enables controller 104 to query calibration module 170 and/or the stored actuator offsets in calibration offsets data source 174 to receive updated calibration values on demand.") when it is determined that the actual joint position of the first joint and the second joint is different from the expected joint position when the robot is in the first pose (Paragraph 0022, "In some embodiments, computing system 162 may include one or more application programs, data sources, and/or functional modules for calibrating robot 102 through controller 104. For example, computing system 162 may include a calibration module 170 and supporting data sources 172, 173, 174, 175 that support the functions, processes, and sub-modules of calibration module 170. Calibration module 170 may provide various processes for capturing a reference image and a calibration image, then processing those images to extract and compare position information for robot 102 to calculate one or more offsets for controller 104. Calibration module 170 may store, access, read, modify, or otherwise use data sources 172, 173, 174, 175. For example, reference images data source 172 may be used to store and retrieve one or more reference images related to robot 102 in one or more reference positions and captured at one or more reference times. Calibration images data source 173 may be used to store and retrieve one or more calibration images related to robot 102 in one or more calibration positions and captured at one or more calibration times. Calibration offsets data source 174 may be used to store and retrieve one or more calibration offsets for the actuators of robot 102 determined at one or more calibration times. Image compare algorithms 175 may include one or more image comparison algorithms, such as a three dimensional image comparison for calculating the relative positions of reference marks from image data, such as reference images and calibration images. One or more functions, processes, and sub-modules of calibration module 170 may interact with controller 104 and/or camera 163 via I/0 interface 168. Camera 163 may be selected from a variety of image data collectors, such as a digital camera or other image sensor, using visible light or another portion of the electromagnetic spectrum, such as infrared or ultraviolet imaging, to create an image that may be stored and processed as image data by calibration module 170.")
Hanson does not specifically disclose the desired pose being such that the joints are aligned or that each servomotor includes an encoder. However, Alifragkis, in the same field of endeavor of robotic calibration, teaches:
… in which the first joint and the second joint are aligned … (Paragraph 0028, “In the present patent application the term "location" comprises a position (e.g. x, y, z in a Cartesian coordinate system) as well as an orientation (e.g. a, b, c around the x-, y-, z-axes) of the distal end of the robot arm. Another term "pose" is used for describing a certain status of the robot arm with the chain links and the joints being in certain positions, orientations and angles. Due to the high degree of freedom regarding the movement of the distal end of the robot arm of an industrial robot, it is possible that one and the same location of the distal end can be achieved with different poses of the robot arm.” As well as Paragraph 0045, “Preferably, the calibration method described for only one calibration location is repeated for a plurality of calibration locations. A sensitivity matrix is determined for each of the calibration locations during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration locations are used for updating the kinematic model of the robot. Furthermore, the method is preferably repeated for a plurality of different calibration poses of the robot arm for each calibration location, each calibration pose corresponding to certain angle values of the articulated joints of the robot arm. A sensitivity matrix is determined for each of the calibration poses during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration poses are used for updating the kinematic model of the robot. The number of calibration poses needed for calibrating the robot depends on the complexity of the robot and the robot arm, respectively. In simple robot arm configurations or in situations where only part of the kinematic model of the robot is to be updated, even one calibration location and one or two corresponding calibration poses may be sufficient to calibrate the robot. In other cases more (e.g. at least five, preferably at least ten) different robot poses are used for determining all kinematic parameters of the robot in order to obtain a complete and precise updated kinematic model. Often, the kinematic model of a conventional industrial robot comprises at least 30 characteristic parameters. Each calibration pose provides for six equations and, hence, for the determination of at the most six robot calibration parameters. In order to compensate the influence of noise or other disturbances, it is suggested to choose the number of calibration poses such that the overall number of equations which can be formed in the various poses is much larger than the number of calibration parameters to be determined for the kinematic model of a certain type of robot.” This demonstrates that a robot may be controlled to reach a plurality of specific poses (each with corresponding joint angles) where the relative positioning of each element is known during the calibration process. This suggests alignment of the joints at each pose as well as in a straight line when joint angles are 0 degrees or 180 degrees.)
However, Logins, in the same field of endeavor of robotic calibration, teaches:
… a first encoder … (Page 3 Paragraphs 2-3, “Preferably, the calibration method described for only one calibration location is repeated for a plurality of calibration locations. A sensitivity matrix is determined for each of the calibration locations during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration locations are used for updating the kinematic model of the robot. Furthermore, the method is preferably repeated for a plurality of different calibration poses of the robot arm for each calibration location, each calibration pose corresponding to certain angle values of the articulated joints of the robot arm. A sensitivity matrix is determined for each of the calibration poses during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration poses are used for updating the kinematic model of the robot. The number of calibration poses needed for calibrating the robot depends on the complexity of the robot and the robot arm, respectively. In simple robot arm configurations or in situations where only part of the kinematic model of the robot is to be updated, even one calibration location and one or two corresponding calibration poses may be sufficient to calibrate the robot. In other cases more (e.g. at least five, preferably at least ten) different robot poses are used for determining all kinematic parameters of the robot in order to obtain a complete and precise updated kinematic model. Often, the kinematic model of a conventional industrial robot comprises at least 30 characteristic parameters. Each calibration pose provides for six equations and, hence, for the determination of at the most six robot calibration parameters. In order to compensate the influence of noise or other disturbances, it is suggested to choose the number of calibration poses such that the overall number of equations which can be formed in the various poses is much larger than the number of calibration parameters to be determined for the kinematic model of a certain type of robot.
Preferably, the calibration method described for only one calibration location is repeated for a plurality of calibration locations. A sensitivity matrix is determined for each of the calibration locations during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration locations are used for updating the kinematic model of the robot. Furthermore, the method is preferably repeated for a plurality of different calibration poses of the robot arm for each calibration location, each calibration pose corresponding to certain angle values of the articulated joints of the robot arm. A sensitivity matrix is determined for each of the calibration poses during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration poses are used for updating the kinematic model of the robot. The number of calibration poses needed for calibrating the robot depends on the complexity of the robot and the robot arm, respectively. In simple robot arm configurations or in situations where only part of the kinematic model of the robot is to be updated, even one calibration location and one or two corresponding calibration poses may be sufficient to calibrate the robot. In other cases more (e.g. at least five, preferably at least ten) different robot poses are used for determining all kinematic parameters of the robot in order to obtain a complete and precise updated kinematic model. Often, the kinematic model of a conventional industrial robot comprises at least 30 characteristic parameters. Each calibration pose provides for six equations and, hence, for the determination of at the most six robot calibration parameters. In order to compensate the influence of noise or other disturbances, it is suggested to choose the number of calibration poses such that the overall number of equations which can be formed in the various poses is much larger than the number of calibration parameters to be determined for the kinematic model of a certain type of robot.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robot and calibration methods as taught by Hanson with the ability to move the robot into poses with known joint angles as taught by Alifragkis as well as with the encoders taught by Logins. This would ensure highly effective calibration of the robotic system so that operation may be performed accurately despite changing elements and wear on components.
Regarding claim 51, Hanson further teaches:
51. (Currently Amended) A controller for a robot, the controller configured to:
receive first image data from at least one sensor when the robot is in a first pose, wherein a first joint of the robot and a second joint of the robot [[is]] represented in the first image data has an expected joint position … when the robot is in the first pose;
determine, based on the first image data, an actual joint position of the first joint and the second joint when the robot is in the first pose; (Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again.") and
configure … of a first actuator associated with the first joint (Paragraph 0018, "In some embodiments, robot 102 may include a mounting base 110 and a rotating base member 112. Rotating base member 112 may rotate around a base axis 113 using a base rotation actuator 114. A lower arm member 116 may movably connect to rotating base member 112 around a lower pivot joint 117 using a lower pivot actuator 118. An upper arm member 120 may movably connect to lower arm member 116 around an upper pivot joint 121 using an upper pivot actuator 122. Upper arm member 120 may movably connect to a rotating wrist member 124 that rotates around a roll axis 125 of upper arm member 120 at wrist roll joint 126 using wrist roll actuator 127. Rotating wrist member 124 may movably connect to a pivoting wrist member 128 that pivots around a wrist pivot joint 129 of rotating wrist member 124 using a wrist pivot actuator (not shown). Pivoting wrist member 128 may movably connect to an end effector mount 132 that rotates around an end effector roll axis 133 of pivoting wrist member 128 using end effector roll actuator 134 at end effector roll joint 135. Robot 102 is merely an example and many other configurations of robotic arms or other robot configurations are possible including at least one positioning member (such as lower arm member 116, upper arm member 120, rotating wrist member 124, pivoting wrist member 128, and end effector mount 132), at least one joint connected to the at least one positioning member (such as lower pivot joint 117, upper pivot joint 121, wrist roll joint 126, wrist pivot joint 129, end effector roll joint 135), and at least one actuator for moving the at least one positioning member around the at least one joint (such as base rotation actuator 114, lower pivot actuator 118, upper pivot actuator 122, wrist roll actuator 127, wrist pivot actuator, end effector roll actuator 134). In some embodiments, actuators 114, 118, 122, 127, 134 may each include a servo motor, such as an electric servo motor with positioning mechanics and a control loop for controllably positioning their respective members around the respective joints, and the servo motor may be control be a control signal from controller 104.") based, at least in part, on the actual joint position of the first joint and/or the second joint (Paragraphs 0024-0025, "In some embodiments, calibration module 170 may include an image processor 186 that extracts position information, such as the relative positions of one or more members and actuators of robot 102, from image data, such as reference images in reference images data source 172 and calibration images in calibration images data source 173. For example, a three dimensional image comparison algorithm may identify each reference mark present in the target image and identify each reference mark as either a robot reference mark or an environment reference mark. Comparison of the environment reference marks may enable the image comparison to correct for differences in camera position and then each robot reference mark from the reference image may be compared to each corresponding robot reference mark in the calibration image and deviations between the two positions (reference position in the reference image and actual position in the calibration image). Image processor 186 may convert the deviations between the images into quantified physical differences that can be compensated for by applying offset values to one or more actuators of robot 102. In some embodiments, image processor 186 may include one or more image comparison algorithms, such as a three dimensional image comparison application with one or more image processing algorithms, stored in and retrieved from image compare algorithms data source 175. In some embodiments, image processor 186 may include an actual position calculator 187 that calculates the actual position of robot 102 from the calibration image using some or all of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 with reference to environment reference marks 154, 155, 156, 157. Image processor 186 may include an image comparison calculator 188 that compares the calibration image to the reference image using at least three of environment reference marks 154, 155, 156, 157 to compensate for any variations in camera position and then determines any variations in the positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 between the compared images. In some embodiments, image processor 186 includes an actuator offset calculator 189 that calculates physical differences and/or actuator offsets from the variations in positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 determined by image comparison calculator 188. For example, actuator offset calculator 189 may include a scaling function based on physical parameters of robot 102 and/or operating environment 150 and/or their respective reference marks. In some embodiments, image processor 186 may determine calibration offsets to be used by controller 104 and store them in calibration offsets data source 174.
In some embodiments, calibration module 170 may include a positioning interface 192 that communicates calibration offsets to controller 104 for use by positioning controller 105. For example, positioning interface 192 may be invoked after a calibration is complete and send the most recent calibration offsets from calibration offsets data source 174 to controller 104 via I/O interface 168 to be stored in actuator offsets 108 and used by positioning controller 105 to control the actuators of robot 102. In some embodiments, positioning interface 192 enables controller 104 to query calibration module 170 and/or the stored actuator offsets in calibration offsets data source 174 to receive updated calibration values on demand.") when it is determined that the actual joint position of the first joint and the second joint is different from the expected joint position when the robot is in the first pose (Paragraph 0022, "In some embodiments, computing system 162 may include one or more application programs, data sources, and/or functional modules for calibrating robot 102 through controller 104. For example, computing system 162 may include a calibration module 170 and supporting data sources 172, 173, 174, 175 that support the functions, processes, and sub-modules of calibration module 170. Calibration module 170 may provide various processes for capturing a reference image and a calibration image, then processing those images to extract and compare position information for robot 102 to calculate one or more offsets for controller 104. Calibration module 170 may store, access, read, modify, or otherwise use data sources 172, 173, 174, 175. For example, reference images data source 172 may be used to store and retrieve one or more reference images related to robot 102 in one or more reference positions and captured at one or more reference times. Calibration images data source 173 may be used to store and retrieve one or more calibration images related to robot 102 in one or more calibration positions and captured at one or more calibration times. Calibration offsets data source 174 may be used to store and retrieve one or more calibration offsets for the actuators of robot 102 determined at one or more calibration times. Image compare algorithms 175 may include one or more image comparison algorithms, such as a three dimensional image comparison for calculating the relative positions of reference marks from image data, such as reference images and calibration images. One or more functions, processes, and sub-modules of calibration module 170 may interact with controller 104 and/or camera 163 via I/0 interface 168. Camera 163 may be selected from a variety of image data collectors, such as a digital camera or other image sensor, using visible light or another portion of the electromagnetic spectrum, such as infrared or ultraviolet imaging, to create an image that may be stored and processed as image data by calibration module 170.")
Hanson does not specifically disclose the desired pose being such that the joints are aligned or that each servomotor includes an encoder. However, Alifragkis, in the same field of endeavor of robotic calibration, teaches:
… in which the first joint and the second joint are aligned … (Paragraph 0028, “In the present patent application the term "location" comprises a position (e.g. x, y, z in a Cartesian coordinate system) as well as an orientation (e.g. a, b, c around the x-, y-, z-axes) of the distal end of the robot arm. Another term "pose" is used for describing a certain status of the robot arm with the chain links and the joints being in certain positions, orientations and angles. Due to the high degree of freedom regarding the movement of the distal end of the robot arm of an industrial robot, it is possible that one and the same location of the distal end can be achieved with different poses of the robot arm.” As well as Paragraph 0045, “Preferably, the calibration method described for only one calibration location is repeated for a plurality of calibration locations. A sensitivity matrix is determined for each of the calibration locations during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration locations are used for updating the kinematic model of the robot. Furthermore, the method is preferably repeated for a plurality of different calibration poses of the robot arm for each calibration location, each calibration pose corresponding to certain angle values of the articulated joints of the robot arm. A sensitivity matrix is determined for each of the calibration poses during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration poses are used for updating the kinematic model of the robot. The number of calibration poses needed for calibrating the robot depends on the complexity of the robot and the robot arm, respectively. In simple robot arm configurations or in situations where only part of the kinematic model of the robot is to be updated, even one calibration location and one or two corresponding calibration poses may be sufficient to calibrate the robot. In other cases more (e.g. at least five, preferably at least ten) different robot poses are used for determining all kinematic parameters of the robot in order to obtain a complete and precise updated kinematic model. Often, the kinematic model of a conventional industrial robot comprises at least 30 characteristic parameters. Each calibration pose provides for six equations and, hence, for the determination of at the most six robot calibration parameters. In order to compensate the influence of noise or other disturbances, it is suggested to choose the number of calibration poses such that the overall number of equations which can be formed in the various poses is much larger than the number of calibration parameters to be determined for the kinematic model of a certain type of robot.” This demonstrates that a robot may be controlled to reach a plurality of specific poses (each with corresponding joint angles) where the relative positioning of each element is known during the calibration process. This suggests alignment of the joints at each pose as well as in a straight line when joint angles are 0 degrees or 180 degrees.)
However, Logins, in the same field of endeavor of robotic calibration, teaches:
… a first encoder … (Page 3 Paragraphs 2-3, “Preferably, the calibration method described for only one calibration location is repeated for a plurality of calibration locations. A sensitivity matrix is determined for each of the calibration locations during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration locations are used for updating the kinematic model of the robot. Furthermore, the method is preferably repeated for a plurality of different calibration poses of the robot arm for each calibration location, each calibration pose corresponding to certain angle values of the articulated joints of the robot arm. A sensitivity matrix is determined for each of the calibration poses during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration poses are used for updating the kinematic model of the robot. The number of calibration poses needed for calibrating the robot depends on the complexity of the robot and the robot arm, respectively. In simple robot arm configurations or in situations where only part of the kinematic model of the robot is to be updated, even one calibration location and one or two corresponding calibration poses may be sufficient to calibrate the robot. In other cases more (e.g. at least five, preferably at least ten) different robot poses are used for determining all kinematic parameters of the robot in order to obtain a complete and precise updated kinematic model. Often, the kinematic model of a conventional industrial robot comprises at least 30 characteristic parameters. Each calibration pose provides for six equations and, hence, for the determination of at the most six robot calibration parameters. In order to compensate the influence of noise or other disturbances, it is suggested to choose the number of calibration poses such that the overall number of equations which can be formed in the various poses is much larger than the number of calibration parameters to be determined for the kinematic model of a certain type of robot.
Preferably, the calibration method described for only one calibration location is repeated for a plurality of calibration locations. A sensitivity matrix is determined for each of the calibration locations during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration locations are used for updating the kinematic model of the robot. Furthermore, the method is preferably repeated for a plurality of different calibration poses of the robot arm for each calibration location, each calibration pose corresponding to certain angle values of the articulated joints of the robot arm. A sensitivity matrix is determined for each of the calibration poses during the previous state of the robot. The differences of the characteristic parameters determined after the iterative closed-loop process for each of the calibration poses are used for updating the kinematic model of the robot. The number of calibration poses needed for calibrating the robot depends on the complexity of the robot and the robot arm, respectively. In simple robot arm configurations or in situations where only part of the kinematic model of the robot is to be updated, even one calibration location and one or two corresponding calibration poses may be sufficient to calibrate the robot. In other cases more (e.g. at least five, preferably at least ten) different robot poses are used for determining all kinematic parameters of the robot in order to obtain a complete and precise updated kinematic model. Often, the kinematic model of a conventional industrial robot comprises at least 30 characteristic parameters. Each calibration pose provides for six equations and, hence, for the determination of at the most six robot calibration parameters. In order to compensate the influence of noise or other disturbances, it is suggested to choose the number of calibration poses such that the overall number of equations which can be formed in the various poses is much larger than the number of calibration parameters to be determined for the kinematic model of a certain type of robot.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robot and calibration methods as taught by Hanson with the ability to move the robot into poses with known joint angles as taught by Alifragkis as well as with the encoders taught by Logins. This would ensure highly effective calibration of the robotic system so that operation may be performed accurately despite changing elements and wear on components.
Claim(s) 4-5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hanson in view of Alifragkis and Logins and in further view of Christopher et al. (US 20230419546 A1), hereinafter Christopher.
Regarding claim 4, where all the limitations of claim 1 are discussed above, Hanson further teaches:
4. (Currently Amended) The method of claim 1, wherein the first image data comprises three-dimensional (3D) image data and determining, based on the first image data, the actual joint position of the first joint and the second joint when the robot is in the first pose comprises:
performing object detection to identify the first joint represented in the 3D image data;
determining, using the 3D image data, (Paragraph 0024, "In some embodiments, calibration module 170 may include an image processor 186 that extracts position information, such as the relative positions of one or more members and actuators of robot 102, from image data, such as reference images in reference images data source 172 and calibration images in calibration images data source 173. For example, a three dimensional image comparison algorithm may identify each reference mark present in the target image and identify each reference mark as either a robot reference mark or an environment reference mark. Comparison of the environment reference marks may enable the image comparison to correct for differences in camera position and then each robot reference mark from the reference image may be compared to each corresponding robot reference mark in the calibration image and deviations between the two positions (reference position in the reference image and actual position in the calibration image). Image processor 186 may convert the deviations between the images into quantified physical differences that can be compensated for by applying offset values to one or more actuators of robot 102. In some embodiments, image processor 186 may include one or more image comparison algorithms, such as a three dimensional image comparison application with one or more image processing algorithms, stored in and retrieved from image compare algorithms data source 175. In some embodiments, image processor 186 may include an actual position calculator 187 that calculates the actual position of robot 102 from the calibration image using some or all of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 with reference to environment reference marks 154, 155, 156, 157. Image processor 186 may include an image comparison calculator 188 that compares the calibration image to the reference image using at least three of environment reference marks 154, 155, 156, 157 to compensate for any variations in camera position and then determines any variations in the positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 between the compared images. In some embodiments, image processor 186 includes an actuator offset calculator 189 that calculates physical differences and/or actuator offsets from the variations in positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 determined by image comparison calculator 188. For example, actuator offset calculator 189 may include a scaling function based on physical parameters of robot 102 and/or operating environment 150 and/or their respective reference marks. In some embodiments, image processor 186 may determine calibration offsets to be used by controller 104 and store them in calibration offsets data source 174.") … and
determining the actual joint position of the first joint when the robot is in the first pose (Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again.") …
Hanson does not specifically discuss using the three-dimensional image data to calculate a distance between the sensor and the joints. However, Christopher, in the same field of endeavor of robotics, teaches:
… a distance from the at least one sensor to the first joint; … based on the distance from the at least one sensor to the first joint. (Paragraph 0029, "As discussed further herein, the one or more sensors associated with the robot may include multiple (e.g., at least two) cameras with at least partially overlapping fields of view, and the multiple cameras may be configured to capture images of the environment of the robot. The multiple cameras may include, for example, a visual camera configured to capture color (e.g., red-blue-green (RGB)) images of the environment and a depth camera (e.g., a stereo camera) configured to capture distance information from the camera to points in the environment. The images captured by the multiple cameras may be used to generate a three-dimensional representation of objects in the environment of the robot. The three-dimensional representation may be used to facilitate localization and/or navigation within the environment to, for instance, execute a mission. Occasionally (e.g., once a day, once a month), each of the multiple cameras may be calibrated using a calibration routine to ensure that the information included in the images captured from each of the cameras is spatially aligned to facilitate generation of an accurate three-dimensional representation of the robot's environment.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and calibration methods as taught by Hanson with the functionality of identifying the distance between the sensor and the joints using the three-dimensional image data as taught by Christopher. This would ensure a highly accurate representation of the environment is known and allow the system to carefully calibrate for operation of the robot with fewer errors and a higher level of safety.
Regarding claim 5, where all the limitations of claim 4 are discussed above, Hanson further teaches:
5. (Original) The method of claim 4, wherein performing object detection to identify the first joint represented in the 3D image data (Paragraph 0024, "In some embodiments, calibration module 170 may include an image processor 186 that extracts position information, such as the relative positions of one or more members and actuators of robot 102, from image data, such as reference images in reference images data source 172 and calibration images in calibration images data source 173. For example, a three dimensional image comparison algorithm may identify each reference mark present in the target image and identify each reference mark as either a robot reference mark or an environment reference mark. Comparison of the environment reference marks may enable the image comparison to correct for differences in camera position and then each robot reference mark from the reference image may be compared to each corresponding robot reference mark in the calibration image and deviations between the two positions (reference position in the reference image and actual position in the calibration image). Image processor 186 may convert the deviations between the images into quantified physical differences that can be compensated for by applying offset values to one or more actuators of robot 102. In some embodiments, image processor 186 may include one or more image comparison algorithms, such as a three dimensional image comparison application with one or more image processing algorithms, stored in and retrieved from image compare algorithms data source 175. In some embodiments, image processor 186 may include an actual position calculator 187 that calculates the actual position of robot 102 from the calibration image using some or all of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 with reference to environment reference marks 154, 155, 156, 157. Image processor 186 may include an image comparison calculator 188 that compares the calibration image to the reference image using at least three of environment reference marks 154, 155, 156, 157 to compensate for any variations in camera position and then determines any variations in the positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 between the compared images. In some embodiments, image processor 186 includes an actuator offset calculator 189 that calculates physical differences and/or actuator offsets from the variations in positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 determined by image comparison calculator 188. For example, actuator offset calculator 189 may include a scaling function based on physical parameters of robot 102 and/or operating environment 150 and/or their respective reference marks. In some embodiments, image processor 186 may determine calibration offsets to be used by controller 104 and store them in calibration offsets data source 174.") comprises:
identifying a visually-identifiable feature on the robot, (Paragraph 0019, "Robot 102 may include a plurality of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 on its exterior surfaces that may be used for calibration operations. For example, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include visual indicia applied to the surface of robot 102 in desired locations that provide sufficient contrast or unique visual patterns that are easily detected within visual image data. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a circle or other shape with consistent dimensions and a high contrast color relative to the surface color of robot 102, such as a red circle on yellow or a black dot on white. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a distinctive pattern, such as a black dot in a white circle. Other visual indicia may be used as robot reference marks 140, 141, 142, 143, 144, 145, 146, 147. These indicia may be printed, painted, adhered, machined, or otherwise placed on robot 102 to mark the desired positions for visual reference. In the embodiment shown, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 are positioned on various positioning members of robot 102 relative to their respective joints to enable the position of each joint to be determined from visual image data. Each positioning member of interest may include one or more reference marks. For example, robot reference mark 140 on rotating base member 112 enables position information and calibration offsets to be calculated for base rotation actuator 114. Robot reference marks 141, 142 on lower arm member 116 enable position information and calibration offsets to be calculated for lower pivot actuator 118. Robot reference marks 143, 144 on upper arm member 120 enable position information and calibration offsets to be calculated for upper pivot actuator 122. Robot reference mark 145 on rotating wrist member 124 enables position information and calibration offsets to be calculated for wrist roll actuator 127. Robot reference mark 146 on pivoting wrist member 128 enables position information and calibration offsets to be calculated for the wrist pivot actuator. Robot reference mark 147 on end effector mount member 132 enables position information and calibration offsets to be calculated for end effector roll actuator 134. In some embodiments, the various members may include additional surfaces with different orientations and those additional surfaces may include additional robot reference marks to enable image data to be collected from additional orientations and fields of view. In some embodiments, two reference marks may be provided per joint or axis, but a single reference mark or greater than two reference marks are also possible where surface space and imaging and processing resources permit.") wherein the visually-identifiable feature is represented in the 3D image data and identifies the first joint. (Paragraph 0024, "In some embodiments, calibration module 170 may include an image processor 186 that extracts position information, such as the relative positions of one or more members and actuators of robot 102, from image data, such as reference images in reference images data source 172 and calibration images in calibration images data source 173. For example, a three dimensional image comparison algorithm may identify each reference mark present in the target image and identify each reference mark as either a robot reference mark or an environment reference mark. Comparison of the environment reference marks may enable the image comparison to correct for differences in camera position and then each robot reference mark from the reference image may be compared to each corresponding robot reference mark in the calibration image and deviations between the two positions (reference position in the reference image and actual position in the calibration image). Image processor 186 may convert the deviations between the images into quantified physical differences that can be compensated for by applying offset values to one or more actuators of robot 102. In some embodiments, image processor 186 may include one or more image comparison algorithms, such as a three dimensional image comparison application with one or more image processing algorithms, stored in and retrieved from image compare algorithms data source 175. In some embodiments, image processor 186 may include an actual position calculator 187 that calculates the actual position of robot 102 from the calibration image using some or all of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 with reference to environment reference marks 154, 155, 156, 157. Image processor 186 may include an image comparison calculator 188 that compares the calibration image to the reference image using at least three of environment reference marks 154, 155, 156, 157 to compensate for any variations in camera position and then determines any variations in the positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 between the compared images. In some embodiments, image processor 186 includes an actuator offset calculator 189 that calculates physical differences and/or actuator offsets from the variations in positions of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 determined by image comparison calculator 188. For example, actuator offset calculator 189 may include a scaling function based on physical parameters of robot 102 and/or operating environment 150 and/or their respective reference marks. In some embodiments, image processor 186 may determine calibration offsets to be used by controller 104 and store them in calibration offsets data source 174.")
Claim(s) 17, 19-21, and 24-25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hanson in view of Alifragkis and Logins and in further view of Poelman et al. (US 20220147026 A1), hereinafter Poelman.
Regarding claim 17, where all the limitations of claim 1 are discussed above, Hanson further teaches:
17. (Currently Amended) The method of claim 1, wherein:
(Paragraph 0019, "Robot 102 may include a plurality of robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 on its exterior surfaces that may be used for calibration operations. For example, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include visual indicia applied to the surface of robot 102 in desired locations that provide sufficient contrast or unique visual patterns that are easily detected within visual image data. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a circle or other shape with consistent dimensions and a high contrast color relative to the surface color of robot 102, such as a red circle on yellow or a black dot on white. In some embodiments, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 may include a distinctive pattern, such as a black dot in a white circle. Other visual indicia may be used as robot reference marks 140, 141, 142, 143, 144, 145, 146, 147. These indicia may be printed, painted, adhered, machined, or otherwise placed on robot 102 to mark the desired positions for visual reference. In the embodiment shown, robot reference marks 140, 141, 142, 143, 144, 145, 146, 147 are positioned on various positioning members of robot 102 relative to their respective joints to enable the position of each joint to be determined from visual image data. Each positioning member of interest may include one or more reference marks. For example, robot reference mark 140 on rotating base member 112 enables position information and calibration offsets to be calculated for base rotation actuator 114. Robot reference marks 141, 142 on lower arm member 116 enable position information and calibration offsets to be calculated for lower pivot actuator 118. Robot reference marks 143, 144 on upper arm member 120 enable position information and calibration offsets to be calculated for upper pivot actuator 122. Robot reference mark 145 on rotating wrist member 124 enables position information and calibration offsets to be calculated for wrist roll actuator 127. Robot reference mark 146 on pivoting wrist member 128 enables position information and calibration offsets to be calculated for the wrist pivot actuator. Robot reference mark 147 on end effector mount member 132 enables position information and calibration offsets to be calculated for end effector roll actuator 134. In some embodiments, the various members may include additional surfaces with different orientations and those additional surfaces may include additional robot reference marks to enable image data to be collected from additional orientations and fields of view. In some embodiments, two reference marks may be provided per joint or axis, but a single reference mark or greater than two reference marks are also possible where surface space and imaging and processing resources permit.") (Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again.")
at least one member of the robot couples the first joint to the second joint, (Paragraph 0018, "In some embodiments, robot 102 may include a mounting base 110 and a rotating base member 112. Rotating base member 112 may rotate around a base axis 113 using a base rotation actuator 114. A lower arm member 116 may movably connect to rotating base member 112 around a lower pivot joint 117 using a lower pivot actuator 118. An upper arm member 120 may movably connect to lower arm member 116 around an upper pivot joint 121 using an upper pivot actuator 122. Upper arm member 120 may movably connect to a rotating wrist member 124 that rotates around a roll axis 125 of upper arm member 120 at wrist roll joint 126 using wrist roll actuator 127. Rotating wrist member 124 may movably connect to a pivoting wrist member 128 that pivots around a wrist pivot joint 129 of rotating wrist member 124 using a wrist pivot actuator (not shown). Pivoting wrist member 128 may movably connect to an end effector mount 132 that rotates around an end effector roll axis 133 of pivoting wrist member 128 using end effector roll actuator 134 at end effector roll joint 135. Robot 102 is merely an example and many other configurations of robotic arms or other robot configurations are possible including at least one positioning member (such as lower arm member 116, upper arm member 120, rotating wrist member 124, pivoting wrist member 128, and end effector mount 132), at least one joint connected to the at least one positioning member (such as lower pivot joint 117, upper pivot joint 121, wrist roll joint 126, wrist pivot joint 129, end effector roll joint 135), and at least one actuator for moving the at least one positioning member around the at least one joint (such as base rotation actuator 114, lower pivot actuator 118, upper pivot actuator 122, wrist roll actuator 127, wrist pivot actuator, end effector roll actuator 134). In some embodiments, actuators 114, 118, 122, 127, 134 may each include a servo motor, such as an electric servo motor with positioning mechanics and a control loop for controllably positioning their respective members around the respective joints, and the servo motor may be control be a control signal from controller 104.") and
…
Hanson does not specifically discuss determining proper calibration by determining the actual position matches the anticipated position. However, Poelman, in the same field of endeavor of robotics, teaches:
… the method further comprises:
determining that a second encoder of a second actuator associated with the second joint is properly calibrated (Paragraph 0088, "At block 440, the process determines whether the observed real-world situation is different from the expectation based on the holistic conception of the robotic cell, that is whether there are discrepancies in the robotic arm position between the predicted position and the observed position. In one embodiment, the system compares the real world pose and movement pattern (ex. approach vector for each joint) of the robotic arm, tool, and/or work piece to the expectation based on the holistic 3D conception of the robotic system. If the discrepancy is within an acceptable threshold, the current step in the process is executed, at block 450. At block 460, the process determines whether the routine has been completed. If not, the process returns to block 430 and continues observing the position/movement of components. Note that while this is illustrated as a flowchart in the real world the monitoring is continuous, and in one embodiment an interrupt is triggered when a difference is detected between the expectation and the reality.") when it is determined that the actual joint position of the first joint matches the expected joint position of the first joint. (Paragraph 0112, "At block 675, the process determines whether the elements are sufficiently accurate to meet an accuracy threshold. In one embodiment, the system may refine the calibrations multiple times. In one embodiment, the system may provide an initial calibration, and then test whether the quality thresholds are met, and refine as necessary. The test whether the calibration, pose estimates, and errors are within the parameters may be performed after any of the steps above, as well as here.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and methods of calibration as taught by Hanson with the ability to determine that the calibration has met a threshold of accuracy as taught by Poelman. This would ensure that the system calibrates efficiently while still maintaining a high level of accuracy.
Regarding claim 19, where all the limitations of claim 1 are discussed above, Hanson further teaches:
19. (Currently Amended) The method of claim 1, wherein configuring the first encoder based, at least in part, on the actual joint position of the first joint and/or the second joint when it is determined that the actual joint position of the first joint and the second joint is different from the expected joint position when the robot is in the first pose (Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again.") comprises:
after configuring the first encoder, receiving second image data from the at least one sensor when the robot is in a second pose, (Paragraphs 0022-0023, "In some embodiments, computing system 162 may include one or more application programs, data sources, and/or functional modules for calibrating robot 102 through controller 104. For example, computing system 162 may include a calibration module 170 and supporting data sources 172, 173, 174, 175 that support the functions, processes, and sub-modules of calibration module 170. Calibration module 170 may provide various processes for capturing a reference image and a calibration image, then processing those images to extract and compare position information for robot 102 to calculate one or more offsets for controller 104. Calibration module 170 may store, access, read, modify, or otherwise use data sources 172, 173, 174, 175. For example, reference images data source 172 may be used to store and retrieve one or more reference images related to robot 102 in one or more reference positions and captured at one or more reference times. Calibration images data source 173 may be used to store and retrieve one or more calibration images related to robot 102 in one or more calibration positions and captured at one or more calibration times. Calibration offsets data source 174 may be used to store and retrieve one or more calibration offsets for the actuators of robot 102 determined at one or more calibration times. Image compare algorithms 175 may include one or more image comparison algorithms, such as a three dimensional image comparison for calculating the relative positions of reference marks from image data, such as reference images and calibration images. One or more functions, processes, and sub-modules of calibration module 170 may interact with controller 104 and/or camera 163 via I/0 interface 168. Camera 163 may be selected from a variety of image data collectors, such as a digital camera or other image sensor, using visible light or another portion of the electromagnetic spectrum, such as infrared or ultraviolet imaging, to create an image that may be stored and processed as image data by calibration module 170.
In some embodiments, calibration module 170 may include a reference position process 178 that communicates a command to robot 102 and/or controller 104 to drive the actuators to a defined reference position. For example, reference position process 178 may issue a command to controller 104 to return robot 102 to the home position. In some embodiments, reference position process 178 may include a plurality of selectable reference positions for different calibration options. In some embodiments, calibration module 170 may include a reference image capture process 180 that captures or retrieves a reference image of robot 102 in a reference position for use in subsequent calibrations. For example, reference image capture process 180 may issue a command to camera 163 to capture an image of robot 102. In some embodiments, reference image capture process 180 may determine whether a previously captured reference image is available in reference images data source 172 and, if so, it may retrieve the stored reference image for use in the subsequent calibration, if not, it may use camera 163 to capture a new reference image of robot 102 and store it in reference images data source 172. In some embodiments, processes 178, 180 are executed proximate the time robot 102 is put into service and/or mechanically calibrated and validated to assure that the reference image or images accurately reflect a baseline image of the desired calibrated state of robot 102. In some embodiments, calibration module 170 may include a calibration position process 182 that communicates a command to robot 102 and/or controller 104 to drive the actuators to a defined calibration position. For example, calibration position process 182 may issue a command to controller 104 to return robot 102 to the home position. In some embodiments, calibration position process 128 may include a plurality of selectable calibration positions for different calibration options, each corresponding to an available reference position with a corresponding reference image in reference images data source 172. In some embodiments, calibration module 170 may include a calibration image capture process 184 that captures or retrieves a calibration image of robot 102 in a calibration position for use in the present calibration. For example, calibration image capture process 184 may issue a command to camera 163 to capture an image of robot 102. In some embodiments, calibration image capture process 180 may store each calibration image in calibration images data source 173. In some embodiments, processes 182, 184 are executed each time a new calibration is conducted.") wherein the first joint of the robot represented in the second image data has an expected joint position when the robot is in the second pose;
determining, based on the second image data, an actual joint position of the first joint when the robot is in the second pose; (Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again.") and
…
Hanson does not specifically discuss determining proper calibration by determining the actual position matches the anticipated position. However, Poelman, in the same field of endeavor of robotics, teaches:
… determining that the first encoder is properly calibrated (Paragraph 0088, "At block 440, the process determines whether the observed real-world situation is different from the expectation based on the holistic conception of the robotic cell, that is whether there are discrepancies in the robotic arm position between the predicted position and the observed position. In one embodiment, the system compares the real world pose and movement pattern (ex. approach vector for each joint) of the robotic arm, tool, and/or work piece to the expectation based on the holistic 3D conception of the robotic system. If the discrepancy is within an acceptable threshold, the current step in the process is executed, at block 450. At block 460, the process determines whether the routine has been completed. If not, the process returns to block 430 and continues observing the position/movement of components. Note that while this is illustrated as a flowchart in the real world the monitoring is continuous, and in one embodiment an interrupt is triggered when a difference is detected between the expectation and the reality.") when it is determined that the actual joint position of the first joint when the robot is in the second pose matches the expected joint position of the first joint when the robot is in the second pose. (Paragraph 0112, "At block 675, the process determines whether the elements are sufficiently accurate to meet an accuracy threshold. In one embodiment, the system may refine the calibrations multiple times. In one embodiment, the system may provide an initial calibration, and then test whether the quality thresholds are met, and refine as necessary. The test whether the calibration, pose estimates, and errors are within the parameters may be performed after any of the steps above, as well as here.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and methods of calibration as taught by Hanson with the ability to determine that the calibration has met a threshold of accuracy as taught by Poelman. This would ensure that the system calibrates efficiently while still maintaining a high level of accuracy.
Regarding claim 20, where all the limitations of claim 1 are discussed above, Hanson does not specifically teach the sensor being coupled to the robot. However, Poelman, in the same field of endeavor of robotics, teaches:
20. (Original) The method of claim 1, wherein the at least one sensor includes a sensor coupled to the robot. (Paragraph 0030, "The robotic cell system also includes a plurality of cameras, in one embodiment. At a minimum, the robotic cell includes one camera. In one embodiment, cameras are rigidly mounted to the most rigid portion of the frame, on the top, side, and other positions. In another embodiment, the cameras may be mounted in other position(s). In one embodiment, a camera may be mounted at end-of-arm. In one embodiment, the field of view of each camera can observe a minimum set of fiducials in a volumetric layout. In one embodiment, all positions and locations within the work area are observable by at least two cameras. The cameras are calibrated, in one embodiment, to determine distortion models. In one embodiment, the system utilizes a set of cameras to provide the holistic view of the workspace. In one embodiment, the data from the cameras is integrated to provide a stereoscopic view or multi-camera view to provide a 3D space view, rather than individual 2D images. In another embodiment, a monocular camera maybe used for the workspace.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and calibration methods as taught by Hanson with the sensor coupled to the arm of the robot as taught by Poelman. This would allow the system to utilize the sensor for a variety of applications along with the calibration methods.
Regarding claim 21, where all the limitations of claim 20 are discussed above, Hanson further teaches:
21. (Original) The method of claim 20, wherein the at least one sensor is included in a vision system of the robot. (Paragraph 0021, "In some embodiments, a calibration unit 160 may include a computing system 162. Computing system 162 may provide a system and user interface for calculating calibration offsets for robot 102 and controller 104 using visual data from a camera 163. In some embodiments, computing system 162 is a general purpose computing devices, such as a personal computer, work station, mobile device, or an embedded system in an industrial control system (using general purpose computing components and operating systems). In some embodiments, computing system 162 may be a specialized data processing system for the task of calibrating system 100. Computing system 162 may include at least one memory 164, processor 166, and input/output (I/O) interface 168 interconnected by a bus (not shown). Further, computing system 162 may include communication with external I/O device/resources and/or storage systems, including connected system, such controller 104, camera 163, and network resources. In general, processor 166 executes computer program code, such as calibration module 170, that is stored in memory 164 and/or a storage system. While executing computer program code, processor 166 can read and/or write data to/from memory 164, storage systems, and I/O devices (through I/O interface 168). The bus provides a communication link between each of the components within computing system 162. I/O devices may comprise any device that enables a user to interact with computing system 162 (e.g., keyboard, pointing device, display, etc.). Computing system 162 is only representative of various possible combinations of hardware and software. For example, the processor may comprise a single processing unit, or be distributed across one or more processing units in one or more locations, e.g., on a client and server. Similarly, memory and/or storage systems may reside at one or more physical locations. Memory and/or storage systems can comprise any combination of various types of non-transitory computer readable storage medium including magnetic media, optical media, random access memory (RAM), read only memory (ROM), etc. In some embodiments, computing system 162 is a laptop computer in communication with controller 104 via a wired (serial, USB, Ethernet, etc.) or wireless (802.11, Bluetooth, etc.) connection and running application software for calibrating system 100. In some embodiments, some or all of the functions of computing system 162 may be on board robot 102 using an integrated computing system, such as an on board control module, with or without wireless communication to one or more user interfaces and/or remote data storage.")
Regarding claim 24, where all the limitations of claim 1 are discussed above, Hanson further teaches:
24. (Original) The method of claim 1, further comprising:
outputting an indication that the robot may be operated safely (Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again.") …
Hanson does not specifically discuss determining proper calibration by determining the actual position matches the anticipated position. However, Poelman, in the same field of endeavor of robotics, teaches:
… when it is determined that the actual joint position matches the expected joint position. (Paragraph 0112, "At block 675, the process determines whether the elements are sufficiently accurate to meet an accuracy threshold. In one embodiment, the system may refine the calibrations multiple times. In one embodiment, the system may provide an initial calibration, and then test whether the quality thresholds are met, and refine as necessary. The test whether the calibration, pose estimates, and errors are within the parameters may be performed after any of the steps above, as well as here.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and methods of calibration as taught by Hanson with the ability to determine that the calibration has met a threshold of accuracy as taught by Poelman. This would ensure that the system calibrates efficiently while still maintaining a high level of accuracy.
Regarding claim 25, where all the limitations of claim 24 are discussed above, Hanson further teaches:
25. (Original) The method of claim 24, wherein outputting the indication that the robot may be operated safely comprises:
controlling the robot to perform a first task. (Paragraph 0032, "In process 430, the robot is operated in accordance with its industrial function until such time as a calibration is called for. For example, the robot may be subject to periodic calibrations in accordance with a maintenance schedule, continue operation until an error is detected in its operations or resulting work product, and/or continue operation until there is a risk event, such as a crash, that suggests the need for a recalibration. In process 432, the robot is positioned in a calibration position that corresponds to a reference position that is available in a reference image. For example, the robot may be commanded to return to a home position or another known configuration corresponding to an available reference position. In process 434, a camera may be positioned in or adjacent the operating environment in such a way that at least one robot reference mark and at least three environment reference marks are in the field of view of the camera. For example, the camera may be positioned in a position similar to that used to capture the reference image, though it may not be necessary to be positioned in the identical position. In process 436, a calibration image may be captured by the camera. For example, one or more digital images of the robot and operating environment may be captured. In process 438, an actual position may be determined for the robot in the calibration image. For example, the image processor of a calibration system may identify the position of each environment reference mark and the position of each robot reference mark within the calibration image. In process 440, calibration offsets may be determined for one or more actuators of the robot. For example, the reference image and the calibration image may be compared and normalized to one another based upon the environment reference marks and deviations between corresponding robot reference marks. These deviations may then be scaled to determine a calibration offset value for the effected actuators. In process 442, a position controller for the robot is calibrated based on the calibration offsets. For example, calibration offset values from the calibration system may be communicated through an interface to the position controller for the robot and those calibration offsets will be used when operating the robot going forward. Method 400 may proceed back to process 430 and operate the robot until the calibration is triggered again.")
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hanson in view of Alifragkis and Logins and in further view of Faraji et al. (US 20230110248 A1), hereinafter Faraji.
Regarding claim 18, where all the limitations of claim 1 are discussed above, Hanson does not specifically teach performing calibration as a part of start-up of the system. However, Faraji, in the same field of endeavor of robotics, teaches:
18. (Original) The method of claim 1, wherein the method is performed as part of a start-up operation of the robot. (Paragraph 0112, "As discussed above, the example stereoscopic visualization camera 106 is configured to provide high-resolution stereoscopic video images of a target surgical site at different magnifications. As part of the stereoscopic robotic platform 302, the stereoscopic visualization camera 106 operates in connection with the robotic arm 114 and/or the coupling plate 306 for precise and clear changes to image focus, working distance, magnification, etc. To accomplish the image acquisition flexibility, the stereoscopic robotic platform 302 is configured to operate one or more calibration, initialization, and/or reset routines. In some embodiments, the stereoscopic visualization camera 106, the robotic arm 114, the coupling plate 306, or more generally, the stereoscopic robotic platform 302 is calibrated during manufacture and/or after installation. Calibration of the camera 106 with the robotic arm 114 provides positioning information of the camera 106 relative to the robotic arm 114 and operator space. After power-up of the stereoscopic robotic platform 302, in some embodiments, the camera 106 and/or the robotic arm 114 is configured to perform further calibration/initialization to measure and verify a location and orientation of the camera 106 at that time.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and calibration methods as taught by Hanson with the calibration upon start-up as taught by Faraji. This would ensure that the system is properly calibrated prior to any attempt at operation and would avoid uncalibrated operation which may cause damage to the robot or environments.
Conclusion
The Examiner has cited particular paragraphs or columns and line numbers in the referencesapplied to the claims above for the convenience of the Applicant. Although the specified citations arerepresentative of the teachings of the art and are applied to specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested of the Applicant in preparing responses, to fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner. See MPEP 2141.02 [R-07.2015] VI. A prior art reference must be considered in its entirety, i.e., as a whole, including portions that would lead away from the claimed Invention. W.L. Gore & Associates, Inc. v. Garlock, Inc., 721 F.2d 1540, 220 USPQ 303 (Fed. Cir. 1983), cert, denied, 469 U.S. 851 (1984). See also MPEP §2123.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
/H.J.K./Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights