DETAILED ACTION
This is a Final Office Action on the Merits in response to communications filed by applicant on March 19th, 2026. Claims 1-20 are currently pending and examined below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendments to the Claims, filed on March 19th, 2026, have been entered. Claims 1-3, 5-13, 15-17, and 19-20 are currently amended and pending, and claims 4, 14, and 18 are original, unamended, and pending. The amendments to the Claims have overcome each and every objection set forth in the previous Non-Final Office action mailed September 22nd, 2025. The amendments to the Drawings, filed on March 19th, 2026, have been entered and have overcome each and every objection set forth in the previous Non-Final Office Action mailed September 22nd, 2025. The amendments to the Specifications, filed on March 19th, 2026, have been entered and have overcome each and every objection set forth in the previous Non-Final Office Action mailed September 22nd, 2025.
Information Disclosure Statement
The Information Disclosure Statement(s) filed on 03/19/2026 is/are being considered by the examiner.
Claim Objections
Claim 9 objected to because of the following informalities:
In Claim 9 lines 1-2, there appears to be a typographical error regarding the phrase “wherein the robot arm is coupled o the weldhead”. It is suggested that the phrase be corrected to “wherein the robot arm is coupled to the weldhead” for the purpose of improving clarity.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3, 4, 7, 11, 12, 14, and 17-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2009/0118864 A1 ("Eldridge") in view of US 20110029132 A1 ("Nemmers") in further view of US 2021/0065356 A1 ("Fisher").
Regarding claim 1, Eldridge teaches a computer-implemented method for calibrating a tool center point (TCP) of a robotic welding system performed by a controller of the robotic welding system, the method comprising (Eldridge: ¶ 0006, “An embodiment of the present invention may further comprise a computerized method for calculating a tool frame tool center point relative to a wrist-frame of a robot for a tool attached at a wrist of the robot using a camera…”, ¶ 0033, “The robot manipulator is typically made up of two subsections, the body and arm 108 and the wrist 110. A tool 112 used by a robot 102 to perform desired tasks is typically attached at the wrist 110 of the robot manipulator 102.”, ¶ 0102, “FIGS. 9A-C show images 900,910,920 of example Metal-Inert Gas (MIG) welding torches. FIG. 9A is an example image of a first type 900 of a MIG welding torch tool. FIG. 9B is an example image of a second type 910 of a MIG welding torch tool. FIG. 9C is an example image of a third type 920 of a MIG welding torch tool.”):
receiving, with the controller, an image from one or more sensors (Eldridge: ¶ 0100, “The camera may be used to continuously capture an image of the target tool in real-time. Embodiments may store an image and/or images at desired times to perform calculations based on the stored image and/or images.”, ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. The system clearly receives an image captured by one or more sensors.);
identifying, with the controller, based on an image, a location of a terminal end of a protrusion extending from the weldhead (Eldridge: ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. As can be seen from the cited passage, the method is configured to determine the location of the TCP which can be the tip of the welding wire that protrudes from the weldhead.),
each image including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference (Eldridge: Figures 10A-C and 11A-C, ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. As can be seen from the cited passage and figures, the images clearly include at least a portion of the protrusion extending from the weldhead and that the first frame of refence is associated with a tip of the weldhead.);
performing one or more robotic operations with the robot arm based on the one or more TCP calibration values (Eldridge: ¶ 0134, “The tool calibration software was implemented on a separate computer, with communication to the robot controller occurring over standard communication link. With the aid of some short programs written for the robot, the calibration software was able to command the robot to move to the positions required.”. The system is clearly controlled based on the TCP calibration.).
Eldridge does not teach receiving, with the controller, multiple image from one or more sensors disposed on a robot arm of the robotic welding system;
identifying, with the controller, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead,
each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference,
determining, with the controller, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference;
generating, with the controller, one or more TCP calibration values based on the second frame of reference
Nemmers, in the same field of endeavor, teaches receiving, with the controller, multiple image from one or more sensors (Nemmers: ¶ 0058, “In step 108, at least two images of the robotic tool are generated and analyzed to determine a tool center point, (TCP). Specifically, the tool is rotated to a first view position (e.g. -45 degree position) and a first image is generated, as shown in step 110. Four pre-determined targets located on the robotic tool are "found" by the system 10 and four offsets are obtained and stored (e.g. in the storage device 30 or vision registers). In step 112, the robotic tool is rotated 90 degrees to a second position (e.g. +45 degree position). In step 114, a second image of the robotic tool is generated based on the second position.”, ¶ 0064, “As more clearly shown in FIG. 10, while the robotic tool is positioned within the field of view of the image generating device, a portion of the robotic tool (i.e. parent tool) is initially located, as identified by an outline 205. In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11.”, ¶ 0065, “The robotic tool is then rotated and the process is repeated to re-locate the points 206, 208, 210, 212 in the second image of the robotic tool. Specifically, the first three dimensional line passing through the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 can be intersected with a second three dimensional line defined from the second image.”. The cited passages clearly show that the method taught in Nemmers clearly involve receiving multiple images from one or more sensors of the robotic welding system.);
identifying, with the controller, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead (Nemmers: ¶ 0058, “In step 108, at least two images of the robotic tool are generated and analyzed to determine a tool center point, (TCP). Specifically, the tool is rotated to a first view position (e.g. -45 degree position) and a first image is generated, as shown in step 110. Four pre-determined targets located on the robotic tool are "found" by the system 10 and four offsets are obtained and stored (e.g. in the storage device 30 or vision registers). In step 112, the robotic tool is rotated 90 degrees to a second position (e.g. +45 degree position). In step 114, a second image of the robotic tool is generated based on the second position.”. The cited passage clearly shows that the system captures multiple images and uses these images to determine the location of the terminal end of a protrusion extending from the weldhead (i.e. the TCP).),
each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference (Nemmers: ¶ 0058, “In step 108, at least two images of the robotic tool are generated and analyzed to determine a tool center point, (TCP). Specifically, the tool is rotated to a first view position (e.g. -45 degree position) and a first image is generated, as shown in step 110. Four pre-determined targets located on the robotic tool are "found" by the system 10 and four offsets are obtained and stored (e.g. in the storage device 30 or vision registers). In step 112, the robotic tool is rotated 90 degrees to a second position (e.g. +45 degree position). In step 114, a second image of the robotic tool is generated based on the second position.”. One of ordinary skill in the art would see that the multiple images would each include at least a portion of the protrusion extending from a tip of the weldhead.)
determining, with the controller, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference (Nemmers: Figured 10-12, ¶ 0066, “As more clearly shown, the point 206 and the point 208 can be connected using a three dimensional reference line R1. A tool center point 214 based upon the wire 202 can be created from the direction of the reference line R1, the origin point 206 and a pre-defined distance ( e.g. 17 mm) associated with the robotic tool. The process is repeated for the wire 204 using the points 210, 212 to generate reference line R2 and tool center point 216. A user tool frame can be calculated from the data points defined above. Specifically, a reference line R3 through the tool center points 214, 216 defines the X direction. The cross product of the reference line R1 and the reference line R3 defines the XZ plane. Since the origin point, the X axis, and the XZ plane are known, there is enough information to calculate the entire user tool (frame), as appreciate by one skilled in the art. It is understood that a similar process is performed for determining the user tool for the wire 204. It is further understood that a third user tool can be created by setting the origin point as a centroid along a reference line R4 based on the user tool for the wire 202 and the user tool for the wire 204, wherein the orientation is the same as the user tool for the wire 202.”. The cited passages and figures clearly show that the reference points that define the tip of the welding wire (points 214 and 216) can be used to define an origin point that is offset from the first reference points and lie on the intersection between reference lines R3 and R4.),
generating, with the controller, one or more TCP calibration values based on the second frame of reference (Nemmers: ¶ 0061, “In step 120, a user tool for a particular tool is updated. In certain embodiments the user tool is updated based upon a measure user tool process that measures the location of a pre-defined portion of the robotic tool and creates a current (Internal) user tool. The current user tools are compared with the nominal user tools to determine any changes or offsets. The system 10 tracks changes in X, Y, Z, W, P, R, distance, and tilt elements, wherein the changes are represented by delta values. Distance is defined as the length of a three dimension line that connects a pre-defined point on the current user tool with the reciprocal point on the nominal tool. Tilt is defined as a combination of the Yaw and Pitch and represents an angle change with respect to the Z axis. It is understood that the distance and tilt measurements are easier to conceptualize than conventional XYZWPR coordinates. It is further understood that the delta values can be further analyzed to indicate undesirable changes in the robotic tool orientation.”, ¶ 0066, “It is further understood that a third user tool can be created by setting the origin point as a centroid along a reference line R4 based on the user tool for the wire 202 and the user tool for the wire 204, wherein the orientation is the same as the user tool for the wire 202.”. The cited passages clearly show that the user tool values determined based on the second reference point are used in to generate TCP calibration values.).
Eldridge teaches a method for calibrating a tool center point (TCP) of a robotic welding system performed by a controller of the robotic system, the method comprising: receiving, with the controller, image from one or more sensors; identifying, with the controller, based image, a location of a terminal end of a protrusion extending from the weldhead, each image including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference; performing one or more robotic operations with the robot arm based on the one or more TCP calibration values. Eldridge does not teach receiving, with the controller, multiple image from one or more sensors; identifying, with the controller, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determining, with the controller, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference, generating, with the controller, one or more TCP calibration values based on the second frame of reference. Nemmers teaches receiving, with the controller, multiple image from one or more sensors; identifying, with the controller, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determining, with the controller, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference, generating, with the controller, one or more TCP calibration values based on the second frame of reference. A person of ordinary skill in the art would have had the technological capabilities required to have modified the method taught in Eldridge with receiving, with the controller, multiple image from one or more sensors; identifying, with the controller, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determining, with the controller, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference, generating, with the controller, one or more TCP calibration values based on the second frame of reference taught in Nemmers. Furthermore, the method taught in Eldridge is already configured to determine the TCP calibration values based on a reference point associated with the terminal end of a protrusion extending form the weldhead using an image, so modifying this method to use multiple images, to define a second reference point, and use this second refence point to determine the TCP calibration values as taught in Nemmers would not change or introduce new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a method for calibrating a tool center point (TCP) of a robotic welding system performed by a controller of the robotic system, the method comprising: receiving, with the controller, multiple image from one or more sensors; identifying, with the controller, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determining, with the controller, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference, generating, with the controller, one or more TCP calibration values based on the second frame of reference.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method for calibrating a tool center point (TCP) of a robotic welding system taught in Eldridge with receiving, with the controller, multiple image from one or more sensors; identifying, with the controller, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determining, with the controller, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference, generating, with the controller, one or more TCP calibration values based on the second frame of reference taught in Nemmers with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Eldridge in view of Nemmers does not teach receiving, with the controller, multiple image from one or more sensors disposed on a robot arm of the robotic welding system;
Fisher, in the same field of endeavor, teaches receiving, with the controller, multiple image from one or more sensors disposed on a robot arm of the robotic welding system (Fisher: Figures 2A-B, ¶ 0043, “FIGS. 2A and 2B illustrate an example robot 120 according to an example embodiment. Robot 120 may include a body 126, an end effector 122 holding a tool 123, and a camera 124.”. The cited passage and figures clearly show that the camera is coupled to the end-effector and tool of the robot.);
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method for controlling a robotic welding system taught in Eldridge in view of Nemmers with receiving, with the controller, multiple image from one or more sensors disposed on a robot arm of the robotic welding system taught in Fisher with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results. Both Eldridge and Nemmers teach the use of imaging devices to capture images of the tool of the robot, sop modifying the system taught in Eldridge in view of Nemmers such that the imaging device is coupled to the end-effector and tool as taught in Fisher would not change or introduce new functionality. Furthermore, such a modification only requires changing the location of a known sensor. No inventive effort would have been required.
Regarding claim 3, Eldridge in view of Nemmers in further view of Fisher teaches further comprising: identifying, with the controller, a pose in 3D space of the weldhead (Eldridge: ¶ 0074, “For some tools and processes, finding only the TCP relationship to the wrist frame is adequate. For example, if a touch probe extends directly along the joint axis of the last joint (i.e., the wrist), the orientation of the tool may be assumed to be equal to the orientation of the wrist.”, ¶ 0075, “One method of finding the tool orientation is to move the tool into a known orientation in the world coordinate frame. The wrist pose may then be recorded and the relative orientation between the tool and the wrist may be computed.”, ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. It is clear from the cited passages that the pose of the weldhead is determined.);
Determining, with the controller, the first frame of reference based on the pose of the weldhead (Eldridge: ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.” Nemmers: ¶ 0064, “As more clearly shown in FIG. 10, while the robotic tool is positioned within the field of view of the image generating device, a portion of the robotic tool (i.e. parent tool) is initially located, as identified by an outline 205. In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11. Specifically, while the robotic tool is in the first position, the processor 14 analyzes the first image to locate the first point 206 and an XYZ coordinate representing a position where a pre-defined view line passes through a calibration plane defined during the calibration step 102.”. In the cited passage of Eldridge, the pose of the weldhead is used to determine the TCP in the case that the TCP corresponds to the tip of a welding wire. In the cited passages of Nemmers, the weldhead is located in the image and then the 3D coordinates of a reference point are determined. It would be clear to one of ordinary skill in the art that the both references teach determining a reference frame based on the pose of the weldhead.);
and triangulating, with the controller, the location of the terminal end of the protrusion in 3D space based on a first projection of the terminal end of the protrusion captured in a first image of the multiple images and a second projection of the terminal end of the protrusion captured in a second image of the multiple that is different from the first image (Nemmers: ¶ 0064, “As more clearly shown in FIG. 10, while the robotic tool is positioned within the field of view of the image generating device, a portion of the robotic tool (i.e. parent tool) is initially located, as identified by an outline 205. In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11. Specifically, while the robotic tool is in the first position, the processor 14 analyzes the first image to locate the first point 206 and an XYZ coordinate representing a position where a pre-defined view line passes through a calibration plane defined during the calibration step 102. As a non-limiting example, the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 are each disposed along a first three dimensional line. The processor 14 then repeats the analysis to locate points 208, 210, 212 in a fashion similar to that described above for the point 206.”, ¶ 0065, “The robotic tool is then rotated and the process is repeated to re-locate the points 206, 208, 210, 212 in the second image of the robotic tool. Specifically, the first three dimensional line passing through the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 can be intersected with a second three dimensional line defined from the second image. The intersection of the first three dimensional line and the second three dimensional line associated with the point 206 represents an XYZ value of the point 206. As similar process is repeated to define the XYZ values for the points 208, 210, 212.”, ¶ 0066, “As more clearly shown, the point 206 and the point 208 can be connected using a three dimensional reference line R1. A tool center point 214 based upon the wire 202 can be created from the direction of the reference line R1, the origin point 206 and a pre-defined distance ( e.g. 17 mm) associated with the robotic tool. The process is repeated for the wire 204 using the points 210, 212 to generate reference line R2 and tool center point 216. A user tool frame can be calculated from the data points defined above. Specifically, a reference line R3 through the tool center points 214, 216 defines the X direction. The cross product of the reference line R1 and the reference line R3 defines the XZ plane. Since the origin point, the X axis, and the XZ plane are known, there is enough information to calculate the entire user tool (frame), as appreciate by one skilled in the art.”. It is clear from the cited passages, that the calibration method taught captures multiple images and uses these multiple images to determine the 3D position of the terminal end of the protrusion.).
Regarding claim 4, Eldridge in view of Nemmers in further view of Fisher teaches wherein the second reference frame is: offset with respect to the first frame of reference; defined based on x-, y-, and z- components; included in a plane that is orthogonal to a longitudinal axis of the weldhead and that includes the first frame of reference; or a combination thereof (Nemmers: ¶ 0066, “As more clearly shown, the point 206 and the point 208 can be connected using a three dimensional reference line R1. A tool center point 214 based upon the wire 202 can be created from the direction of the reference line R1, the origin point 206 and a pre-defined distance ( e.g. 17 mm) associated with the robotic tool. The process is repeated for the wire 204 using the points 210, 212 to generate reference line R2 and tool center point 216. A user tool frame can be calculated from the data points defined above. Specifically, a reference line R3 through the tool center points 214, 216 defines the X direction. The cross product of the reference line R1 and the reference line R3 defines the XZ plane. Since the origin point, the X axis, and the XZ plane are known, there is enough information to calculate the entire user tool (frame), as appreciate by one skilled in the art. It is understood that a similar process is performed for determining the user tool for the wire 204. It is further understood that a third user tool can be created by setting the origin point as a centroid along a reference line R4 based on the user tool for the wire 202 and the user tool for the wire 204, wherein the orientation is the same as the user tool for the wire 202.”. The cited passage teaches that the second frame of reference, referred to the third user tool, is clearly offset from the first frame of reference (i.e. the TCPs located at the tip of the welding wires) and is defined in a three-dimensional xyz coordinate system.).
Regarding claim 7, Eldridge in view of Nemmers in further view of Fisher teaches further comprising: identifying, with the controller, in each image of the multiple images, the tip of the weldhead and a location on the weldhead; defining, with the controller, a longitudinal axis of the weldhead based on the tip of the weldhead, the location on the weldhead, or a combination thereof (Eldridge: Figure 11B, ¶ 0109, “FIG. 11B is an example image 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102.” Nemmers: ¶ 0064, “As more clearly shown in FIG. 10, while the robotic tool is positioned within the field of view of the image generating device, a portion of the robotic tool (i.e. parent tool) is initially located, as identified by an outline 205. In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11. Specifically, while the robotic tool is in the first position, the processor 14 analyzes the first image to locate the first point 206 and an XYZ coordinate representing a position where a pre-defined view line passes through a calibration plane defined during the calibration step 102. As a non-limiting example, the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 are each disposed along a first three dimensional line. The processor 14 then repeats the analysis to locate points 208, 210, 212 in a fashion similar to that described above for the point 206.”, ¶ 0065, “The robotic tool is then rotated and the process is repeated to re-locate the points 206, 208, 210, 212 in the second image of the robotic tool. Specifically, the first three dimensional line passing through the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 can be intersected with a second three dimensional line defined from the second image. The intersection of the first three dimensional line and the second three dimensional line associated with the point 206 represents an XYZ value of the point 206. As similar process is repeated to define the XYZ values for the points 208, 210, 212.”. The cited passage of Eldridge clearly teaches determining the longitudinal axis of the weldhead based on the location of the weldhead. The cited passages of Nemmers clearly teaches identifying the tip of the weld head and location of the weldhead in multiple images.);
and determining, with the controller, the first frame of reference based on the longitudinal axis of the weldhead (Eldridge: ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. The cited passages clearly teach that the location of the weldhead is determined based on the longitudinal axis of the weldhead.).
Regarding claim 11, Eldridge teaches a robotic welding system for welding a part, the system comprising (Eldridge: ¶ 0006, “An embodiment of the present invention may further comprise a computerized method for calculating a tool frame tool center point relative to a wrist-frame of a robot for a tool attached at a wrist of the robot using a camera…”, ¶ 0033, “The robot manipulator is typically made up of two subsections, the body and arm 108 and the wrist 110. A tool 112 used by a robot 102 to perform desired tasks is typically attached at the wrist 110 of the robot manipulator 102.”, ¶ 0102, “FIGS. 9A-C show images 900,910,920 of example Metal-Inert Gas (MIG) welding torches. FIG. 9A is an example image of a first type 900 of a MIG welding torch tool. FIG. 9B is an example image of a second type 910 of a MIG welding torch tool. FIG. 9C is an example image of a third type 920 of a MIG welding torch tool.”):
a weldhead configured to receive a protrusion of a weld material (¶ 0102, “FIGS. 9A-C show images 900,910,920 of example Metal-Inert Gas (MIG) welding torches. FIG. 9A is an example image of a first type 900 of a MIG welding torch tool. FIG. 9B is an example image of a second type 910 of a MIG welding torch tool. FIG. 9C is an example image of a third type 920 of a MIG welding torch tool.”, ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. The cited passages clearly show that the weldhead receives a protrusion of weld material.);
a controller in signal communication with the sensor unit, the robotic arm, and the weldhead, wherein the controller is configured to (Eldridge: ¶ 0134, “The computer may have computer accessible memory (e.g., hard drive, flash drive, RAM, etc.) to store information and or programs needed to implement the algorithms/processes to find the tool-frame relative to the wrist-frame of the robot. The computer may send commands to and receive data from the robot and robot controller as necessary to find then relative tool-frame.”):
receive an image from the sensor unit Eldridge: ¶ 0100, “The camera may be used to continuously capture an image of the target tool in real-time. Embodiments may store an image and/or images at desired times to perform calculations based on the stored image and/or images.”, ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. The system clearly receives an image captured by one or more sensors.);
identify, based on an image, a location of a terminal end of the protrusion extending from the weldhead (Eldridge: ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. As can be seen from the cited passage, the method is configured to determine the location of the TCP which can be the tip of the welding wire that protrudes from the weldhead.),
each image including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference (Eldridge: Figures 10A-C and 11A-C, ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. As can be seen from the cited passage and figures, the images clearly include at least a portion of the protrusion extending from the weldhead and that the first frame of refence is associated with a tip of the weldhead.);
and cause one or more robotic operations to be performed by the robotic arm based on the one or more TCP calibration values(Eldridge: ¶ 0134, “The tool calibration software was implemented on a separate computer, with communication to the robot controller occurring over standard communication link. With the aid of some short programs written for the robot, the calibration software was able to command the robot to move to the positions required.”. The system is clearly controlled based on the TCP calibration.).
Eldridge does not teach a sensor unit positioned along a robotic arm;
receive multiple images from the sensor unit;
identify, based on the multiple images, a location of a terminal end of a protrusion extending from the weldhead,
each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference,
determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference;
generate one or more TCP calibration values based on the second frame of reference
Nemmers, in the same field of endeavor, teaches receive multiple images from the sensor unit (Nemmers: ¶ 0058, “In step 108, at least two images of the robotic tool are generated and analyzed to determine a tool center point, (TCP). Specifically, the tool is rotated to a first view position (e.g. -45 degree position) and a first image is generated, as shown in step 110. Four pre-determined targets located on the robotic tool are "found" by the system 10 and four offsets are obtained and stored (e.g. in the storage device 30 or vision registers). In step 112, the robotic tool is rotated 90 degrees to a second position (e.g. +45 degree position). In step 114, a second image of the robotic tool is generated based on the second position.”, ¶ 0064, “As more clearly shown in FIG. 10, while the robotic tool is positioned within the field of view of the image generating device, a portion of the robotic tool (i.e. parent tool) is initially located, as identified by an outline 205. In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11.”, ¶ 0065, “The robotic tool is then rotated and the process is repeated to re-locate the points 206, 208, 210, 212 in the second image of the robotic tool. Specifically, the first three dimensional line passing through the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 can be intersected with a second three dimensional line defined from the second image.”. The cited passages clearly show that the method taught in Nemmers clearly involve receiving multiple images from one or more sensors of the robotic welding system.);
identify, based on the multiple images, a location of a terminal end of a protrusion extending from the weldhead (Nemmers: ¶ 0058, “In step 108, at least two images of the robotic tool are generated and analyzed to determine a tool center point, (TCP). Specifically, the tool is rotated to a first view position (e.g. -45 degree position) and a first image is generated, as shown in step 110. Four pre-determined targets located on the robotic tool are "found" by the system 10 and four offsets are obtained and stored (e.g. in the storage device 30 or vision registers). In step 112, the robotic tool is rotated 90 degrees to a second position (e.g. +45 degree position). In step 114, a second image of the robotic tool is generated based on the second position.”. The cited passage clearly shows that the system captures multiple images and uses these images to determine the location of the terminal end of a protrusion extending from the weldhead (i.e. the TCP).),
each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference (Nemmers: ¶ 0058, “In step 108, at least two images of the robotic tool are generated and analyzed to determine a tool center point, (TCP). Specifically, the tool is rotated to a first view position (e.g. -45 degree position) and a first image is generated, as shown in step 110. Four pre-determined targets located on the robotic tool are "found" by the system 10 and four offsets are obtained and stored (e.g. in the storage device 30 or vision registers). In step 112, the robotic tool is rotated 90 degrees to a second position (e.g. +45 degree position). In step 114, a second image of the robotic tool is generated based on the second position.”. One of ordinary skill in the art would see that the multiple images would each include at least a portion of the protrusion extending from a tip of the weldhead.)
determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference (Nemmers: Figured 10-12, ¶ 0066, “As more clearly shown, the point 206 and the point 208 can be connected using a three dimensional reference line R1. A tool center point 214 based upon the wire 202 can be created from the direction of the reference line R1, the origin point 206 and a pre-defined distance ( e.g. 17 mm) associated with the robotic tool. The process is repeated for the wire 204 using the points 210, 212 to generate reference line R2 and tool center point 216. A user tool frame can be calculated from the data points defined above. Specifically, a reference line R3 through the tool center points 214, 216 defines the X direction. The cross product of the reference line R1 and the reference line R3 defines the XZ plane. Since the origin point, the X axis, and the XZ plane are known, there is enough information to calculate the entire user tool (frame), as appreciate by one skilled in the art. It is understood that a similar process is performed for determining the user tool for the wire 204. It is further understood that a third user tool can be created by setting the origin point as a centroid along a reference line R4 based on the user tool for the wire 202 and the user tool for the wire 204, wherein the orientation is the same as the user tool for the wire 202.”. The cited passages and figures clearly show that the reference points that define the tip of the welding wire (points 214 and 216) can be used to define an origin point that is offset from the first reference points and lie on the intersection between reference lines R3 and R4.),
generate one or more TCP calibration values based on the second frame of reference (Nemmers: ¶ 0061, “In step 120, a user tool for a particular tool is updated. In certain embodiments the user tool is updated based upon a measure user tool process that measures the location of a pre-defined portion of the robotic tool and creates a current (Internal) user tool. The current user tools are compared with the nominal user tools to determine any changes or offsets. The system 10 tracks changes in X, Y, Z, W, P, R, distance, and tilt elements, wherein the changes are represented by delta values. Distance is defined as the length of a three dimension line that connects a pre-defined point on the current user tool with the reciprocal point on the nominal tool. Tilt is defined as a combination of the Yaw and Pitch and represents an angle change with respect to the Z axis. It is understood that the distance and tilt measurements are easier to conceptualize than conventional XYZWPR coordinates. It is further understood that the delta values can be further analyzed to indicate undesirable changes in the robotic tool orientation.”, ¶ 0066, “It is further understood that a third user tool can be created by setting the origin point as a centroid along a reference line R4 based on the user tool for the wire 202 and the user tool for the wire 204, wherein the orientation is the same as the user tool for the wire 202.”. The cited passages clearly show that the user tool values determined based on the second reference point are used in to generate TCP calibration values.).
Eldridge teaches a robotic welding system for welding a part, the system comprising: a weldhead configured to receive a protrusion of a weld material; a controller in signal communication with the sensor unit, wherein the controller is configured to: receive an image from the sensor unit; identify, based image, a location of a terminal end of a protrusion extending from the weldhead, each image including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference. Eldridge does not teach receive multiple images from the sensor unit; identify, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference; generate one or more TCP calibration values based on the second frame of reference. Nemmers teaches receive multiple images from the sensor unit; identify, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference; generate one or more TCP calibration values based on the second frame of reference. A person of ordinary skill in the art would have had the technological capabilities required to have modified the system taught in Eldridge with receive multiple images from the sensor unit; identify, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference; generate one or more TCP calibration values based on the second frame of reference taught in Nemmers. Furthermore, the system taught in Eldridge is already configured to determine the TCP calibration values based on a reference point associated with the terminal end of a protrusion extending form the weldhead using an image, so modifying this method to use multiple images, to define a second reference point, and use this second refence point to determine the TCP calibration values as taught in Nemmers would not change or introduce new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a robotic welding system for welding a part, the system comprising: receive multiple images from the sensor unit; identify, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference, generate one or more TCP calibration values based on the second frame of reference.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robotic welding system for taught in Eldridge with receive multiple images from the sensor unit; identify, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference, generate one or more TCP calibration values based on the second frame of reference taught in Nemmers with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Eldridge in view of Nemmers does not teach a sensor unit positioned along a robotic arm.
Fisher, in the same filed of endeavor teaches a sensor unit positioned along a robotic arm (Fisher: Figures 2A-B, ¶ 0043, “FIGS. 2A and 2B illustrate an example robot 120 according to an example embodiment. Robot 120 may include a body 126, an end effector 122 holding a tool 123, and a camera 124.”. The cited passage and figures clearly show that the camera is coupled to the end-effector and tool of the robot.);
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robotic welding system taught in Eldridge in view of Nemmers with a sensor unit positioned along a robotic arm taught in Fisher with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results. Both Eldridge and Nemmers teach the use of imaging devices to capture images of the tool of the robot, so modifying the system taught in Eldridge in view of Nemmers such that the imaging device is coupled to the end-effector and tool as taught in Fisher would not change or introduce new functionality. Furthermore, such a modification only requires changing the location of a known sensor. No inventive effort would have been required.
Regarding claim 12, Eldridge in view of Nemmers in further view of Fisher teaches wherein the sensor unit has a field of view that includes at least a portion of the weldhead (Eldridge: ¶ 0100, “The camera may be used to continuously capture an image of the target tool in real-time. Embodiments may store an image and/or images at desired times to perform calculations based on the stored image and/or images.”, ¶ 0108, “FIGS. 10A-C show example images of the process of segmenting the original image 1000 into a convex hull image 1004 for step 1 of the process described above using a MIG welding torch as the tool. FIG. 10A is an example image of an original image 1000 captured in a process for locating a TCP of a tool on the camera image.”. The cited passage clearly shows that the sensors capture images including at least a portion of the weldhead.);
and wherein: the sensor unit is coupled to the weldhead (Fisher: Figures 2A-B, ¶ 0043, “FIGS. 2A and 2B illustrate an example robot 120 according to an example embodiment. Robot 120 may include a body 126, an end effector 122 holding a tool 123, and a camera 124.”. The cited passage and figures clearly show that the camera is couples to the end-effector and tool of the robot.); and
the controller is configured to: determine the first frame of reference based on the multiple images (Nemmers: ¶ 0064, “As more clearly shown in FIG. 10, while the robotic tool is positioned within the field of view of the image generating device, a portion of the robotic tool (i.e. parent tool) is initially located, as identified by an outline 205. In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11. Specifically, while the robotic tool is in the first position, the processor 14 analyzes the first image to locate the first point 206 and an XYZ coordinate representing a position where a pre-defined view line passes through a calibration plane defined during the calibration step 102. As a non-limiting example, the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 are each disposed along a first three dimensional line. The processor 14 then repeats the analysis to locate points 208, 210, 212 in a fashion similar to that described above for the point 206.”, ¶ 0065, “The robotic tool is then rotated and the process is repeated to re-locate the points 206, 208, 210, 212 in the second image of the robotic tool. Specifically, the first three dimensional line passing through the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 can be intersected with a second three dimensional line defined from the second image. The intersection of the first three dimensional line and the second three dimensional line associated with the point 206 represents an XYZ value of the point 206. As similar process is repeated to define the XYZ values for the points 208, 210, 212.”. The cited passages clearly show that a reference frame is determined using the multiple images.)
the first frame of reference defines a position of the tip of the weldhead, an orientation of the tip of the weldhead (Eldridge: ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. The cited passage clearly teaches that the reference frame defines the position and orientation of the tip of the weldhead.);
Regarding claim 14, Eldridge in view of Nemmers in further view of Fisher teaches wherein the controller is configured to: identify a pose in 3D space of the weldhead (Eldridge: ¶ 0074, “For some tools and processes, finding only the TCP relationship to the wrist frame is adequate. For example, if a touch probe extends directly along the joint axis of the last joint (i.e., the wrist), the orientation of the tool may be assumed to be equal to the orientation of the wrist.”, ¶ 0075, “One method of finding the tool orientation is to move the tool into a known orientation in the world coordinate frame. The wrist pose may then be recorded and the relative orientation between the tool and the wrist may be computed.”, ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. It is clear from the cited passages that the pose of the weldhead is determined.);
determine the first frame of reference based on the pose of the weldhead (Eldridge: ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.” Nemmers: ¶ 0064, “As more clearly shown in FIG. 10, while the robotic tool is positioned within the field of view of the image generating device, a portion of the robotic tool (i.e. parent tool) is initially located, as identified by an outline 205. In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11. Specifically, while the robotic tool is in the first position, the processor 14 analyzes the first image to locate the first point 206 and an XYZ coordinate representing a position where a pre-defined view line passes through a calibration plane defined during the calibration step 102.”. In the cited passage of Eldridge, the pose of the weldhead is used to determine the TCP in the case that the TCP corresponds to the tip of a welding wire. In the cited passages of Nemmers, the weldhead is located in the image and then the 3D coordinates of a reference point are determined. It would be clear to one of ordinary skill in the art that the both references teach determining a reference frame based on the pose of the weldhead.); and
triangulate a location in 3D space of the terminal end of the protrusion based on a first projection of the terminal end of the protrusion captured in a first image of the multiple images and a second projection of the terminal end of the protrusion captured in a second image of the multiple that is different from the first image (Nemmers: ¶ 0064, “As more clearly shown in FIG. 10, while the robotic tool is positioned within the field of view of the image generating device, a portion of the robotic tool (i.e. parent tool) is initially located, as identified by an outline 205. In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11. Specifically, while the robotic tool is in the first position, the processor 14 analyzes the first image to locate the first point 206 and an XYZ coordinate representing a position where a pre-defined view line passes through a calibration plane defined during the calibration step 102. As a non-limiting example, the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 are each disposed along a first three dimensional line. The processor 14 then repeats the analysis to locate points 208, 210, 212 in a fashion similar to that described above for the point 206.”, ¶ 0065, “The robotic tool is then rotated and the process is repeated to re-locate the points 206, 208, 210, 212 in the second image of the robotic tool. Specifically, the first three dimensional line passing through the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 can be intersected with a second three dimensional line defined from the second image. The intersection of the first three dimensional line and the second three dimensional line associated with the point 206 represents an XYZ value of the point 206. As similar process is repeated to define the XYZ values for the points 208, 210, 212.”, ¶ 0066, “As more clearly shown, the point 206 and the point 208 can be connected using a three dimensional reference line R1. A tool center point 214 based upon the wire 202 can be created from the direction of the reference line R1, the origin point 206 and a pre-defined distance ( e.g. 17 mm) associated with the robotic tool. The process is repeated for the wire 204 using the points 210, 212 to generate reference line R2 and tool center point 216. A user tool frame can be calculated from the data points defined above. Specifically, a reference line R3 through the tool center points 214, 216 defines the X direction. The cross product of the reference line R1 and the reference line R3 defines the XZ plane. Since the origin point, the X axis, and the XZ plane are known, there is enough information to calculate the entire user tool (frame), as appreciate by one skilled in the art.”. It is clear from the cited passages, that the calibration method taught captures multiple images and uses these multiple images to determine the 3D position of the terminal end of the protrusion.).
Regarding claim 17, Eldridge system for calibrating a tool center point (TCP) of a robotic welding system, the system comprising (Eldridge: ¶ 0006, “An embodiment of the present invention may further comprise a computerized method for calculating a tool frame tool center point relative to a wrist-frame of a robot for a tool attached at a wrist of the robot using a camera…”, ¶ 0033, “The robot manipulator is typically made up of two subsections, the body and arm 108 and the wrist 110. A tool 112 used by a robot 102 to perform desired tasks is typically attached at the wrist 110 of the robot manipulator 102.”, ¶ 0102, “FIGS. 9A-C show images 900,910,920 of example Metal-Inert Gas (MIG) welding torches. FIG. 9A is an example image of a first type 900 of a MIG welding torch tool. FIG. 9B is an example image of a second type 910 of a MIG welding torch tool. FIG. 9C is an example image of a third type 920 of a MIG welding torch tool.”):
a processor (Eldridge: ¶ 0134, “The computer may have computer accessible memory (e.g., hard drive, flash drive, RAM, etc.) to store information and or programs needed to implement the algorithms/processes to find the tool-frame relative to the wrist-frame of the robot. The computer may send commands to and receive data from the robot and robot controller as necessary to find then relative tool-frame.”); and
a non-transitory memory storing processor executable instructions that, when executed by the processor, cause the processor to (: ¶ 0134, “The computer may have computer accessible memory (e.g., hard drive, flash drive, RAM, etc.) to store information and or programs needed to implement the algorithms/processes to find the tool-frame relative to the wrist-frame of the robot. The computer may send commands to and receive data from the robot and robot controller as necessary to find then relative tool-frame.”):
receive an image from one or more sensors (Eldridge: ¶ 0100, “The camera may be used to continuously capture an image of the target tool in real-time. Embodiments may store an image and/or images at desired times to perform calculations based on the stored image and/or images.”, ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. The system clearly receives an image captured by one or more sensors.);
identify, based on an image, a location of a tip of a protrusion extending from the weldhead (Eldridge: ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. As can be seen from the cited passage, the method is configured to determine the location of the TCP which can be the tip of the welding wire that protrudes from the weldhead.),
each image including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference (Eldridge: Figures 10A-C and 11A-C, ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. As can be seen from the cited passage and figures, the images clearly include at least a portion of the protrusion extending from the weldhead and that the first frame of refence is associated with a tip of the weldhead.); and
cause one or more robotic operations to be performed with the robot arm based on the one or more TCP calibration values (Eldridge: ¶ 0134, “The tool calibration software was implemented on a separate computer, with communication to the robot controller occurring over standard communication link. With the aid of some short programs written for the robot, the calibration software was able to command the robot to move to the positions required.”. The system is clearly controlled based on the TCP calibration.).
Eldridge does not teach receive multiple image from one or more sensors disposed on a robot arm of the robotic welding system
identify, based on the multiple images, a location of a terminal end of a protrusion extending from the weldhead,
each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference,
determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference;
generate one or more TCP calibration values based on the second frame of reference
Nemmers, in the same field of endeavor, teaches receive multiple image from one or more sensors (Nemmers: ¶ 0058, “In step 108, at least two images of the robotic tool are generated and analyzed to determine a tool center point, (TCP). Specifically, the tool is rotated to a first view position (e.g. -45 degree position) and a first image is generated, as shown in step 110. Four pre-determined targets located on the robotic tool are "found" by the system 10 and four offsets are obtained and stored (e.g. in the storage device 30 or vision registers). In step 112, the robotic tool is rotated 90 degrees to a second position (e.g. +45 degree position). In step 114, a second image of the robotic tool is generated based on the second position.”, ¶ 0064, “As more clearly shown in FIG. 10, while the robotic tool is positioned within the field of view of the image generating device, a portion of the robotic tool (i.e. parent tool) is initially located, as identified by an outline 205. In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11.”, ¶ 0065, “The robotic tool is then rotated and the process is repeated to re-locate the points 206, 208, 210, 212 in the second image of the robotic tool. Specifically, the first three dimensional line passing through the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 can be intersected with a second three dimensional line defined from the second image.”. The cited passages clearly show that the method taught in Nemmers clearly involve receiving multiple images from one or more sensors of the robotic welding system.);
identify, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead (Nemmers: ¶ 0058, “In step 108, at least two images of the robotic tool are generated and analyzed to determine a tool center point, (TCP). Specifically, the tool is rotated to a first view position (e.g. -45 degree position) and a first image is generated, as shown in step 110. Four pre-determined targets located on the robotic tool are "found" by the system 10 and four offsets are obtained and stored (e.g. in the storage device 30 or vision registers). In step 112, the robotic tool is rotated 90 degrees to a second position (e.g. +45 degree position). In step 114, a second image of the robotic tool is generated based on the second position.”. The cited passage clearly shows that the system captures multiple images and uses these images to determine the location of the terminal end of a protrusion extending from the weldhead (i.e. the TCP).),
each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference (Nemmers: ¶ 0058, “In step 108, at least two images of the robotic tool are generated and analyzed to determine a tool center point, (TCP). Specifically, the tool is rotated to a first view position (e.g. -45 degree position) and a first image is generated, as shown in step 110. Four pre-determined targets located on the robotic tool are "found" by the system 10 and four offsets are obtained and stored (e.g. in the storage device 30 or vision registers). In step 112, the robotic tool is rotated 90 degrees to a second position (e.g. +45 degree position). In step 114, a second image of the robotic tool is generated based on the second position.”. One of ordinary skill in the art would see that the multiple images would each include at least a portion of the protrusion extending from a tip of the weldhead.)
determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference (Nemmers: Figured 10-12, ¶ 0066, “As more clearly shown, the point 206 and the point 208 can be connected using a three dimensional reference line R1. A tool center point 214 based upon the wire 202 can be created from the direction of the reference line R1, the origin point 206 and a pre-defined distance ( e.g. 17 mm) associated with the robotic tool. The process is repeated for the wire 204 using the points 210, 212 to generate reference line R2 and tool center point 216. A user tool frame can be calculated from the data points defined above. Specifically, a reference line R3 through the tool center points 214, 216 defines the X direction. The cross product of the reference line R1 and the reference line R3 defines the XZ plane. Since the origin point, the X axis, and the XZ plane are known, there is enough information to calculate the entire user tool (frame), as appreciate by one skilled in the art. It is understood that a similar process is performed for determining the user tool for the wire 204. It is further understood that a third user tool can be created by setting the origin point as a centroid along a reference line R4 based on the user tool for the wire 202 and the user tool for the wire 204, wherein the orientation is the same as the user tool for the wire 202.”. The cited passages and figures clearly show that the reference points that define the tip of the welding wire (points 214 and 216) can be used to define an origin point that is offset from the first reference points and lie on the intersection between reference lines R3 and R4.),
generate one or more TCP calibration values based on the second frame of reference (Nemmers: ¶ 0061, “In step 120, a user tool for a particular tool is updated. In certain embodiments the user tool is updated based upon a measure user tool process that measures the location of a pre-defined portion of the robotic tool and creates a current (Internal) user tool. The current user tools are compared with the nominal user tools to determine any changes or offsets. The system 10 tracks changes in X, Y, Z, W, P, R, distance, and tilt elements, wherein the changes are represented by delta values. Distance is defined as the length of a three dimension line that connects a pre-defined point on the current user tool with the reciprocal point on the nominal tool. Tilt is defined as a combination of the Yaw and Pitch and represents an angle change with respect to the Z axis. It is understood that the distance and tilt measurements are easier to conceptualize than conventional XYZWPR coordinates. It is further understood that the delta values can be further analyzed to indicate undesirable changes in the robotic tool orientation.”, ¶ 0066, “It is further understood that a third user tool can be created by setting the origin point as a centroid along a reference line R4 based on the user tool for the wire 202 and the user tool for the wire 204, wherein the orientation is the same as the user tool for the wire 202.”. The cited passages clearly show that the user tool values determined based on the second reference point are used in to generate TCP calibration values.).
Eldridge teaches a system for calibrating a tool center point (TCP) of a robotic welding system, the system comprising: a processor; and a non-transitory memory storing processor executable instructions that, when executed by the processor, cause the processor to: receive an image from one or more sensors; identify, based image, a location of a terminal end of a protrusion extending from the weldhead, each image including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference. Eldridge does not teach receive multiple image from one or more sensors; identify, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference; generate one or more TCP calibration values based on the second frame of reference. Nemmers teaches receive multiple image from one or more sensors; identify, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference; generate one or more TCP calibration values based on the second frame of reference. A person of ordinary skill in the art would have had the technological capabilities required to have modified the system taught in Eldridge with receive multiple image from one or more sensors; identify, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference; generate one or more TCP calibration values based on the second frame of reference taught in Nemmers. Furthermore, the system taught in Eldridge is already configured to determine the TCP calibration values based on a reference point associated with the terminal end of a protrusion extending form the weldhead using an image, so modifying this method to use multiple images, to define a second reference point, and use this second refence point to determine the TCP calibration values as taught in Nemmers would not change or introduce new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of system for calibrating a tool center point (TCP) of a robotic welding system, the system comprising: receive multiple image from one or more sensors; identify, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference, generate one or more TCP calibration values based on the second frame of reference.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the system for calibrating a tool center point (TCP) of a robotic welding system taught in Eldridge with receive multiple image from one or more sensors; identify, based on multiple images, a location of a terminal end of a protrusion extending from the weldhead, each image of the multiple images including at least a portion of the protrusion extending from a tip of the weldhead, the tip of the weldhead associated with a first frame of reference, determine, based on the location of the terminal end of the protrusion, a second frame of reference that is offset from the first frame of reference, generate one or more TCP calibration values based on the second frame of reference taught in Nemmers with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Eldridge in view of Nemmers does not teach receive multiple image from one or more sensors disposed on a robot arm of the robotic welding system.
Fisher, in the same field of endeavor, teaches receive multiple image from one or more sensors disposed on a robot arm of the robotic welding system (Fisher: Figures 2A-B, ¶ 0043, “FIGS. 2A and 2B illustrate an example robot 120 according to an example embodiment. Robot 120 may include a body 126, an end effector 122 holding a tool 123, and a camera 124.”. The cited passage and figures clearly show that the camera is coupled to the end-effector and tool of the robot.);
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method for controlling a robotic welding system taught in Eldridge in view of Nemmers with receive multiple image from one or more sensors disposed on a robot arm of the robotic welding system taught in Fisher with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results. Both Eldridge and Nemmers teach the use of imaging devices to capture images of the tool of the robot, sop modifying the system taught in Eldridge in view of Nemmers such that the imaging device is coupled to the end-effector and tool as taught in Fisher would not change or introduce new functionality. Furthermore, such a modification only requires changing the location of a known sensor. No inventive effort would have been required.
Regarding claim 18, Eldridge in view of Nemmers in further view of Fisher teaches wherein the instructions, when executed by the processor, further cause the processor to: identify a pose in 3D space of the weldhead (Eldridge: ¶ 0074, “For some tools and processes, finding only the TCP relationship to the wrist frame is adequate. For example, if a touch probe extends directly along the joint axis of the last joint (i.e., the wrist), the orientation of the tool may be assumed to be equal to the orientation of the wrist.”, ¶ 0075, “One method of finding the tool orientation is to move the tool into a known orientation in the world coordinate frame. The wrist pose may then be recorded and the relative orientation between the tool and the wrist may be computed.”, ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. It is clear from the cited passages that the pose of the weldhead is determined.); and
determine the first frame of reference based on the pose of the weldhead (Eldridge: ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.” Nemmers: ¶ 0064, “As more clearly shown in FIG. 10, while the robotic tool is positioned within the field of view of the image generating device, a portion of the robotic tool (i.e. parent tool) is initially located, as identified by an outline 205. In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11. Specifically, while the robotic tool is in the first position, the processor 14 analyzes the first image to locate the first point 206 and an XYZ coordinate representing a position where a pre-defined view line passes through a calibration plane defined during the calibration step 102.”. In the cited passage of Eldridge, the pose of the weldhead is used to determine the TCP in the case that the TCP corresponds to the tip of a welding wire. In the cited passages of Nemmers, the weldhead is located in the image and then the 3D coordinates of a reference point are determined. It would be clear to one of ordinary skill in the art that the both references teach determining a reference frame based on the pose of the weldhead.).
Regarding claim 19, Eldridge in view of Nemmers in further view of Fisher teaches wherein the instructions, when executed by the processor, further cause the processor to: triangulate the location of the terminal end of the protrusion in 3D space based on a first projection of the terminal end of the protrusion captured in a first image of the multiple images and a second projection of the terminal end of the protrusion captured in a second image of the multiple images that is different from the first image (Nemmers: ¶ 0064, “As more clearly shown in FIG. 10, while the robotic tool is positioned within the field of view of the image generating device, a portion of the robotic tool (i.e. parent tool) is initially located, as identified by an outline 205. In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11. Specifically, while the robotic tool is in the first position, the processor 14 analyzes the first image to locate the first point 206 and an XYZ coordinate representing a position where a pre-defined view line passes through a calibration plane defined during the calibration step 102. As a non-limiting example, the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 are each disposed along a first three dimensional line. The processor 14 then repeats the analysis to locate points 208, 210, 212 in a fashion similar to that described above for the point 206.”, ¶ 0065, “The robotic tool is then rotated and the process is repeated to re-locate the points 206, 208, 210, 212 in the second image of the robotic tool. Specifically, the first three dimensional line passing through the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 can be intersected with a second three dimensional line defined from the second image. The intersection of the first three dimensional line and the second three dimensional line associated with the point 206 represents an XYZ value of the point 206. As similar process is repeated to define the XYZ values for the points 208, 210, 212.”, ¶ 0066, “As more clearly shown, the point 206 and the point 208 can be connected using a three dimensional reference line R1. A tool center point 214 based upon the wire 202 can be created from the direction of the reference line R1, the origin point 206 and a pre-defined distance ( e.g. 17 mm) associated with the robotic tool. The process is repeated for the wire 204 using the points 210, 212 to generate reference line R2 and tool center point 216. A user tool frame can be calculated from the data points defined above. Specifically, a reference line R3 through the tool center points 214, 216 defines the X direction. The cross product of the reference line R1 and the reference line R3 defines the XZ plane. Since the origin point, the X axis, and the XZ plane are known, there is enough information to calculate the entire user tool (frame), as appreciate by one skilled in the art.”. It is clear from the cited passages, that the calibration method taught captures multiple images and uses these multiple images to determine the 3D position of the terminal end of the protrusion.).
Claim(s) 2 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2009/0118864 A1 ("Eldridge") in view of US 20110029132 A1 ("Nemmers") in further view of US 2021/0065356 A1 ("Fisher") in further view of US 10926414 B2 ("Huang").
Regarding claim 2, Eldridge in view of Nemmers in further view of Fisher teaches further comprising: determining, with the controller, the first frame of reference based on the multiple images (Nemmers: ¶ 0064, “As more clearly shown in FIG. 10, while the robotic tool is positioned within the field of view of the image generating device, a portion of the robotic tool (i.e. parent tool) is initially located, as identified by an outline 205. In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11. Specifically, while the robotic tool is in the first position, the processor 14 analyzes the first image to locate the first point 206 and an XYZ coordinate representing a position where a pre-defined view line passes through a calibration plane defined during the calibration step 102. As a non-limiting example, the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 are each disposed along a first three dimensional line. The processor 14 then repeats the analysis to locate points 208, 210, 212 in a fashion similar to that described above for the point 206.”, ¶ 0065, “The robotic tool is then rotated and the process is repeated to re-locate the points 206, 208, 210, 212 in the second image of the robotic tool. Specifically, the first three dimensional line passing through the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 can be intersected with a second three dimensional line defined from the second image. The intersection of the first three dimensional line and the second three dimensional line associated with the point 206 represents an XYZ value of the point 206. As similar process is repeated to define the XYZ values for the points 208, 210, 212.”. The cited passages clearly show that a reference frame is determined using the multiple images.),
wherein the first frame of reference defines a position of the tip of the weldhead, an orientation of the tip of the weldhead, or a combination thereof (Eldridge: ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”. The cited passage clearly teaches that the reference frame defines the position and orientation of the tip of the weldhead.);
the tip of the weldhead corresponds to a base of the protrusion; the protrusion comprises a welding wire; the second reference point is associated with the TCP; or a combination thereof (Eldridge: Figures 11A-C, ¶ 0109, “FIGS. 11A-C show example images of the remaining sub-process steps 2-4 for finding the TCP (1124 or 1126) of the tool 1102 in the original camera image 1000. FIG. 11A is an example image 1100 showing the sub-process for step 2 of finding a rough orientation 1114 of the tool 1102 by fitting an ellipse 1104 around the convex hull image 1004 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. FIG. 11B is an example image. 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102. Usually the TCP of the tool 1102 is defined to be where the wire exits the nozzle 1124, so in step 4 of the process for finding the TCP in the camera image 1000, the algorithm is really searching for the end of the gas cup of the tool 1124. For some embodiments, the TCP may alternatively be defined to be the actual end of the torch tool 1102 at the tip of the wire 1126.”, Nemmers: Figures 11 and 12, ¶ 0066, “As more clearly shown, the point 206 and the point 208 can be connected using a three dimensional reference line R1. A tool center point 214 based upon the wire 202 can be created from the direction of the reference line R1, the origin point 206 and a pre-defined distance ( e.g. 17 mm) associated with the robotic tool. The process is repeated for the wire 204 using the points 210, 212 to generate reference line R2 and tool center point 216. A user tool frame can be calculated from the data points defined above. Specifically, a reference line R3 through the tool center points 214, 216 defines the X direction. The cross product of the reference line R1 and the reference line R3 defines the XZ plane. Since the origin point, the X axis, and the XZ plane are known, there is enough information to calculate the entire user tool (frame), as appreciate by one skilled in the art. It is understood that a similar process is performed for determining the user tool for the wire 204. It is further understood that a third user tool can be created by setting the origin point as a centroid along a reference line R4 based on the user tool for the wire 202 and the user tool for the wire 204, wherein the orientation is the same as the user tool for the wire 202.”. As can clearly be seen in the cited figures and passage of Eldridge, the protrusion is a welding wire. As can be seen from the cited figures and passages of Nemmers, the first frame of reference (i.e. points 206 and 208) correspond to the base of the protrusion and the second frame of reference is associated with the TCP (i.e. points 214 and 216).).
Eldridge in view of Nemmers in further view of Fisher does not teach and wherein: the one or more sensors include a pair of cameras arranged stereoscopically in relation to the weldhead;
Huang, in the same field of endeavor, teaches and wherein: the one or more sensors include a pair of cameras arranged stereoscopically in relation to the weldhead (Huang: Figure 4, Column 9 lines 59-67 , “As shown in FIG. 7, the method of the system 1 for calibrating tool center point of robot includes the following steps. Step S71: Providing a first image sensor 11 having a first image central axis A. Step S72: Providing a second image sensor 12 having a second image central axis B not parallel to the first image central axis A and intersecting the first image central axis A at an intersection point I.”);
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method for calibrating a tool center point (TCP) of a robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher with wherein: the one or more sensors include a pair of cameras arranged stereoscopically in relation to the weldhead taught in Huang with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would only require the simple substitution of one known sensor for another. Both Eldridge and Nemmers teach capturing images of the weldhead of the robotic system, so the combination would only require simply substituting the camera taught in Eldridge in view of Nemmers in further view of Fisher for a stereoscopic cameras taught in Huang. A person of ordinary skill in the art would have had the technological capabilities required to substitute the camera taught in Eldridge in view of Nemmers in further view of Fisher for a stereoscopic camera taught in Huang. The substitution would not have changed or introduced new functionality. No inventive effort would have been required.
Regarding claim 16, Eldridge in view of Nemmers in further view of Fisher does not teach wherein the sensor unit comprises a pair of cameras arranged stereoscopically in relation to the weldhead.
Huang, in the same field of endeavor, teaches wherein the sensor unit comprises a pair of cameras arranged stereoscopically in relation to the weldhead (Huang: Figure 4, Column 9 lines 59-67 , “As shown in FIG. 7, the method of the system 1 for calibrating tool center point of robot includes the following steps. Step S71: Providing a first image sensor 11 having a first image central axis A. Step S72: Providing a second image sensor 12 having a second image central axis B not parallel to the first image central axis A and intersecting the first image central axis A at an intersection point I.”);
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher with wherein the sensor unit comprises a pair of cameras arranged stereoscopically in relation to the weldhead taught in Huang with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would only require the simple substitution of one known sensor for another. Both Eldridge and Nemmers teach capturing images of the weldhead of the robotic system, so the combination would only require simply substituting the camera taught in Eldridge in view of Nemmers in further view of Fisher for a stereoscopic cameras taught in Huang. A person of ordinary skill in the art would have had the technological capabilities required to substitute the camera taught in Eldridge in view of Nemmers in further view of Fisher for a stereoscopic camera taught in Huang. The substitution would not have changed or introduced new functionality. No inventive effort would have been required.
Claim(s) 5, 8, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2009/0118864 A1 ("Eldridge") in view of US 20110029132 A1 ("Nemmers") in further view of US 2021/0065356 A1 ("Fisher") in further view of US 2018/0154518 A1 ("Rossano").
Regarding claim 5, Eldridge in view of Nemmers in further view of Fisher teaches and wherein: the first frame of reference is located on the longitudinal axis; a line that passes through the terminal end of the protrusion and the second frame of reference is parallel to the longitudinal axis of the weldhead; or a combination thereof (Eldridge: Figure 11B, ¶ 0109, “FIG. 11B is an example image 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102.” Nemmers: ¶ 0066, “It is further understood that a third user tool can be created by setting the origin point as a centroid along a reference line R4 based on the user tool for the wire 202 and the user tool for the wire 204, wherein the orientation is the same as the user tool for the wire 202.”. It is clear from the cited passage of Eldridge that the first frame of reference lies on the longitudinal axis that is defined for the weldhead. Furthermore, the cited passages of Nemmers shows that the second reference point (i.e. the third user tool)is parallel to the longitudinal axis of the weldhead.).
Eldridge in view of Nemmers in further view of Fisher does not teach further comprising: identifying, with the controller, a trajectory in 3D space of a longitudinal axis of the weldhead of the weldhead;
Rossano, in the same field of endeavor, teaches further comprising: identifying, with the controller, a trajectory in 3D space of a longitudinal axis of the weldhead of the weldhead (Rossano: ¶ 0047, “The first path 164 can correspond to a predetermined path the TCP 162 of the robot 106, such as, for example a path the robot 106 is planned or selected to take either before or during the operation of the robot 106, and can be depicted by the GUI 136 on the display 124 in a variety of manners.”).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method for calibrating a tool center point (TCP) of a robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher with the trajectory planning taught in Rossano with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because it would have been obvious to try. To facilitate the automation of a robotic system configured to carry out a task, it is necessary to define the robot’s trajectory in space. This is done so that the robot can move from a starting position to an ending position, avoid obstacles, or when the task itself involves following a path (i.e., welding). A person of ordinary skill in the art would have had the technological capabilities and knowledge to recognize the necessity of trajectory planning and be able to implement it in the robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher. No inventive effort would have been required.
Regarding claim 8, Eldridge in view of Nemmers in further view of Fisher does not teach further comprising: determining, with the controller, based on the TCP calibration values: a contact tip to work distance (CTWD); a placement of the weldhead for a weld operation; or a combination thereof.
Rossano, in the same field of endeavor, teaches further comprising: determining, with the controller, based on the TCP calibration values: a contact tip to work distance (CTWD); a placement of the weldhead for a weld operation; or a combination thereof (Rossano: Figure 3B, ¶ 0041, “Additionally, as indicated by FIG. 3B, the GUI 136 can also be configured to provide information regarding the relative distances between a second reference indicator 146, such as, for example, a reference location of the robot 106, subpart 132, and/or end effector 108, relative to a target indicator 148, such as, for example, a target location of or around the main part 134. For example, as illustrated, the GUI 136 can include a legend 150 or other graphical indications or representations of the distance along one or more axes of the coordinate system of the GUI 136 between second reference location 146 and target location 148 of the subpart 132, main part 134, and/or certain components of the robot 106. For example, as shown in FIG. 6, according to certain embodiments, the GUI 136 can display a distance along three axes of the GUI coordinate system of the distance between the second reference indicator 146, such as a TCP of the subpart 132 and/or robot 106, and the target indicator 148. The distance(s) depicted by the GUI 136 can correlate to one or more actual distance or reference distances.”. The cited passage and figure clearly teaches that the distance between the tip of the end effector and the part is determined. Additionally, it is clear from the cited passage and image the target location 148 represents a desired placement for the end effector. Such a target location could easily be applied to the placement of a weldhead for a weld operation. Therefore, the cited passage also teaches the placement of the weldhead.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method for calibrating a tool center point (TCP) of a robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher with further comprising: determining, with the controller, based on the TCP calibration values: a contact tip to work distance (CTWD); a placement of the weldhead for a weld operation; or a combination thereof taught in Rossano with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because it would have been obvious to try. To facilitate the automation of a robotic system configured to carry out a task, it is necessary to define the robot’s trajectory in space. This requires knowledge of the relative distance between the robot’s tool and the workpiece so that the robot can properly place the tool in order to properly carry out its task. Furthermore, in order to carry out a task involving processing a workpiece (i.e. welding), a location of where to place the tool is required. A person of ordinary skill in the art would have had the technological capabilities and knowledge to recognize the necessity of knowing a distance between the tool and the workpiece, as well has having a location to place the tool of the robot, and be able to implement it in the robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher. No inventive effort would have been required.
Regarding claim 20, Eldridge in view of Nemmers in further view of Fisher does not teach wherein the one or more robotic operations comprise: determining, based on the TCP calibration values, a contact tip to work distance (CTWD) associated with the weldhead and a part to be welded;
and generating one or more instructions to move the weldhead in relation to the part based on the CTWD.
Rossano, in the same field of endeavor, teaches wherein the one or more robotic operations comprise: determining, based on the TCP calibration values, a contact tip to work distance (CTWD) associated with the weldhead and a part to be welded (Rossano: Figure 3B, ¶ 0041, “Additionally, as indicated by FIG. 3B, the GUI 136 can also be configured to provide information regarding the relative distances between a second reference indicator 146, such as, for example, a reference location of the robot 106, subpart 132, and/or end effector 108, relative to a target indicator 148, such as, for example, a target location of or around the main part 134. For example, as illustrated, the GUI 136 can include a legend 150 or other graphical indications or representations of the distance along one or more axes of the coordinate system of the GUI 136 between second reference location 146 and target location 148 of the subpart 132, main part 134, and/or certain components of the robot 106. For example, as shown in FIG. 6, according to certain embodiments, the GUI 136 can display a distance along three axes of the GUI coordinate system of the distance between the second reference indicator 146, such as a TCP of the subpart 132 and/or robot 106, and the target indicator 148. The distance(s) depicted by the GUI 136 can correlate to one or more actual distance or reference distances.”. The cited passage and figure clearly teaches that the distance between the tip of the end effector and the part is determined.);
and generating one or more instructions to move the weldhead in relation to the part based on the CTWD (Rossano: ¶ 0041, ¶ 0047, “FIG. 7 illustrates an exemplary side view of a digital representation of a workpiece 160 by a GUI 136 on a display 124 that provides guidance in the form of a first path 164 the TCP 162 of the robot 106 is to travel during a work process on a workpiece. According to the depicted example, the robot 106 can be operating a cutting device in connection with the removal of material from the workpiece 160, among other types of possible work processes. The first path 164 can correspond to a predetermined path the TCP 162 of the robot 106, such as, for example a path the robot 106 is planned or selected to take either before or during the operation of the robot 106, and can be depicted by the GUI 136 on the display 124 in a variety of manners.”. One of ordinary skill in the art would see that in order for tool of the robot to perform a work process on a work piece the distance between the tool and the work piece determined in ¶ 0041 must be used.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the system for calibrating a tool center point (TCP) of a robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher with wherein the instructions, when executed by the processor, further cause the processor to: determine, based on the TCP calibration values, a contact tip to work distance (CTWD) associated with the weldhead and a part to be welded and generate one or more instructions to move the weldhead in relation to the part based on the CTWD taught in Rossano with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because it would have been obvious to try. To facilitate the automation of a robotic system configured to carry out a task, it is necessary to define the robot’s trajectory in space. This requires knowledge of the relative distance between the robot’s tool and the workpiece so that the robot can properly place the tool in order to properly carry out its task. A person of ordinary skill in the art would have had the technological capabilities and knowledge to recognize the necessity of knowing a distance between the tool and the workpiece and be able to implement it in the robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher. No inventive effort would have been required.
Claim(s) 6 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2009/0118864 A1 ("Eldridge") in view of US 20110029132 A1 ("Nemmers") in further view of US 2021/0065356 A1 ("Fisher") in further view of US 6603870 B1 ("Bascle").
Regarding claim 6, Eldridge in view of Nemmers in further view of Fisher teaches further comprising: annotating, with the controller, at least one of the multiple images to indicate a first location associated with the tip of the weldhead and a second location associated with the weldhead (Nemmers: Figure 12, ¶ 0064, “In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11.”, ¶ 0066, “ A tool center point 214 based upon the wire 202 can be created from the direction of the reference line R1, the origin point 206 and a pre-defined distance (e.g. 17 mm) associated with the robotic tool. The process is repeated for the wire 204 using the points 210, 212 to generate reference line R2 and tool center point 216.”. As can be seen from the cited passages, the image is annotated to include a first location associated with the tip of the weldhead (i.e. points 206 and 208) as well as a weldhead (i.e. points 210 and 212));
Eldridge in view of Nemmers in further view of Fisher does not teach defining, with the controller, a first plane in a first image of the multiple images based on the annotated first location;
defining, with the controller, a second plane in a second image of the multiple images based on the annotated second location;
and intersecting, with the controller, the first plane with the second plane to define a longitudinal axis of the weldhead.
Bascle, in the same field of endeavor, teaches defining, with the controller, a first plane in a first image of the multiple images based on the annotated first location (Bascle: Column 9 lines 47-67, “Apparatus for positioning or aligning a biopsy needle for proper insertion into the body of a patient from a selected point on a surface of the body, so as to enter in a straight line passing through a designated target region within the body, in conjunction with an imaging system utilizing radiation from a first source position for deriving a first radiographic image on a first image plane of a portion of the body including a first image of the selected point and a first image of the target region, the first source position, the first image of the selected point, and the first image of the target region defining a first viewing plane .pi.,…”);
Defining, with the controller, a second plane in a second image of the multiple images based on the annotated second location (Bascle: Column 9 lines 47-67, “the imaging system utilizing radiation from a second source position for deriving a second radiographic image on a second image plane of the portion of the body, including a second image of the selected point and a second image of the target region, the second source position, the second image of the selected point, and the second image of the target region define a second viewing plane .pi..sup.1,…”);
and intersecting, with the controller, the first plane with the second plane to define a longitudinal axis of the weldhead (Bascle: Column 9 line 47 – Column 10 line 24, “…measuring circle apparatus having a first position for establishing a first auxiliary plane at a first plane angle .THETA..sub.1 with respect to a selected set of coordinates and for constraining a pointer for moving rotatably about the selected point and within the first auxiliary plane to a first angle of inclination .phi..sub.1 relative to the set of coordinates such that a projection or extension of an image of the pointer on the first image plane passes through the first image of the target region; second measuring circle apparatus for establishing a second auxiliary plane at a second plane angle .THETA..sub.2 with respect to the selected set of coordinates, the second plane angle being different from the first plane angle such that the first and second auxiliary planes form an intersection line…”. The line created by the intersection of the two planes would define the longitudinal axis.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method for calibrating a tool center point (TCP) of a robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher with the method of defining an axis from the intersection of two planes in two images taught in Bascle with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination of the method for calibrating a tool center point (TCP) of a robotic welding system and image plane intersection method would have yielded the predictable result of obtaining the line defined by the intersection of the two planes. A person of ordinary skill in the art would have had the technological capabilities to have combine the method with the image plane intersection method. No inventive effort would have been required. Furthermore, even though the image plane intersection method uses the same point in both images, such a method could easily be configured to a different point in the second image and would not change the functionality of the method or introduce new functionality.
Regarding claim 15, Eldridge in view of Nemmers in further view of Fisher teaches wherein the controller is configured to: annotate at least one of the multiple images to indicate a first location associated with the tip of the weldhead and a second location associated with the weldhead (Nemmers: Figure 12, ¶ 0064, “In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11.”, ¶ 0066, “ A tool center point 214 based upon the wire 202 can be created from the direction of the reference line R1, the origin point 206 and a pre-defined distance (e.g. 17 mm) associated with the robotic tool. The process is repeated for the wire 204 using the points 210, 212 to generate reference line R2 and tool center point 216.”. As can be seen from the cited passages, the image is annotated to include a first location associated with the tip of the weldhead (i.e. points 206 and 208) as well as a weldhead (i.e. points 210 and 212));
identify, in each image of the multiple images, the tip of the weldhead and a location on the weldhead (Eldridge: Figure 11B, ¶ 0109, “FIG. 11B is an example image 1110 showing the sub-process for step 3 of refining the orientation 1116 of the tool 1102 by searching for the sides 1112 of the tool 1102 in the process for locating the TCP (1124 or 1126) of the tool 1102 on the camera image 1000. Step 3 to find a refined orientation 1116 of the tool 1102 of the process for finding the TCP (1124 or 1126) in the camera image 1000 is necessary because the neck of the torch tool 1102 may cause the fitted ellipse 1104 to have a slightly different orientation (i.e., rough orientation 1114) than the nozzle of the tool 1102.” Nemmers: ¶ 0064, “As more clearly shown in FIG. 10, while the robotic tool is positioned within the field of view of the image generating device, a portion of the robotic tool (i.e. parent tool) is initially located, as identified by an outline 205. In order to teach the user tool for the tandem welding tool 200, four points (i.e. targets) 206, 208, 210, 212 need to be located in three dimension space, as shown in FIG. 11. Specifically, while the robotic tool is in the first position, the processor 14 analyzes the first image to locate the first point 206 and an XYZ coordinate representing a position where a pre-defined view line passes through a calibration plane defined during the calibration step 102. As a non-limiting example, the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 are each disposed along a first three dimensional line. The processor 14 then repeats the analysis to locate points 208, 210, 212 in a fashion similar to that described above for the point 206.”, ¶ 0065, “The robotic tool is then rotated and the process is repeated to re-locate the points 206, 208, 210, 212 in the second image of the robotic tool. Specifically, the first three dimensional line passing through the XYZ coordinate of the view line intersection, the first point 206, and the known focal point of the image generating device 22 can be intersected with a second three dimensional line defined from the second image. The intersection of the first three dimensional line and the second three dimensional line associated with the point 206 represents an XYZ value of the point 206. As similar process is repeated to define the XYZ values for the points 208, 210, 212.”. The cited passage of Eldridge clearly teaches determining the longitudinal axis of the weldhead based on the location of the weldhead. The cited passages of Nemmers clearly teaches identifying the tip of the weld head and location of the weldhead in multiple images.).
Eldridge in view of Nemmers in further view of Fisher does not teach define a first plane in a first image of the multiple images based on the annotated first location;
define a second plane in a second image of the multiple images based on the annotated second location;
intersect the first plane with the second plane to define a longitudinal axis of the weldhead;
Bascle, in the same field of endeavor, teaches define a first plane in a first image of the multiple images based on the annotated first location (Bascle: Column 9 lines 47-67, “Apparatus for positioning or aligning a biopsy needle for proper insertion into the body of a patient from a selected point on a surface of the body, so as to enter in a straight line passing through a designated target region within the body, in conjunction with an imaging system utilizing radiation from a first source position for deriving a first radiographic image on a first image plane of a portion of the body including a first image of the selected point and a first image of the target region, the first source position, the first image of the selected point, and the first image of the target region defining a first viewing plane .pi.,…”);
define a second plane in a second image of the multiple images based on the annotated second location (Bascle: Column 9 lines 47-67, “the imaging system utilizing radiation from a second source position for deriving a second radiographic image on a second image plane of the portion of the body, including a second image of the selected point and a second image of the target region, the second source position, the second image of the selected point, and the second image of the target region define a second viewing plane .pi..sup.1,…”);
intersect the first plane with the second plane to define a longitudinal axis of the weldhead (Bascle: Column 9 line 47 – Column 10 line 24, “…measuring circle apparatus having a first position for establishing a first auxiliary plane at a first plane angle .THETA..sub.1 with respect to a selected set of coordinates and for constraining a pointer for moving rotatably about the selected point and within the first auxiliary plane to a first angle of inclination .phi..sub.1 relative to the set of coordinates such that a projection or extension of an image of the pointer on the first image plane passes through the first image of the target region; second measuring circle apparatus for establishing a second auxiliary plane at a second plane angle .THETA..sub.2 with respect to the selected set of coordinates, the second plane angle being different from the first plane angle such that the first and second auxiliary planes form an intersection line…”. The line created by the intersection of the two planes would define the longitudinal axis.);
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher with the method of defining an axis from the intersection of two planes in two images taught in Bascle with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination of the robotic welding system and image plane intersection method would have yielded the predictable result of obtaining the line defined by the intersection of the two planes. A person of ordinary skill in the art would have had the technological capabilities to have combine the robotic welding apparatus with the image plane intersection method. No inventive effort would have been required. Furthermore, even though the image plane intersection method uses the same point in both images, such a method could easily be configured to a different point in the second image and would not change the functionality of the method or introduce new functionality.
Claim(s) 9 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2009/0118864 A1 ("Eldridge") in view of US 20110029132 A1 ("Nemmers") in further view of US 10926414 B2 ("Huang") in further view of US 2021/0065356 A1 ("Fisher") in further view of US 20240083023 A1 ("Hamaya").
Regarding claim 9, wherein the robot arm is coupled to the weldhead, the method further comprising (Eldridge: ¶ 0006, “An embodiment of the present invention may further comprise a computerized method for calculating a tool frame tool center point relative to a wrist-frame of a robot for a tool attached at a wrist of the robot using a camera…”, ¶ 0033, “The robot manipulator is typically made up of two subsections, the body and arm 108 and the wrist 110. A tool 112 used by a robot 102 to perform desired tasks is typically attached at the wrist 110 of the robot manipulator 102.”, ¶ 0102, “FIGS. 9A-C show images 900,910,920 of example Metal-Inert Gas (MIG) welding torches. FIG. 9A is an example image of a first type 900 of a MIG welding torch tool. FIG. 9B is an example image of a second type 910 of a MIG welding torch tool. FIG. 9C is an example image of a third type 920 of a MIG welding torch tool.”):
Eldridge in view of Nemmers in further view of Fisher does not teach moving the robot arm from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position;
determining the second position with the controller while the robot arm is in the second state; and
based on the first position of the first frame of reference and the second position of the second frame of reference, generating a model using the controller and artificial intelligence to predict, based on movement of the robot arm, a position of the second frame of reference with respect to a position of the first frame of reference.
Huang, in the same field of endeavor, teaches moving the robot arm from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position (Huang: Figure 4, Column 6 lines 1-39, “As shown in FIG. 4, the controller 13 controls the robot R to move the tool center point TCP thereof to an initial point O in the image overlapping area IA of the first image sensor 11 and the second image sensor 12. The initial point 0 may be any point in the image overlapping area IA. Then, the controller 13 controls the robot R to move the tool center point TCP toward the first image central axis A from the initial point Oto the point Tl, as shown by the path PH. Afterward, the controller 13 controls the robot R to move the tool center point TCP toward the second image central axis B from the point Tl to the point T2, as shown by the path PH2. Similarly, the controller 13 controls the robot R to move the tool center point TCP toward the first image central axis A from the point T2 to the point T3, as shown by the path PH3. Then the controller 13 controls the robot R to move the tool center point TCP toward the second image central axis B from the point T3 to the point T4, as shown by the path PH4. Finally, the controller 13 controls the robot R to move the tool center point TCP toward the first image central axis A from the point T4 to the intersection point I, and records a first calibration point CP1 when the tool center point TCP overlaps the intersection point I.” Column 6 lines 56-60, “Afterward, the controller 13 controls the robot R to repeatedly move the tool center point TCP thereof between the first image central axis A and the second image central axis B, and records a second calibration point CP2 when the tool center point TCP overlaps the intersection point I.”, Column 6 lines 61-67, “Next, the controller 13 determines whether the number of the calibration points is greater or equal to 3. If the controller 13 determines that the number of the calibration points is less than 3, the controller 13 repeats the above steps to obtain and record the third calibration point CP3 until the controller 13 determines that the number of the calibration points is greater or equal to 3.”. As can be seen from the cited passages, the system is configured to move the TCP of the robot after it has been determined from a first state to a second state. Furthermore, one of ordinary skill in the art would see from the cited figure that the first and second reference frames would be at a different position than when first determined.);
determining the second position with the controller while the robot arm is in the second state (Huang: Column 10 lines 1-7, “Step S73: Controlling a robot R to repeatedly move the tool center point TCP of the tool T thereof between the first image central axis A and the second image central axis B. Step S74: Recording a calibration point including the coordinates of the joints J1-J 6 of the robot R when the tool center point TCP overlaps the intersection point I.”. The cited passage clear show that the system is configured to determine the second position of the TCP once it reaches the second state.).
Eldridge in view of Nemmers in further view of Fisher teaches wherein the robot arm is coupled to the weldhead, the method further comprising. Eldridge in view of Nemmers in further view of Fisher does not teach moving the robot arm from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position; determining the second position with the controller while the robot arm is in the second state. Huang teaches moving the robot arm from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position; determining the second position with the controller while the robot arm is in the second state. A person of ordinary skill in the art would have had the technological capabilities required to have combine the method taught in Eldridge in view of Nemmers in further view of Fisher with moving the robot arm from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position; determining the second position with the controller while the robot arm is in the second state taught in Huang. Furthermore, the method taught in Eldridge in view of Nemmers in further view of Fisher is already configured to rotate the robot tool and determine a position, so modifying the system such that the robot tools is moved from a first state to a second state and a position is determined at this second state would not change or introduce new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of wherein the robot arm is coupled to the weldhead, the method further comprising moving the robot arm from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position; determining the second position with the controller while the robot arm is in the second state.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method for calibrating a tool center point (TCP) of a robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher with moving the robot arm from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position; determining the second position with the controller while the robot arm is in the second state taught in Huang with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Eldridge in view of Nemmers in further view of Fisher in further view of Huang does not teach based on the first position of the first frame of reference and the second position of the second frame of reference, generating a model using the controller and artificial intelligence to predict, based on movement of the robot arm, a position of the second frame of reference with respect to a position of the first frame of reference.
Hamaya, in the same field of endeavor, teaches based on the first position of the first frame of reference and the second position of the second frame of reference, generating a model using the controller and artificial intelligence to predict, based on movement of the robot arm, a position of the second frame of reference with respect to a position of the first frame of reference (Hamaya: ¶ 0016, “A second aspect of the present disclosure is a robot model machine learning method including: preparing a robot model including a state transition model that, based on an actual value of a position and a posture of a robot at a certain time and an action command that can be given to the robot, calculates a predicted value of the position and the posture of the robot at a next time…”, ¶ 0066, “The robot control device 40 functions as a learning device that learns a robot model by machine learning. Furthermore, the robot control device 40 also functions as a control device that controls the robot 10 using the learned robot model.”, ¶ 0075, “As illustrated in FIG. 5, the robot model LM includes a state transition model DM that calculates a predicted value of the position and the posture of the robot 10 at the next time on the basis of the actual value (measurement value) of the position and the posture at a certain time and an action command (candidate value or determined value) that can be given to the robot 10, and an external force model EM that calculates a predicted value of an external force applied to the robot 10.”. The cited passages clearly teach that the method generates a model using machine learning to predicted a future position of the robot.).
Eldridge in view of Nemmers in further view of Fisher in further view of Huang teaches wherein the robot arm is coupled to the weldhead, the method further comprising moving the robot arm from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position; determining the second position with the controller while the robot arm is in the second state. Eldridge in view of Nemmers in further view of Fisher in further view of Huang does not teach based on the first position of the first frame of reference and the second position of the second frame of reference, generating a model using the controller and artificial intelligence to predict, based on movement of the robot arm, a position of the second frame of reference with respect to a position of the first frame of reference. Hamaya teaches based on the first position of the first frame of reference and the second position of the second frame of reference, generating a model using the controller and artificial intelligence to predict, based on movement of the robot arm, a position of the second frame of reference with respect to a position of the first frame of reference. A person of ordinary skill in the art would have had the technological capabilities required to have combine the method taught in Eldridge in view of Nemmers in further view of Fisher in further view of Huang with based on the first position of the first frame of reference and the second position of the second frame of reference, generating a model using the controller and artificial intelligence to predict, based on movement of the robot arm, a position of the second frame of reference with respect to a position of the first frame of reference taught in Hamaya. Furthermore, because the method taught in Hamaya predicts the future position as a whole, one of ordinary skill in the art would see that this method would also predict the future location of the first reference frame. Additionally, the robot model used by the machine learning algorithm can be easily modified to include the second reference frame taught in Eldridge in view of Nemmers in further view of Fisher in further view of Huang as it is associated with already know components of the robot model.. Therefore, the second reference frame in Eldridge in view of Nemmers in further view of Fisher in further view of Huang can be predicted using the method taught in Hamaya. This modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a method for calibrating a tool center point (TCP) of a robotic welding system wherein based on the first position of the first frame of reference and the second position of the second frame of reference, generating a model using the controller and artificial intelligence to predict, based on movement of the robot arm, a position of the second frame of reference with respect to a position of the first frame of reference.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method for calibrating a tool center point (TCP) of a robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher in further view of Huang with based on the first position of the first frame of reference and the second position of the second frame of reference, generating a model using the controller and artificial intelligence to predict, based on movement of the robot arm, a position of the second frame of reference with respect to a position of the first frame of reference taught in Hamaya. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Regarding claim 10, Eldridge in view of Nemmers in further view of Fisher does not teach further comprising: moving the robot from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position; and
predicting, based on a model, the second position of the second frame of reference while the robot is in the second state.
Huang, in the same field of endeavor, teaches further comprising: moving the robot from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position (Huang: Figure 4, Column 6 lines 1-39, “As shown in FIG. 4, the controller 13 controls the robot R to move the tool center point TCP thereof to an initial point O in the image overlapping area IA of the first image sensor 11 and the second image sensor 12. The initial point 0 may be any point in the image overlapping area IA. Then, the controller 13 controls the robot R to move the tool center point TCP toward the first image central axis A from the initial point Oto the point Tl, as shown by the path PH. Afterward, the controller 13 controls the robot R to move the tool center point TCP toward the second image central axis B from the point Tl to the point T2, as shown by the path PH2. Similarly, the controller 13 controls the robot R to move the tool center point TCP toward the first image central axis A from the point T2 to the point T3, as shown by the path PH3. Then the controller 13 controls the robot R to move the tool center point TCP toward the second image central axis B from the point T3 to the point T4, as shown by the path PH4. Finally, the controller 13 controls the robot R to move the tool center point TCP toward the first image central axis A from the point T4 to the intersection point I, and records a first calibration point CP1 when the tool center point TCP overlaps the intersection point I.” Column 6 lines 56-60, “Afterward, the controller 13 controls the robot R to repeatedly move the tool center point TCP thereof between the first image central axis A and the second image central axis B, and records a second calibration point CP2 when the tool center point TCP overlaps the intersection point I.”, Column 6 lines 61-67, “Next, the controller 13 determines whether the number of the calibration points is greater or equal to 3. If the controller 13 determines that the number of the calibration points is less than 3, the controller 13 repeats the above steps to obtain and record the third calibration point CP3 until the controller 13 determines that the number of the calibration points is greater or equal to 3.”. As can be seen from the cited passages, the system is configured to move the TCP of the robot after it has been determined from a first state to a second state. Furthermore, one of ordinary skill in the art would see from the cited figure that the first and second reference frames would be at a different position than when first determined.).
Eldridge in view of Nemmers in further view of Fisher teaches a method for calibrating a tool center point (TCP) of a robotic welding system. Eldridge in view of Nemmers in further view of Fisher does not teach further comprising: moving the robot from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position. Huang teaches further comprising: moving the robot from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position. A person of ordinary skill in the art would have had the technological capabilities required to have combine the method taught in Eldridge in view of Nemmers in further view of Fisher with further comprising: moving the robot from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position taught in Huang. Furthermore, the method taught in Eldridge in view of Nemmers in further view of Fisher is already configured to rotate the robot tool, so modifying the system such that the robot tools is moved from a first state to a second state would not change or introduce new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a method for calibrating a tool center point (TCP) of a robotic welding system further comprising: moving the robot from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method for calibrating a tool center point (TCP) of a robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher with further comprising: moving the robot from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position taught in Huang with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Eldridge in view of Nemmers in further view of Fisher in further view of Huang does not teach predicting, based on a model, the second position of the second frame of reference while the robot is in the second state.
Hamaya, in the same field of endeavor, teaches predicting, based on a model, the second position of the second frame of reference while the robot is in the second state (Hamaya: ¶ 0016, “A second aspect of the present disclosure is a robot model machine learning method including: preparing a robot model including a state transition model that, based on an actual value of a position and a posture of a robot at a certain time and an action command that can be given to the robot, calculates a predicted value of the position and the posture of the robot at a next time…”, ¶ 0066, “The robot control device 40 functions as a learning device that learns a robot model by machine learning. Furthermore, the robot control device 40 also functions as a control device that controls the robot 10 using the learned robot model.”, ¶ 0075, “As illustrated in FIG. 5, the robot model LM includes a state transition model DM that calculates a predicted value of the position and the posture of the robot 10 at the next time on the basis of the actual value (measurement value) of the position and the posture at a certain time and an action command (candidate value or determined value) that can be given to the robot 10, and an external force model EM that calculates a predicted value of an external force applied to the robot 10.”. The cited passages clearly teach that the method generates a model using machine learning to predicted a future position of the robot.).
Eldridge in view of Nemmers in further view of Fisher in further view of Huang teaches a method for calibrating a tool center point (TCP) of a robotic welding system further comprising: moving the robot from a first state at which the second frame of reference is determined to a second state such that the first frame of reference is at a first position and the second frame of reference is at a second position. Eldridge in view of Nemmers in further view of Fisher in further view of Huang does not teach predicting, based on a model, the second position of the second frame of reference while the robot is in the second state. Hamaya teaches predicting, based on a model, the second position of the second frame of reference while the robot is in the second state. A person of ordinary skill in the art would have had the technological capabilities required to have combine the method taught in Eldridge in view of Nemmers in further view of Fisher in further view of Huang with predicting, based on a model, the second position of the second frame of reference while the robot is in the second state taught in Hamaya. Furthermore, because the method taught in Hamaya predicts the future position as a whole, one of ordinary skill in the art would see that this method would also predict the future location of the second reference frame. Additionally, the robot model used by the machine learning algorithm can be easily modified to include the second reference frame taught in Eldridge in view of Nemmers in further view of Fisher in further view of Huang as it is associated with already know components of the robot model.. Therefore, the second reference frame determined in Eldridge in view of Nemmers in further view of Fisher in further view of Huang can be predicted using the method taught in Hamaya. This modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a method for calibrating a tool center point (TCP) of a robotic welding system wherein predicting, based on a model, the second position of the second frame of reference while the robot is in the second state.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method for calibrating a tool center point (TCP) of a robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher in further view of Huang with predicting, based on a model, the second position of the second frame of reference while the robot is in the second state taught in Hamaya. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2009/0118864 A1 ("Eldridge") in view of US 20110029132 A1 ("Nemmers") in further view of US 2021/0065356 A1 ("Fisher") in further view of US 2024/0009748 A1 ("Kotera").
Regarding claim 13, Eldridge in view of Nemmers in further view of Fisher does not teach further comprising: a fixture configured to hold the part to be welded.
Kotera, in the same field of endeavor, further comprising: a fixture configured to hold the part to be welded (Kotera: Figure 1 jig 50, ¶ 0020, “…a jig 50 that holds a target object W to be welded.”).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher with the fixture for holding the part to be welded taught in Kotera with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because it would have been obvious to one of ordinary skill in the art. It is common to secure a work piece in a fixture before performing an operation on said work piece in many arts, not just in robotic applications. This is done to prevent the work piece from moving during the operation, either caused by the operation itself or the robot coming into contact with the work piece. A person of ordinary skill in the art would have had the technological capabilities and knowledge to have combine a fixture to hold the work piece with the robotic welding system taught in Eldridge in view of Nemmers in further view of Fisher. No inventive effort would have been required.
Response to Arguments
Applicant’s arguments, see Pages 14-19, filed March 19th, 2026, with respect to the 35 U.S.C. § 101 rejection of claims 1-20 have been fully considered and are persuasive. Applicant has since amended independent claims 1, 11, and 17 to recite “performing one or more robotic operations with the robot arm based on the one or more TCP calibration values” and “cause one or more robotic operations to be performed by the robotic arm based on the one or more” respectively. The amended limitation clearly demonstrates an active control step of the system using the data generated by the abstract idea. Such a limitation shows clear integration into a practical application. Therefore, the 35 U.S.C. § 101 rejection of claims 1-20 has been withdrawn.
Applicant's arguments filed March 19th, 2026, have been fully considered but they are not persuasive.
Regarding Applicant’s Arguments on Pages 20-22, Applicant argues that the prior art on record does not teach the limitations of the amended independent claims 1, 11, and 17.
Specifically on Pages 20-22, Applicant argues that the combination of Eldridge in view of Nemmers in further view of Fisher does not teach the limitation “receiving, with the controller, multiple image from one or more sensors disposed on a robot arm of the robotic welding system” because one of ordinary skill in the art would not have modified the system taught in Eldridge in view of Nemmers with the imaging device being mounted on the robot as taught in Fisher because Eldridge and Nemmers teach away from such a modification, and that said modification would make them inoperable. The Examiner respectfully disagrees. The process of controlling and calibrating a robotic system using a camera mounted on the robot is a process that would be familiar to one of ordinary skill in the art. In such a case, the coordinate frame of the camera can be immediately determined at each time step as it is based on the configuration of the robot (i.e. the joint angles), the length from joint to joint (i.e. the length of each link), and the distances to the camera itself. All of these parameters are typically known or easily gathered using known sensors such as angle sensors. Additionally, the transformation from the camera frame to the robot base frame is merely the inverse of the transformation from the robot base frame to the camera frame. Furthermore, the transformation from the world coordinate frame to the robot base frame (and by extension, the transformation from the robot base frame to the world coordinate frame), is known. Typically, in the cases of the claimed invention and the prior, where the robot base frame is static and does not move (i.e. the base of the robot does not move or is fixed in place), the transformation from the world coordinate frame to the robot base frame is a constant. Therefore, because the transformation from the robot base frame to the camera frame is a function of the joint angles which are known, and the transformation from the world coordinate frame to the robot base frame is a constant, the extrinsic properties of the camera (i.e. the transformation from the world coordinate frame to the camera frame) are easily determined at every timestep as being a function of the joint angles and a known constant. In conclusion, because the extrinsic properties of the camera are based on the joint angles and known constants, a person of ordinary skill in the art would have the technological capabilities required to modify the system taught in Eldridge in view of Nemmers such that the camera(s) are mounted on the robot as taught in Fisher. Changing the location of the camera(s) would therefore not render the system taught in Eldridge in view of Nemmers inoperable. Furthermore, as long as the camera(s) have a sufficient view of the tool of the robotic system, the overall process taught in Eldridge in view of Nemmers would not change simply by virtue of the camera coordinate frame and extrinsic properties having been changed. Therefore, Eldridge in view of Nemmers in further view of Fisher clearly teaches the limitation “receiving, with the controller, multiple image from one or more sensors disposed on a robot arm of the robotic welding system”.
Specifically on Page 22. Applicant argues that the secondary reference Fisher is not analogous art. In response to applicant's argument that US 2021/0065356 A1 (“Fisher”) is nonanalogous art, it has been held that a prior art reference must either be in the field of the inventor’s endeavor or, if not, then be reasonably pertinent to the particular problem with which the inventor was concerned, in order to be relied upon as a basis for rejection of the claimed invention. See In re Oetiker, 977 F.2d 1443, 24 USPQ2d 1443 (Fed. Cir. 1992). In this case, Fisher is reasonably pertinent to the particular problem with which the inventor was concerned, specifically a sensor unit that is coupled to the weldhead. Fisher clearly teaches a camera that is coupled to the weldhead of a robot, and as such, is clearly reasonably pertinent to the particular problem with which the inventor was concerned. Therefore, Fisher is clearly analogous art.
Therefore, for the reasons stated above and in the 35 U.S.C. § 103 rejection section, the 35 U.S.C. § 103 of independent claims 1, 11, and 17 are maintained.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Noah W Stiebritz whose telephone number is (571)272-3414. The examiner can normally be reached Monday thru Friday 7-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached at (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.W.S./ Examiner, Art Unit 3658
/Ramon A. Mercado/Supervisory Patent Examiner, Art Unit 3658
Prosecution Insights