DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
This office action is in response to amendments filed 05/22/2026. Claims 1-7 are pending.
Applicant’s arguments and amendments to the claims with respect to interpretation of Claims 1-7 under 35 USC 112(f) have been fully considered and are persuasive. The interpretations of Claims 1-7 under 35 USC 112(f) have been withdrawn. (all claim elements invoking 112(f) have been removed)
Applicant’s arguments and amendments to the claims with respect to prior art rejections of Claims 1-7 under 35 USC 102/103 have been fully considered and are persuasive. The rejections of Claims 1-7 under 35 USC 102/103 have been withdrawn. However, upon further consideration, a new rejection is made in view of Martel et al ( US 20220097238, hereinafter Martel)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lager et al (US 20200164518, hereinafter Lager) in view of Martel et al (US 20220097238, hereinafter Martel).
Regarding Claim 1, Lager teaches:
A device configured to set a transport device coordinate system in a robot coordinate system set to a robot configured to carry out work on a workpiece (see at least “The control system 16 controls the sensor 24 and the conveyor member 18. In this example, the control system 16 also controls the robot 12. The control system 16 is configured to control the sensor 24, the conveyor member 18 and the robot 12 to carry out the calibration methods as described herein.” In par. 0058 and "The control system 16 comprises a data processing device 26 (e.g. a central processing unit, CPU) and a memory 28." in par. 0059) ,
the transport device coordinate system defining a transport direction of a transport device configured to transport the workpiece (see at least "the conveyor coordinate system X.sub.con" in par. 0059 and “The conveyor member 18, but not the sensor 24, is then moved in the movement direction 20 from the first operating position to a second operating position which is illustrated in FIG. 5b. When the conveyor member 18 is positioned at the second operating position, the sensor 24 detects a further position of the calibration marker 30 in the sensor coordinate system X.sub.sen.” in par. 0073) , the device comprising:
a first index representing a first index coordinate system and placed on the transport device so as to be transported by the transport device (see at least "the calibration marker 30" in par. 0072 and Figs. 5A-5B) ;
a second index on a known position in the robot coordinate system and representing a second index coordinate system (see at least " When the conveyor member 18 is positioned at the first operating position, the sensor 24 detects a position of the tool 22 and a position of the calibration marker 30 in the sensor coordinate system X.sub.sen. In other words, the transformation between the tool 22 and the sensor 24 and the transformation between the calibration marker 30 and the sensor 24 are measured." in par. 0072)
a camera configured to acquire first image data obtained by imaging the first index, and second image data obtained by imaging the first index transported by the transport device after imaging the first image data (see at least “The robot system 10 further comprises a sensor 24. The sensor 24 is a non-contact sensor and may for example be constituted by a 2D or 3D vision sensor (e.g. camera). A Cartesian sensor coordinate system X.sub.sen is associated with the sensor 24.” In par. 0047 and "The sensor 24 is placed outside the conveyor member 18 facing both the top of the conveyor member 18 and the robot 12." in par. 0072 and “The conveyor member 18, but not the sensor 24, is then moved in the movement direction 20 from the first operating position to a second operating position which is illustrated in FIG. 5b. When the conveyor member 18 is positioned at the second operating position, the sensor 24 detects a further position of the calibration marker 30 in the sensor coordinate system X.sub.sen.” in par. 0073);
a processor (see at least " The control system 16 comprises a data processing device 26 (e.g. a central processing unit, CPU) and a memory 28. A computer program is stored in the memory 28. In the first embodiment, the computer program may comprise program code" in par. 0059) configured:
acquire first position data indicating a three-dimensional position, with respect to the camera, of the first index coordinate system represented by the first index captured in the first image data, and second position data indicating a three-dimensional position, with respect to the camera, of the first index coordinate system represented by the first index captured in the second image data (see at least “When the conveyor member 18 is positioned at the first operating position, the sensor 24 detects a position of the tool 22 and a position of the calibration marker 30 in the sensor coordinate system X.sub.sen.” in par. 0072 and “The conveyor member 18, but not the sensor 24, is then moved in the movement direction 20 from the first operating position to a second operating position which is illustrated in FIG. 5b. When the conveyor member 18 is positioned at the second operating position, the sensor 24 detects a further position of the calibration marker 30 in the sensor coordinate system X.sub.sen.” in par. 0073;
determine the transport direction as a vector in the (see at least "From the data of the calibration marker 30 collected by the sensor 24 the movement direction 20 can be determined. The relative positions in space of the calibration marker 30 are used to find the movement direction 20. For example, a movement vector can be calculated and expressed in the robot coordinate system X.sub.base, X.sub.mi, X.sub.tool as follows. The positions of the calibration marker 30 in the sensor coordinate system X.sub.sen are subtracted to get a movement vector in the sensor coordinate system X.sub.sen." in par. 0074) ; and
set the transport device coordinate system in the robot coordinate system, based on the determined transport direction. (see at least "This movement vector can be transformed from the sensor coordinate system X.sub.sen to the robot coordinate system X.sub.base, X.sub.mi, X.sub.tool since the location of the robot 12 in the (now fixed) sensor coordinate system X.sub.sen has been measured. To get the final transformation between the tool 22 and the conveyor member 18, the position of the calibration marker 30 in the conveyor coordinate system X.sub.con may be known beforehand. Alternatively, it can e.g. be detected or set during the calibration." in par. 0074).
Lager does not appear to explicitly teach all of the following, but Martel does teach:
a second index placed on a known position in the robot coordinate system and representing a second index coordinate system (see at least " A check is preferably made whether one of the two markers respectively detected pair-wise had already been previously detected. The desired behavior of the fitter would be to respectively detect an already known marker and then a new marker. All the already detected markers are at least indirectly linked with one another in this manner and the link structure is successively expanded by a respective one marker. If both newly detected markers are still unknown, the processing can be rejected and the configuration can be continued after the detection of at least one other marker." in par. 0015 and “The templates 22 can also be designed for different objects than sensors 18, for example machine parts, or a generic template 22 can be attached to another object. Such objects are thus included in the visualization.” In par. 0043 and “One object marker 24 is attached per object to be localized, that is per sensor 18, but also per machine part, controller 20, or the like.” In par. 0045 and “At least one reference marker 28 is attached in the machine zone 10 in addition to the object markers 24. The reference markers 28 can be positioned as desired by the fitter. They contain a unique code, for example a 32 digit identification number (universally unique identification, UUID) to preclude confusion with other markers in the machine zone 10. The reference markers 28 serve as reference points. It is later determined in the visualization with reference to a reference marker 28 read from the proximity where the origin of the visualization is and which sensors 18 are in the environment.” In par. 0046 and “In a step S1, a first marker 24, 28 is read, the fitter is therefore prompted to direct the detection device to a marker 24, 28 to be read and to trigger an image recording of a camera, for example. It is of advantage at the start of the configuration for a reference marker 28 to be read first. This then forms the reference point or point of origin.” In par. 0048)
determine (see at least " In a step S2, a second marker 24, 28 is read. A pair of two markers has thus then been read, and indeed by choice of the fitter a pair of two object markers 24, of an object marker 24 and a reference marker 28, or of two reference markers 28 As already stated with respect to step S1, the fitter can be prompted at the first pair to choose at least one reference marker 28 so that there is a point of origin from the start. In later iterations, during the reading of further pairs, the detection device can require that a respective one of the read markers 24, 28 is already known to successively expand the link structure of the markers 24, 28 read during the configuration. Alternatively, two or even more initially separate link structures are generated that can then be joined together as soon as they overlap one another in at least one marker 24, 28 that has become known. " in par. 0049 and “In a step S3, a relationship between the two read markers 24, 28 is automatically determined. There is already an abstract relationship in that the two markers 24, 28 are read together and are now automatically referenced to one another. A graph can, for example, be produced with this relationship that will be explained below with reference to FIG. 6. The geometrical relationship between the two read markers 24, 28 should, however, also further be determined, that is a transformation or a path from the one marker 24, 28 to the other marker 24, 28. Different image evaluation processes can be used for this that are known per se and that will not be further explained here. A conclusion can, for example, be drawn, only in outline, from the size of a marker 24, 28 or its perspective distortion on the mutual distance and position.” In par. 0050 and “At the end of this configuration, the geometrical relationships between all the markers 24, 28 are known and thus all the sensors 18 and other objects such as the controller 20 with object markers 24 are localized.” In par. 0054)
Given that Lager already teaches identifying in images multiple specific parts of the robot for calibration (see par. 0072-0075), it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device taught by Lager to incorporate the teachings of Martel wherein a known marker position serves as the origin of the robot cell coordinate system and other marker positions are defined relative to the origin in a series of images, in order to arrive at performing the same control with the markers and camera of Lager. The motivation to incorporate the teachings of would be to Martel would be to reduce localization errors (see par. 0020)
Regarding Claim 2, Lager as modified by Martel (references to Lager) teaches:
The device of claim 1, further comprising
wherein the camera is configured to acquire the first image data obtained by imaging the first index and the second index, and the second image data obtained by imaging the transported first index and the second index, (see at least “When the conveyor member 18 is positioned at the first operating position, the sensor 24 detects a position of the tool 22 and a position of the calibration marker 30 in the sensor coordinate system X.sub.sen. In other words, the transformation between the tool 22 and the sensor 24 and the transformation between the calibration marker 30 and the sensor 24 are measured." in par. 0072 " The sensor 24 may alternatively or additionally detect a position of the tool 22 when the conveyor member 18 is positioned at the second operating position or at any other operating position. For this detection, the robot 12 may either be in the same pose as in FIG. 5a or in a different pose." in par. 0075)
wherein the processor is configured to further acquire third position data indicating a three-dimensional position, with respect to the camera, of the second index coordinate system represented by the second index captured in the first image data, and fourth position data indicating a three-dimensional position, with respect to the camera, of the second index coordinate system represented by the second index captured in the second image data (see at least "when the conveyor member 18 is positioned at the first operating position, the sensor 24 detects a position of the tool 22 and a position of the calibration marker 30 in the sensor coordinate system X.sub.sen. In other words, the transformation between the tool 22 and the sensor 24 and the transformation between the calibration marker 30 and the sensor 24 are measured. From these transformations, the transformation between the calibration marker 30 to the tool 22 can be found. The position of the tool 22 is detected for at least one pose of the robot 12 when the conveyor member 18 is positioned in the first operating position." in par. 0072 and “When the conveyor member 18 is positioned at the second operating position, the sensor 24 detects a further position of the calibration marker 30 in the sensor coordinate system X.sub.sen.” in par. 0073 and “The sensor 24 may alternatively or additionally detect a position of the tool 22 when the conveyor member 18 is positioned at the second operating position or at any other operating position. For this detection, the robot 12 may either be in the same pose as in FIG. 5a or in a different pose.” In par. 0075), and
wherein the processor is configured to determine the transport direction, further based on the third position data and the fourth position data. (see at least “The relative positions in space of the calibration marker 30 are used to find the movement direction 20. For example, a movement vector can be calculated and expressed in the robot coordinate system X.sub.base, X.sub.mi, X.sub.tool as follows. The positions of the calibration marker 30 in the sensor coordinate system X.sub.sen are subtracted to get a movement vector in the sensor coordinate system X.sub.sen. This movement vector can be transformed from the sensor coordinate system X.sub.sen to the robot coordinate system X.sub.base, X.sub.mi, X.sub.tool since the location of the robot 12 in the (now fixed) sensor coordinate system X.sub.sen has been measured. To get the final transformation between the tool 22 and the conveyor member 18, the position of the calibration marker 30 in the conveyor coordinate system X.sub.con may be known beforehand. Alternatively, it can e.g. be detected or set during the calibration.” In par. 0074 and "The sensor 24 may alternatively or additionally detect a position of the tool 22 when the conveyor member 18 is positioned at the second operating position or at any other operating position. For this detection, the robot 12 may either be in the same pose as in FIG. 5a or in a different pose.” In par. 0075)
Regarding Claim 3, Lager as modified by Martel (references to Lager) teaches:
The device of claim 2,
wherein the processor is configured to:
determine a first positional relationship between the first index coordinate system and the second index coordinate system in the first image data, based on the first position data and the third position data (see at least "A method for calibrating the robot coordinate system X.sub.base, X.sub.mi, X.sub.tool with the conveyor coordinate system X.sub.con according to the third embodiment will now be described. The sensor 24 is placed outside the conveyor member 18 facing both the top of the conveyor member 18 and the robot 12. In FIG. 5a, the conveyor member 18 is positioned at a first operating position. When the conveyor member 18 is positioned at the first operating position, the sensor 24 detects a position of the tool 22 and a position of the calibration marker 30 in the sensor coordinate system X.sub.sen. In other words, the transformation between the tool 22 and the sensor 24 and the transformation between the calibration marker 30 and the sensor 24 are measured. From these transformations, the transformation between the calibration marker 30 to the tool 22 can be found. The position of the tool 22 is detected for at least one pose of the robot 12 when the conveyor member 18 is positioned in the first operating position." in par. 0072) ;
determine a second positional relationship between the first index coordinate system and the second index coordinate system in the second image data, based on the second position data and the fourth position data (see at least "This movement vector can be transformed from the sensor coordinate system X.sub.sen to the robot coordinate system X.sub.base, X.sub.mi, X.sub.tool since the location of the robot 12 in the (now fixed) sensor coordinate system X.sub.sen has been measured. To get the final transformation between the tool 22 and the conveyor member 18, the position of the calibration marker 30 in the conveyor coordinate system X.sub.con may be known beforehand. Alternatively, it can e.g. be detected or set during the calibration." in par. 0074 and “The sensor 24 may alternatively or additionally detect a position of the tool 22 when the conveyor member 18 is positioned at the second operating position or at any other operating position. For this detection, the robot 12 may either be in the same pose as in FIG. 5a or in a different pose.” In par. 0075); and
determine the transport direction, based on the first positional relationship and the second positional relationship. (see at least "This movement vector can be transformed from the sensor coordinate system X.sub.sen to the robot coordinate system X.sub.base, X.sub.mi, X.sub.tool since the location of the robot 12 in the (now fixed) sensor coordinate system X.sub.sen has been measured. To get the final transformation between the tool 22 and the conveyor member 18, the position of the calibration marker 30 in the conveyor coordinate system X.sub.con may be known beforehand. Alternatively, it can e.g. be detected or set during the calibration." in par. 0074 and “The sensor 24 may alternatively or additionally detect a position of the tool 22 when the conveyor member 18 is positioned at the second operating position or at any other operating position. For this detection, the robot 12 may either be in the same pose as in FIG. 5a or in a different pose.” In par. 0075)
Regarding Claim 4, Lager as modified by Martel (references to Lager) teaches:
the device of claim 1, comprising:
wherein the processor is configured to determine the transport direction (see at least "This movement vector can be transformed from the sensor coordinate system X.sub.sen to the robot coordinate system X.sub.base, X.sub.mi, X.sub.tool since the location of the robot 12 in the (now fixed) sensor coordinate system X.sub.sen has been measured. To get the final transformation between the tool 22 and the conveyor member 18, the position of the calibration marker 30 in the conveyor coordinate system X.sub.con may be known beforehand. Alternatively, it can e.g. be detected or set during the calibration." in par. 0074 and “The sensor 24 may alternatively or additionally detect a position of the tool 22 when the conveyor member 18 is positioned at the second operating position or at any other operating position. For this detection, the robot 12 may either be in the same pose as in FIG. 5a or in a different pose.” In par. 0075)
wherein the camera is configured to acquire the first image data obtained by imaging the first index and the second index, and the second image data obtained by imaging the transported first index and the second index (see at least “When the conveyor member 18 is positioned at the first operating position, the sensor 24 detects a position of the tool 22 and a position of the calibration marker 30 in the sensor coordinate system X.sub.sen. In other words, the transformation between the tool 22 and the sensor 24 and the transformation between the calibration marker 30 and the sensor 24 are measured." in par. 0072 " The sensor 24 may alternatively or additionally detect a position of the tool 22 when the conveyor member 18 is positioned at the second operating position or at any other operating position. For this detection, the robot 12 may either be in the same pose as in FIG. 5a or in a different pose." in par. 0075),
Lager does not appear to explicitly teach all of the following, but Martel does teach:
wherein the processor is configured to:
further acquire third position data indicating a three-dimensional position, with respect to the camera, of the second index coordinate system represented by the second index captured in the first image data, and fourth position data indicating a three-dimensional position, with respect to the camera, of the second index coordinate system represented by the second index captured in the second image data, and wherein the processor is configured to perform configuration (see at least " The geometrical linking of the markers preferably takes place in that the movement of the detection device between the detections of different markers is monitored, in that at least two markers are detected at the same time, or in that detections are evaluated during the alignment of the detection apparatus from the one marker to the other marker. If the fitter detects two markers after one another and these two markers are then localized relative to the detection device, an error can result due to interim movements and a rotation of the detection device on the conclusion of the relative arrangement of the two markers to one another. This can be eliminated by calculation in that the movement of the detection apparatus is detected, for example using an inertial measurement unit (IMU). A conclusion can also be drawn on the movement by intermediate detections during the alignment of the one marker with the other marker, for instance using a method of optical flow or in that intermediate images are joined to one another. The situation is comparatively simple when both markers are detected in the same image; the discussed error then does not arise." in par. 0020 and “In a step S3, a relationship between the two read markers 24, 28 is automatically determined. There is already an abstract relationship in that the two markers 24, 28 are read together and are now automatically referenced to one another. A graph can, for example, be produced with this relationship that will be explained below with reference to FIG. 6. The geometrical relationship between the two read markers 24, 28 should, however, also further be determined, that is a transformation or a path from the one marker 24, 28 to the other marker 24, 28. Different image evaluation processes can be used for this that are known per se and that will not be further explained here.” In par. 0050 )
Given that Lager already teaches identifying in images multiple specific parts of the robot for calibration (see par. 0072-0075), it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device taught by Lager to incorporate the teachings of Martel wherein a known marker position serves as the origin of the robot cell coordinate system and other marker positions are defined relative to the origin in a series of images, in order to arrive at performing the same control with the markers and camera of Lager. The motivation to incorporate the teachings of would be to Martel would be to reduce localization errors (see par. 0020)
Regarding Claim 5, Lager as modified by Martel (references to Lager) teaches:
the device of claim 1,
wherein the second index includes a pattern or a shape of the robot representing a three-dimensional position of the second index coordinate system in a camera coordinate system set to the camera that images the image data. (see at least “The sensor may be of any type to detect positions of the robot in a non-contact manner. The sensor may for example be constituted by a vision sensor, such as a two dimensional (2D) or three dimensional (3D) vision sensor, e.g. camera.” In par. 0014 and " Thus, the sensor 24 is configured to detect positions of the robot 12 and to detect positions of the conveyor member 18." in par. 0072 and “The positions of the calibration marker 30 in the sensor coordinate system X.sub.sen are subtracted to get a movement vector in the sensor coordinate system X.sub.sen. This movement vector can be transformed from the sensor coordinate system X.sub.sen to the robot coordinate system X.sub.base, X.sub.mi, X.sub.tool since the location of the robot 12 in the (now fixed) sensor coordinate system X.sub.sen has been measured. To get the final transformation between the tool 22 and the conveyor member 18, the position of the calibration marker 30 in the conveyor coordinate system X.sub.con may be known beforehand. Alternatively, it can e.g. be detected or set during the calibration.” In par. 0074)
Regarding Claim 6, Lager as modified by Martel (references to Lager) teaches:
The device of claim 1,
wherein the first index includes a pattern representing a three-dimensional position of the first index coordinate system in a camera coordinate system set to the camera that images the image data. (see at least " A calibration marker 30 is provided on the conveyor member 18. The sensor 24 can thereby detect the position of the conveyor member 18 in the sensor coordinate system X.sub.sen. The calibration marker 30 may be any type of feature on the conveyor member 18 that is recognizable by the sensor 24. The calibration marker 30 may be permanently provided on the conveyor member 18 (e.g. a painted mark) or may be temporarily provided on the conveyor member 18 (e.g. an attached pyramid)." in par. 0062)
Regarding Claim 7, Lager as modified by Martel (references to Lager) teaches:
a method of setting a transport device coordinate system in a robot coordinate system set to a robot configured to carry out work on a workpiece, the transport device coordinate system defining a transport direction of a transport device configured to transport the workpiece, the method comprising: (see at least "According to one aspect, there is provided a method for calibrating a robot coordinate system of a robot with a conveyor coordinate system of a movable conveyor member." in par. 0011)
implementing, step by step, each function of the device of Claim 1 (see Claim 1 analysis for rejection of the device)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN M KATZ whose telephone number is (571)272-2776. The examiner can normally be reached Mon-Thurs. 8:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on (571) 270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN M KATZ/Primary Examiner, Art Unit 3657
Prosecution Insights