Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to because
Figure 2 has multiple elements labeled 222 and no elements labeled 242. The specification states “Figure 2 shows an example setup for calibrating a robotic system 200 including a robotic manipulator 221 (e.g. an implementation of the robotic manipulator 121 described in previous examples). In accordance with such examples, the robotic manipulator 221 includes a robotic arm 220, an end effector 222, and a motion subsystem 230”(pg 8) and “For example, the calibration object 242 is a fiducial marker, as shown in Figure 2” (pg 10).
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 14 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non- statutory subject matter. The claim(s) does/do not fall within at least one of the four categories (process, machine, manufacture, or composition of matter) of patent eligible subject matter because the claim is directed as a “program” that is reasonably interpreted as a software, and software claims are not patentable. Based on the provided claim language, the claimed “program” is not supported by hardware such as tangible computer storage medium or execution engine, which would enable one skill in the art to construe that the apparatus is built from tangible product to carry out any functionality being conveyed from the claim. Thus, the “program” is software per se and therefore is not being embodied in a manner as to be executable. Therefore, Applicant is suggested to amend claim 14 to recite for example, “A non-transitory computer readable storage medium having program code comprising the programming instructions to carry out the method of claim 1”.
Accordingly, appropriate correction and/or clarification is requested. Claim 14 will be examined in light of prior art as if the suggested corrections have been made.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-12, and 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Xing et al. (S. Xing, F. Jing and M. Tan, "Reconstruction-Based Hand–Eye Calibration Using Arbitrary Objects," in IEEE Transactions on Industrial Informatics, vol. 19, no. 5, pp. 6545-6555, May 2023 hereinafter “Xing”) in view of Yang et all (Z. Yang, L. Gong and C. Liu, "Efficient TCP Calibration Method for Vision Guided Robots Based on Inherent Constraints of Target Object," in IEEE Access, vol. 9, pp. 8902-8911, 2021 hereinafter “Yang”).
Regarding Claim 1, Xing discloses
A method of calibrating a camera mounted on a robotic manipulator comprising an end effector, the method comprising:
obtaining a plurality of images of a single calibration object installed at a location in a workspace of the robotic manipulator, wherein the images are captured by the camera at different respective configurations of the robotic manipulator;
processing the images to obtain respective first object positions of the calibration object relative to the camera;
Xing pertains to an eye-in-hand robot and the use of point cloud registration to solve for the hand-eye transformation in the traditional form AX=XB. Xing discloses the robot in figures 1 and 6, and discloses the method by stating “Consider a serial manipulator with a rigidly mounted 3-D sensors making a series of n motions from the first robot pose to (n + 1)th robot pose. A measurement in form of point cloud of a stationary object is taken at each robot poses” (6546). The object positions obtained by Xing are referred to as the first object positions. Xing does not teach contacting the calibration object with the robotic end effector to obtain an additional point cloud. However, Yang teaches
obtaining respective second object positions of the calibration object relative to the end effector based on a determined location of the calibration object relative to the robotic manipulator;
Yang pertains to an eye-in-hand robot and the use of tool center point calibration (TCP) to solve for the hand-eye transformation. Yang discloses the eye-in-hand version of the robot in figure 6, and discloses obtaining the object positions of the calibration object by contacting the calibration object by stating “Determining [pi]TB is called object calibration. This is done after hand-eye calibration and TCP calibration are completed, by manually moving the robot to make it brushes against each target point pi to derive [p’i]B from the robot controller. Then, one transfers [p’i]B to [pi]TB” (8904). The object positions obtained by Yang are referred to as the second object positions. When combined Xing in view of Yang discloses
performing point-cloud registration based on a first point cloud comprising the first object positions and a second point cloud comprising the second object positions to determine a hand-eye transformation.
Xing specifies “All the captured point clouds are transformed to the robot base frame by left multiplying the first initial guess of hand–eye relation X then corresponding robot poses Ai, where both X and Ai are homogeneous transformation matrices in the special Euclidean group SE(3)”(6546). Yang specifies determining a hand-eye transformation based the second object positions in pg 8904.
It would have been obvious to one having ordinary skill in the art at the time of applicant's effective filing date to combine the second object positions of Yang with the method of Xing because physical object positions provide redundancy to compensate for any errors associated with imaging systems, such as lens distortion.
Furthermore when combining Xing in view of Yang, a point cloud registration would be performed on the second object positions of Yang to determine the hand-eye transformation in a similar fashion as the point cloud registration of the first positions of Xing.
Regarding Claim 2, Xing in view of Yang discloses all the limitations of claim 1 and Yang further discloses
wherein obtaining the respective second object positions comprises:
positioning the end effector at the location of the calibration object in the workspace to determine a third object position relative to the robotic manipulator; and
determining the second object positions based on the third object position and respective robot-hand transformations corresponding to the respective configurations of the robotic manipulator.
It is understood by those of ordinary skill in the art of robotic coordinate frame transformations that when a tool at the end of a robotic end effector contacts a calibration object, a TCP calibration method is the conventional method to solve for the hand-eye transformation. Yang details this in section III. Background Knowledge: A. Conventional TCP Calibration Method (pg 8904). Yang refers to the third object position as the invariant coordinate of the fixed point in the coordinate frame of the robot base, as defined in equation 2. Yang refers to the second object positions as the collection of points G(∙) in the coordinate frame of the robot base, as defined by equations 3 and 4. Yang refers to the robot-hand transformation as a base-wrist transformation, as defined in equation 1.
Regarding Claim 3, Xing in view of Yang discloses all the limitations of claim 1 and Xing further discloses wherein the plurality of images comprises a set of at least four images in tables I and III.
Regarding Claim 4, Xing in view of Yang discloses all the limitations of claim 1 and Xing further discloses wherein processing the images comprises performing at least one of line detection, rectangle detection, or object detection to detect the calibration object in each of the images. Xing refers to object detection in terms of removing the background from the image, further specifying the image processing as “For each of the arbitrary objects, eight point clouds were captured at eight different robot poses and down sampled using a voxel grid filter with a leaf size of 0.001 m. The background plane was segmented out; then, the calibration was carried out using ICP variant and different numbers of processed point clouds” (6549).
Regarding Claim 5, Xing in view of Yang discloses all the limitations of claim 4 and Xing further discloses wherein processing the images comprises performing position estimation based on the detected calibration object in each of the images to determine the respective first object positions. Xing refers to the respective first object positions in terms of initial values, further specifying the x,y,z position estimate as “For each of the arbitrary calibration objects, 100 random initial values of hand–eye relation (with a random error of up to 50 mm for each of x, y, and z in position and a random error of up to 10◦ in rotation) were utilized to run the algorithm with all eight point clouds” (6553).
Regarding Claim 6, Xing in view of Yang discloses all the limitations of claim 1 and Xing further discloses wherein the single calibration object is installed at an unknown orientation in the workspace in figure 6. Xing teaches how arbitrary objects in arbitrary orientations can be used as calibration objects to determine the hand-eye transformation, with the calibration rig or Rubik being the most analogous to the present application.
Regarding Claim 7, Xing in view of Yang discloses all the limitations of claim 1 and Xing further discloses wherein performing the point-cloud registration comprises assigning to each of the first object positions a corresponding independent feature vector. Xing discloses the point-cloud registration process in algorithm 1 on pg 6548 with the independent feature vector specified as {(pij,qij)}.
Regarding Claim 8, Xing in view of Yang discloses all the limitations of claim 7 and Xing further discloses wherein performing the point-cloud registration comprises performing point-cloud registration based on feature matching using the independent feature vectors. Xing discloses the point-cloud registration process in algorithm 1 on pg 6548 with the feature matching specified as ||ΔX||≥ϵ2.
Regarding Claim 9, Xing in view of Yang discloses all the limitations of claim 1 and Xing further discloses wherein the calibration object comprises a fiducial marker. Xing refers to fiducial markers in terms of a custom checkerboard calibration rig by stating “A similar 12 × 9 checkerboard with side length of 10 mm (±0.01mm) was used as accurate calibration rig to carry out the comparison experiment, shown in Fig. 6(a)” (6549). Additionally, the calibration object of Yang comprises a fiducial marker (Tag B) as seen in figure 1.
Regarding Claim 10, Xing in view of Yang discloses all the limitations of claim 9 and Yang further discloses wherein the fiducial marker comprises one of an: ARTag, AprilTag, ArUco or STag marker. Xing is silent on the details if the checkerboard fiducial marker used is a common open-source tag. However, Yang discloses the specifics of such open-source fiducial markers by stating “Since it is usually tricky for the camera to detect the robot and target object directly, visual fiducial tags such as AprilTag [2] and RUNE-Tag [3] are often employed in practical situations. Figure 1 shows a typical VGR system. Fiducial tags A and B are attached to the robot and the target object, respectively” (8902). Therefore, it would have been obvious to one having ordinary skill in the art at the time of applicant’s invention to have the calibration rig based on open-source fiducial markers such as: ARTag, AprilTag, ArUco or STag marker.
Regarding Claim 11, Xing in view of Yang discloses all the limitations of claim 1 and Yang further discloses wherein the first and second object positions comprise respective positions of a centre-point of the calibration object. Yang specifies the centre-point calculation for the second object positions in EQ 3. Xing is silent on the centre-point calculation for the first object positions. However, such calculations are known to those of ordinary skill in the art of robotic coordinate transformation given the point-cloud information of the calibration rig, spherical orange, or Rubik cube, as disclosed by Xing. Therefore, it would have been obvious to one having ordinary skill in the art at the time of applicant’s effective filing date to have the first and second object positions comprise respective positions of a centre-point of the calibration object.
Regarding Claim 12, Xing in view of Yang discloses all the limitations of claim 1 and Xing further discloses comprising storing the hand-eye transformation for transforming coordinates of a detected object in a first coordinate system of the camera to a second coordinate system of the robotic manipulator or end effector. Xing discloses storing the hand-eye transformation in algorithm 1 on pg 6548 with the stored transformation specified as RX.
Regarding Claim 14, Xing in view of Yang discloses all the limitations of claim 1 and Xing further discloses A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claim 1 by detailing “Processing multiple point clouds simultaneously and iteratively can be computationally expensive, so we evaluated the complexity from a runtime perspective. All algorithms were coded with C++ and Point Cloud Library into a single-core implementation on a Dell Precision 7540 notebook featuring an Intel Core i7 9750H CPU and 16 GB of memory” (6552).
Regarding Claim 15, Xing in view of Yang discloses all the limitations of claim 14 and Xing further discloses A computer-readable data carrier having stored thereon the computer program of claim 14 by detailing “Processing multiple point clouds simultaneously and iteratively can be computationally expensive, so we evaluated the complexity from a runtime perspective. All algorithms were coded with C++ and Point Cloud Library into a single-core implementation on a Dell Precision 7540 notebook featuring an Intel Core i7 9750H CPU and 16 GB of memory” (6552).
Regarding Claim 16, Xing in view of Yang discloses all the limitations of claim 1 and Xing further discloses A controller for a robotic manipulator, wherein the controller is configured to perform the method of claim 1 by detailing “Processing multiple point clouds simultaneously and iteratively can be computationally expensive, so we evaluated the complexity from a runtime perspective. All algorithms were coded with C++ and Point Cloud Library into a single-core implementation on a Dell Precision 7540 notebook featuring an Intel Core i7 9750H CPU and 16 GB of memory” (6552).
Regarding Claim 17, Xing in view of Yang discloses
A system comprising:
a robotic manipulator having an end effector and a mounted camera;
In figure 6 Xing discloses a JAKA 6 DOF robot with a mounted Mechmind sensor. In figure 6 Yang discloses a JAKA Zu 7 2.0 6-axis robot with a mounted camera.
a single calibration object installed at a location in a workspace of the robotic manipulator; and
In figure 6 Xing discloses a calibration object in the workspace of the robotic manipulator. In figure 6 Yang discloses a fiducial tag in the workspace of the robotic manipulator.
a controller configured to:
Xing discloses the use of a controller by stating “All algorithms were coded with C++ and Point Cloud Library into a single-core implementation on a Dell Precision 7540 notebook featuring an Intel Core i7 9750H CPU and 16 GB of memory” (6552). Xing in view of Yang discloses
obtain and process a plurality of images of the single calibration object, captured by the mounted camera at different respective configurations of the robotic manipulator, to obtain respective first object positions of the calibration object relative to the camera;
obtain respective second object positions of the calibration object relative to the end effector based on a determined location of the calibration object relative to the robotic manipulator; and
perform point-cloud registration based on a first point cloud comprising the first object positions and a second point cloud comprising the second object positions to determine a hand-eye transformation.
rejected in a similar fashion to the rejected claim 1 above.
Claims 13, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Xing in view of Yang, further in view of Ulrich (U.S. patent application publication 20220383547 hereinafter “Ulrich”).
Regarding Claim 13, Xing in view of Yang discloses all the limitations of claim 1, which pertains to the hand-eye transformation of an eye-in-hand robot system. If the camera is not attached to the robot, but is static, the robot is referred to as an eye-to-hand robot system. It is known to those of ordinary skill in the art of robotic transformations that the solution to the well-known hand-eye transformation AX=XB applies to both eye-in-hand and eye-to-hand configurations. This is further supported by figures 1 and 2 and equations 4 and 8 of Ulrich.
Hence, Xing discloses
obtaining and processing an image of the single calibration object, , to determine a first pose of the calibration object relative to the static camera;
determining a second pose of the calibration object in a reference frame of the robotic manipulator based on another image of the single calibration object, captured by the camera mounted on the robotic manipulator, and the hand-eye transformation; and
determining a pose of the static camera, in the reference frame of the robotic manipulator, based on the first and second poses of the calibration object.
but fails to teach an eye-to-hand configuration. However, Ulrich teaches the interchangeability of eye-in-hand and eye-to-hand configurations for hand-eye transformations. Thus one of ordinary skill in the art could have substituted the eye-in-hand configuration for the eye-to-hand configuration, and the results of the substitution of the solution to the hand-eye transformation AX=XB would have been predictable.
Regarding Claim 18, Xing in view of Yang discloses all the limitations of claim 17, and the remainder of claim 18 is rejected in a similar manner as claim 13 above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
J. Heller, D. Henrion and T. Pajdla, "Hand-eye and robot-world calibration by global polynomial optimization," 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, 2014, pp. 3157-3164. The present office action refers to hand-eye transformations in the standard AX=XB form. There are other known forms of other transformations. Heller pertains to the object-hand transformation in the AX=ZB form.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Nathan Daniel Neckel whose telephone number is (571)272-9537. The examiner can normally be reached M-F, 7-3.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wade Miles can be reached at 571-270-7777. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NATHAN DANIEL NECKEL/Examiner, Art Unit 3656
/WADE MILES/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights