Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Should applicant desire to obtain the benefit of foreign priority under 35 U.S.C. 119(a)-(d) prior to declaration of an interference, a certified English translation of the foreign application must be submitted in reply to this action. 37 CFR 41.154(b) and 41.202(e).
Failure to provide a certified translation may result in no benefit being accorded for the non-English application.
Election/Restrictions
Applicant’s election of Group I, claims 1-4 in the reply filed on 05/29/2026 is acknowledged. Because applicant did not distinctly and specifically point out the supposed errors in the restriction requirement, the election has been treated as an election without traverse (MPEP § 818.01(a)).
Claims 5-41 are withdrawn from further consideration pursuant to 37 CFR 1.142(b) as being drawn to nonelected inventions, there being no allowable generic or linking claim. Election was made without traverse in the reply filed on 05/29/2026.
Status of Claims
This communication is a first office action, non-final rejection on the merits. Claims 1-4 as filed, are currently pending and have been considered below.
Specification
The disclosure is objected to because of the following informalities:
0039 (PGPUB 20230381877 version) “…includes a robot arm 210, which may be referred to as a second moving unit. The robot arm 120 is provided with…”: Possible typographical error leading to inconsistent labeling - the robot arm is identified as item 210 as well as 120. Examiner suggests, and will evaluate the specifications as if element number 120 is changed to 210.
Appropriate correction is required.
Claim Objections
Claim 1 is objected to because of the following informalities: improper placement of the conjunction “that”. Claim 1 contains multiple instances of the phrase “…image data obtained by that the imaging apparatus images…” Examiner suggests changing to, and will further evaluate the claim if as written as “…image data obtained by that images…”
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 1 is rejected under 35 U.S.C. 103 as being unpatentable over Lippiello et al. (V. Lippiello, B. Siciliano and L. Villani, "Eye-in-Hand/Eye-to-Hand Multi-Camera Visual Servoing," Proceedings of the 44th IEEE Conference on Decision and Control, Seville, Spain, 2005, pp. 5354-5359 hereinafter “Lippiello”) in view of Jeanty et all (U.S. Patent Application 20210337734 A1 hereinafter “Jeanty”).
Regarding Claim 1, Lippiello teaches in figures 2-4
A control apparatus that performs data processing for controlling a driver of a robot arm that is provided with an end effector , an imaging apparatus, and a camera, and that moves the end effector, the imaging apparatus, and the camera, wherein
Lippiello pertains to a robotic system to track a moving object with multiple cameras fixed to a robotic arm (known as “eye-in-hand”) and to move the end effector of the robotic arm to the object. Lippiello discloses the control apparatus in figure 4, and discloses the robot arm, end effector, cameras, and target object in figure 1. Lippiello teaches that a plurality of cameras can be attached to the robot by stating “Consider a system of nf video cameras fixed in the workspace (eye-to-hand cameras) and nm video cameras mounted on the end effector of one or more robots (eye-in-hand cameras), with n = nf +nm” (5354). Lippiello is silent on the specifics of the process that the end effector performs on the object, and if the multiple cameras are configured to be stereo cameras. However, Jeanty teaches an end effector that performs a process on a target object, complete with stereo cameras. Jeanty pertains to a harvesting robot and discloses in figure 5a a suction device (522), multiple fingers to grip (540), and multiple cameras (550). Jeanty discloses the use of stereo cameras by specifying “The end effector may further include one or more cameras 550 or other imaging devices. The cameras 550 include photodetectors configured to image an area below and/or laterally displaced from the end effector. In some embodiments, image data from two or more cameras 550 may be used for stereo imaging or other techniques for determining depth information of the imaged region” (0072). Therefore, it would have been known to one of ordinary skill in the art of robotic control to use the end effector of Jeanty to provide the missing details on the process of the end effector and camera configuration of the system disclosed by Lippiello in order to obtain more accurate depth information when grasping moving objects with a robotic end effector. Lippiello further discloses
the control apparatus comprises a processor, and the processor:
calculates a position and an attitude of the target object, on the basis of image data obtained by that the imaging apparatus images the target object with the stereo camera during such a relative movement of the robot arm and the target object that the imaging apparatus and the stereo camera are brought close to the target object, and shape data that are generated from image data obtained by imaging the target object with the stereo camera and that indicate a three-dimensional shape of the target object;
Lippiello discloses the calculation of the position and attitude of the target object in figure 3 and equation 2, specifying “Without loss of generality, the case of a single moving object is considered. The position and orientation of a frame attached to the object Oₒ–xₒyₒzₒ with respect to a base coordinate frame O–xyz can be expressed in terms of the coordinate vector of the origin oₒ = [xₒ yₒ zₒ]T and of the rotation matrix Rₒ (φₒ), where φₒ = [ϕₒ αₒ ψₒ ]T is the vector of the roll, pitch and yaw angles. The vector xₒ = [oₒT ϕₒT] defines a minimal representation of the object pose with respect to the base frame”(5355). Lippiello discloses the calculation of the shape data of the target object in figure 3 and equation 2, specifying “Consider s feature points of the object. The homogeneous coordinate vector p̃j =[xj yj zj 1]T of the feature point Pj (j = 1, . . . , s) can be expressed in the base frame as p̃j (xo) = Ho(xo) p̃oj, (2) “ (5355). Lippiello further discloses in figure 3 and equations 6-10
calculates a displacement of the target object between different time points , on the basis of two image data obtained by that the imaging apparatus that images the target object at the different time points during the relative movement, and two shape data that are generated from image data obtained by that the stereo camera that images the target object at the different time points and that indicate the three-dimensional shape of the target object at each of the different time points; and
Lippiello calculates a displacement of the target as a function of time via the motion tracking algorithms, specifying “The prediction of the object pose and velocity at the next sampling time, computed by the EKF, is input to an occlusion prediction algorithm that evaluates, for each camera, the set of visible object image features as well as an estimate of their projection on the image plane” (5357). Lippiello further discloses in figure 3 and equations 6, 16, and 18
outputs information about the position and the attitude of the target object calculated by adding the calculated displacement of the target object to the calculated position and attitude of the target object.
Lippiello calculates the target object position in equation 6, determines the displacement in terms of robot joint angles in equation 16, and generates an output signal to be used by the robot actuators in equation 18.
Regarding Claim 2, Lippiello in view of Jeanty discloses all the limitations of claim 1, and Lippiello discloses in figure 4 and section IV. Position-Based Visual Servoing wherein the control apparatus generates a control signal for controlling the driver of the robot arm on the basis of the information outputted from the processor, and outputs the generated control signal to a robot control unit that controls the driver of the robot arm. Lippiello discloses the control of the robotic arm based off of the information outputted from the eye-in-hand camera tracking algorithms, specifying “pose control is performed through an inner-outer control loop running at different frequency. The inner loop, running at the higher frequency (500 Hz in the experiments), implements motion control (independent joint control or any kind of joint space or task space control). In the outer loop, the block named dynamic trajectory planner computes the trajectory for the end effector on the basis of the current object pose and on the desired task” (5357).
Regarding Claim 3, Lippiello in view of Jeanty discloses all the limitations of claim 1, and Jeanty discloses in figure 5a wherein the end effector is at least one of a light irradiation apparatus that is configured to apply a processing light, a discharge apparatus that is configured to discharge a solder, and a holding apparatus that is configured to hold the target object. Jeanty discloses that the end effector includes the application of a processing light by detailing “In some embodiments, the robot controller 440 may be configured to control the lighting elements 4311, 4312, 4313 based at least in part on one or more light sensors located near the cameras 4301, 4312, 4303, and/or based on detected proximity to an object” (0052). Jeanty further discloses that the end effector includes a holding apparatus by detailing “the end effector 500 includes, among other components, a shaft 520, a carriage 530, a suction device such as a suction cup 522, and a plurality of fingers 540” (0074).
Regarding Claim 4, Lippiello in view of Jeanty discloses all the limitations of claim 1, and Jeanty discloses in figure 5a wherein the stereo camera includes two imaging apparatuses that are different from the imaging apparatus. As described in the rejection of claim 1 above, Lippiello discloses the use of multiple eye-in-hand cameras but is silent on their stereo configuration. Jeanty also discloses the use of multiple eye-in-hand cameras and discloses their stereo configuration by detailing “The end effector may further include one or more cameras 550 or other imaging devices. The cameras 550 include photodetectors configured to image an area below and/or laterally displaced from the end effector. In some embodiments, image data from two or more cameras 550 may be used for stereo imaging or other techniques for determining depth information of the imaged region” (0072).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Nathan Daniel Neckel whose telephone number is (571)272-9537. The examiner can normally be reached M-F, 7-3.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wade Miles can be reached at 571-270-7777. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NATHAN DANIEL NECKEL/Examiner, Art Unit 3656
/WADE MILES/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights