Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This Office action is in response to the amendment filed on 07/15/2026. Claims 9, 15, 21, and 27 are currently pending with claims 9 and 21 being amended.
Response to Amendment
The amendments to the claims submitted on 07/15/2026 overcome the claim objections set forth in the previous Office action except for those set forth in the claim objection section.
Response to Arguments
Examiner notes wherein Applicant argues the newly amended limitations, which have not been addressed by the prior art of record. As such, Examiner has augmented the below rejection(s) in view of the prior art of record to address the newly amended limitations.
The currently provided claim language does not provide the term posture with a definition beyond what is generally understood in the art. Therefore, posture will be interpreted to be both position and orientation. Further, the currently provided claim language does not require any specific weight be given to each of these components. Neither the claims nor the written disclosure describe how a degree of similarity may be identified/determined. Under the broadest reasonable interpretation of the claim language, this would encompass the selection of an object closest in position to the end effector, an object closest in orientation to an end effector, or any combination of these elements. The rejection below has been augmented in view of the amendments and in line with this interpretation of the claim limitations. Arimatsu et al. (US 20100004778 A1) discusses identifying a target object according to both the inclination and the height of the object (the orientation of the object relative to a normal surface and the relative distance of the object to the robotic hand).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 9 and 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Suzuki et al. (US 20130238124 A1), hereinafter Suzuki in view of Corkum et al. (US 20180126553 A1), hereinafter Corkum, Leroux et al. (US 7583835 B2), hereinafter Leroux, Shi et al. (US 20140163729 A1), hereinafter Shi, and Arimatsu et al. (US 20100004778 A1), hereinafter Arimatsu.
Regarding claim 9, Suzuki teaches:
9. (currently amended) An object grasping system, comprising:
a camera (Paragraphs 0031-0032, "The projector of the sensor unit 101 can irradiate a target object with uniform-luminance light. The camera of the sensor unit 101 senses an image of the target object irradiated with the uniform-luminance light, and outputs a two-dimensional image to the sensor information acquisition unit 111.
The sensor unit 102 includes a compact projector and a compact camera for sensing a two-dimensional image. The sensor unit 102 is fixed and mounted near the hand whose position and orientation can be controlled (changed) by the angle of each joint of the robot 100. The sensor unit 102 senses a target object gripped by the hand. Assume that the relative positional relationship between the projector and camera of the sensor unit 102 has been obtained in advance by calibration. Although the image processing unit 110 processes an image sensed by the sensor unit 102 in the embodiment, the sensor unit 102 may incorporate an image processing mechanism to output an image processing result.");
a grasping unit (Paragraph 0027, " A robot 100 is an articulated robot and operates in response to a control instruction from a robot controller unit 120. A hand serving as an end effector is mounted on the distal end of the robot 100 and can do work for a target object. In the embodiment, a hand with a chuck mechanism capable of gripping a target object is used as the end effector. The end effector may use a motor-driven hand or a suction pad for sucking a target object by air pressure."); and
a control unit (Paragraph 0027, " A robot 100 is an articulated robot and operates in response to a control instruction from a robot controller unit 120.") for moving the grasping unit toward -an object (See Figure 10, step S1006, the robot executes the instructions according to the work requested. This includes moving the end effector towards the target object. Paragraph 0063, "For example, if the robot work instruction unit 121 receives the position and orientation of a target object from the position and orientation measurement processing unit 113, it generates an instruction signal to move the hand to the position and orientation in which a target object in the received position and orientation can be gripped, and grip the target object.") … specifying a relative position of the object with respect to the grasping unit based on … taken by the camera (Paragraph 0044, "A plurality of cameras may be arranged by setting up a scaffold for image sensing, the user may hold a camera to sense an image, or a camera mounted on the robot may sense an image while moving the robot. Although an image may be sensed by any method, the relative position and orientation between the camera and the target object 103 in image sensing is obtained and stored in association with the sensed image. When a plurality of cameras are arranged on the scaffold, the relative position and orientation can be obtained from the shape of the scaffold. When the user holds a camera, the relative position and orientation can be obtained from a position and orientation sensor by mounting it on the camera. When the camera mounted on the robot senses an image, the relative position and orientation can be obtained using control information of the robot." See also Figures 4 and 10, which demonstrates that the process of sensing (imaging the environment) is done repeatedly until the object has been successfully gripped and the task is complete. This shows that sensor information is collected and then processed in order to determine information on the target object and determine if the robot has successfully grasped the target object. This is done within a loop which anticipates the continuous aspect of the claimed limitation. In figure 4, the process also includes, within the loop, the step S409 (“EXECUTE ROBOT WORK”) which demonstrates that this process is done as the grasping mechanism is brought closer to the target object in order to grasp it. Also see Paragraph 0028, “Assume that calibration work has been performed in advance by a well-known technique for the position and orientation of a sensor unit 101, the positions and orbits of the robot 100 and the hand, and the relative position and orientation between the arm of the robot 100 and a sensor unit 102. This makes it possible to convert the position and orientation of a target object in a pallet 104 that is measured by a position and orientation measurement processing unit 113, and the position and orientation of the target object measured by an extraction state measurement processing unit 115 into those in a work space coordinate system fixed in a space where the pallet 104 is placed. The robot 100 can also be controlled to move the hand to a position and orientation designated in the work space coordinate system.” Which discusses the sensor unit 102 that is used to capture information on the target object which can then be processed for control of the system. Also see Paragraphs 0160-0161, “In step S1002, the position and orientation measurement processing unit 913 obtains (measures) the position and orientation of at least one target object among a plurality of target objects in the sensed image received from the sensor information acquisition unit 911. The obtained position and orientation is used to predict whether extraction of a target object will succeed or to grip a target object by the robot. The processing in step S1002 is the same as that in step S402, and a description thereof will not be repeated.
After obtaining the position and orientation of the gripping candidate object, the position and orientation measurement processing unit 913 sends the obtained position and orientation to the prediction unit 914 and the target object selection unit 915. If the position and orientation measurement processing unit 913 obtains the positions and orientations of a plurality of gripping candidate objects, it sends all the obtained positions and orientations to the prediction unit 914 and the target object selection unit 915.”) as the grasping unit moves towards the object (See Figure 10, step S1006, the robot executes the instructions according to the work requested. This includes moving the end effector towards the target object. Paragraph 0063, "For example, if the robot work instruction unit 121 receives the position and orientation of a target object from the position and orientation measurement processing unit 113, it generates an instruction signal to move the hand to the position and orientation in which a target object in the received position and orientation can be gripped, and grip the target object.") … and
wherein said control unit is further configured to:
(i) detect a plurality of objects; (Paragraph 0070, “In step S402, the position and orientation measurement processing unit 113 obtains (measures) the position and orientation of at least one target object among a plurality of target objects in the sensed image received from the sensor information acquisition unit 111. The obtained position and orientation is used to extract and grip a target object by the robot.”)
(ii) identify a first object from the plurality of objects (Paragraph 0100, “In the embodiment, one target object is selected from a plurality of target objects whose positions and orientations have been measured by the robot work instruction unit 121. However, the position and orientation measurement processing unit 113 may select one target object and then output its position and orientation to the robot work instruction unit 121.” As well as Paragraph 0115, “In step S411, the target object selection unit 117 determines whether a target object to be gripped next has been selected in step S410. For example, in FIG. 7, the positions and orientations of the three target objects except for the target object 103' have been measured. However, if the position and orientation of even one target object has not been measured except for the target object 103', a target object to be gripped next cannot be selected. As a matter of course, when a target object matching the selection condition is not detected in step S410, a target object to be gripped next cannot be selected. These situations can be determined by referring to the position and orientation measurement result in step S402 and the gripping instruction list.”) … [[and]]
(iii) grasp and move the first object …
(Paragraphs 0118-0119, “In step S412, the target object selection unit 117 sends, to the gripping instruction unit 118, the position and orientation of the selected target object (a position and orientation selected as the position and orientation of a gripping candidate among the positions and orientations obtained in step S402). The gripping instruction unit 118 sends the position and orientation of the selected target object to the robot work instruction unit 121.
The robot work instruction unit 121 generates an instruction signal to move the hand of the robot 100 to the position and orientation received from the gripping instruction unit 118, and then sends the generated instruction signal to the robot control unit 122. The gripping instruction unit 118 registers the selected target object in the gripping instruction list. Thereafter, the process returns to step S404.” Please also see Paragraph 0111)
Suzuki does not specifically disclose the camera capturing images of the workspace continuously for alterations to the trajectory, determining a distance between the camera and the object based on focal length and image data, or comparing the orientation and position of the grasper with that of the target object and identifying a degree of priority for the grasping of different objects according to values describing the environment including the object and end effector. However, Corkum, in the same field of endeavor of robotic control, teaches:
… while repeatedly … a plurality of continuous images … such that movement of the grasping arm may be adjusted accordingly (Paragraph 0017, "For example, the controller 160 can: locate an object reference frame—including an object coordinate system—relative to a target object identified in an image recorded from the camera 150; orient the object reference frame in real space relative to the end effector 140 based on the position and orientation of the target object identified in the image and based on a known offset between the camera 150 and the end effector 140; project the preplanned trajectory into the object reference frame; implement closed-loop controls to move the end effector 140 along the preplanned trajectory toward the terminus of the preplanned trajectory at which the end effector 140 may accurately engage the target object; refine the location and orientation of the object reference coordinate system as the arm moves the end effector 140 and the camera 150 closer to the target object (which may yield a higher-resolution image of the target object); and repeat this process regularly—such as at a rate of 2 Hz, 20 Hz, or for every 10-millimeter interval traversed by the end effector 140—until the end effector 140 engages the target object. By thus realigning the preplanned trajectory to the target object (e.g., to an object feature or constellation of object features) detected in the field of view of the camera 150 as the end effector 140 approaches the target object, the system 100 can achieve increased locational accuracy of the end effector 140 relative to the target object as the end effector 140 nears the target object while also accommodating wide variances in the location and orientation of the target object from its expected location and orientation and/or accommodating wide variances in the location and orientation of one unit of the target object to a next unit of the target object.")
However, Leroux, in the same field of endeavor of robotic grasping, teaches:
and calculating a three dimensional distance between the object and camera based on a focal length of the camera and an occupied ratio of an image area corresponding to the target object with respect to an entire image area; (Column 3 Line 45 – Column 4 Line 12, “We can distinguish the relationship: .DELTA..sub.m=T.sub.p.DELTA..sub.p where .DELTA.m and T.sub.p are expressed in mm, .DELTA..sub.p is the movement in pixels of the object between the two images (without dimensions) and T.sub.p represents the size of a pixel on the image of the camera 2. We can thus calculate the coordinates of the object 4: Z=F(D/(.DELTA..sub.pT.sub.p)-1)
Where
Z is the focal length of the camera 2 at the surface of the object 4;
F is the focal length of the lens (in mm);
D is the movement of the camera 2 between the taking of two images (in mm);
T.sub.p is the size of a pixel of the CCD sensor (in mm).
We can then deduce the other coordinates of the object 4: X=X.sub.pT.sub.p(Z+F)/F and Y=Y.sub.pT.sub.p(Z+F)/F, where
X is the abscissa of the object in the camera indicator (in mm);
Y is the ordinate of the object in the camera indicator (in mm);
X.sub.p is the abscissa of the object in the image (in pixels, therefore without dimension);
Y.sub.p is the ordinate of the object in the image (in pixels, therefore without dimension);
The method may be applied to chains of three or more images. Each image has characteristic points that may be paired with a point or no points of the previous image and the following image. If a point can be found by pairing on all of the images of a chain, it is more likely that it truly belongs to the object 4, which increases the reliability of the method.” Please also see figure 1 which demonstrates that this system is operating in a three dimensional environment and the arm has multiple degrees of freedom therefor the distance is a three dimensional distance. Examiner further notes that any distance may be understood to be a three dimensional distance even if one or two of the dimensions have a value of zero.)
However, Shi, in the same field of endeavor of robotics, teaches:
… [[the]] a current posture of the grasping unit; (Paragraph 0096, "The method 300 commences 301 and flow of the algorithm proceeds to block 302 whereat the computer processor obtains data regarding the grasping device 100. The present algorithm for planning a grasp considers the grasping device data, along with object data, to determine how the grasping device characteristics match up with object dimensions and pose, for determining how to position the grasping device for grasping the object.") …
However, Arimatsu, in the same field of endeavor of robotic grasp planning, teaches:
… that has a posture most similar to (Paragraph 0032, “A target detecting part 22a of image processing part 22 of robot controller 20 processes an image captured by video camera 18 and detects a plurality of objects. A target selecting part 22b selects an object as a target to be picked among the plurality of objects, according to the procedure as described below. The procedure based on size information of the image of the detected object, a degree of coincidence (or a degree of matching) between the image of the detected object and the feature model, a degree of inclination of the image of the detected object in relation to the feature model, presence or absence of hiding or overlapping of the object by another object, or the combination of at least two of them. For example, as the simplest case, the highest degree of priority of picking is given to the object having the largest detected size, whereby the object positioned at the top layer within the container is picked first. Or, the object having the higher degree of matching compared to the feature model may be judged to be more correctly detected. Further, by taking both the detected size and the degree of matching into consideration, the object, which is positioned at the highest level and correctly detected, is preferentially or firstly picked. Then, as described below, a view line direction calculating part 22c calculates the direction of a view line of the camera toward the selected object, and a target height estimating part 22d estimates the position and the height of the selected object based on the size information of the image of the object. Next, a grip correction calculating part 22e calculates an amount of correction of movement which is required for the robot so as to grip the object. A data storing part 22f may store a teaching model and/or data regarding a feature model for calculating the degree of matching and/or the degree of inclination.”) … before the remaining plurality of objects; (Paragraph 0036, "Among the above values, some of the values, which may be effective for calculating the degree of priority of picking the object, are selected and multiplied each other to obtain the degree of priority. By sequentially picking the object in descending order of the degree of priority, the efficient picking operation may be carried out.") and
(iv) repeating steps (i)-(iii) until all objects are grasped and moved. (Please see Figure 4 which demonstrates the iterative approach of the process.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic grasping system as taught by Suzuki with the visual servoing methods as taught by Corkum as well as the ability to determine a relative position of the object with respect to the camera using focal length and image data (pixel size) as taught by Leroux. Further, Shi teaches a method of comparing the pose of the end effector with the pose of the object to be grasped in order to determine how closely they match and how to adjust the grasping device for an effective grasp. It would be obvious to incorporate this processing into the method of selecting a gripping candidate as taught by Suzuki in order to select a candidate requiring the least movement before grasping. As it is well known in the art to plan a grasp based on the pose of the end effector and based on the pose of a plurality of target objects and to further prioritize different objects in the order of grasping, it would be obvious to one of ordinary skill in the art to combine these concepts to prioritize grasping of an object which is closest to the end effectors pose wherein the pose of the object and the end effector describes the position and orientation as disclosed by Arimatsu. Using this combined method, the system would be capable of determining the position of objects even if partially occluded and to identify an optimal candidate for the next grasping action. This would ensure highly accurate control of the robotic system and allow the system to react efficiently to unexpected movement or changes in the workspace or to the workpiece position/orientation.
Regarding claim 21, Suzuki teaches:
21. (currently amended) An object grasping method, comprising:
shooting with a camera a plurality of continuous images (Paragraph 0141, "The sensor unit 901 includes a projector and a camera for sensing a two-dimensional image as two-dimensional information. The sensor unit 901 is fixed above the pallet 904, senses an image of heaped target objects 903, and outputs the sensed image to a sensor information acquisition unit 911. Although an image processing unit 910 processes the image sensed by the sensor unit 901 in the embodiment, the sensor unit 901 may incorporate an image processing mechanism to output an image processing result. Assume that the relative positional relationship between the projector and camera of the sensor unit 901 has been obtained in advance by calibration." And Paragraphs 0160-0161, “In step S1002, the position and orientation measurement processing unit 913 obtains (measures) the position and orientation of at least one target object among a plurality of target objects in the sensed image received from the sensor information acquisition unit 911. The obtained position and orientation is used to predict whether extraction of a target object will succeed or to grip a target object by the robot. The processing in step S1002 is the same as that in step S402, and a description thereof will not be repeated.
After obtaining the position and orientation of the gripping candidate object, the position and orientation measurement processing unit 913 sends the obtained position and orientation to the prediction unit 914 and the target object selection unit 915. If the position and orientation measurement processing unit 913 obtains the positions and orientations of a plurality of gripping candidate objects, it sends all the obtained positions and orientations to the prediction unit 914 and the target object selection unit 915.” also see Figure 10, step S1001 and the loop that this processing is a part of. This demonstrates that as the robot executes commands to grip the target object, the system continues to sense the object and environment and use that information to determine control of the robot as long as the end has not been reached.) to detect a plurality of objects; (Paragraph 0070, “In step S402, the position and orientation measurement processing unit 113 obtains (measures) the position and orientation of at least one target object among a plurality of target objects in the sensed image received from the sensor information acquisition unit 111. The obtained position and orientation is used to extract and grip a target object by the robot.”)
identifying a first object from the plurality of objects (Paragraph 0100, “In the embodiment, one target object is selected from a plurality of target objects whose positions and orientations have been measured by the robot work instruction unit 121. However, the position and orientation measurement processing unit 113 may select one target object and then output its position and orientation to the robot work instruction unit 121.” As well as Paragraph 0115, “In step S411, the target object selection unit 117 determines whether a target object to be gripped next has been selected in step S410. For example, in FIG. 7, the positions and orientations of the three target objects except for the target object 103' have been measured. However, if the position and orientation of even one target object has not been measured except for the target object 103', a target object to be gripped next cannot be selected. As a matter of course, when a target object matching the selection condition is not detected in step S410, a target object to be gripped next cannot be selected. These situations can be determined by referring to the position and orientation measurement result in step S402 and the gripping instruction list.”) …
repeatedly identifying a relative three-dimensional position of the first object with respect to the grasping unit based on the plurality of captured images (Paragraph 0044, "A plurality of cameras may be arranged by setting up a scaffold for image sensing, the user may hold a camera to sense an image, or a camera mounted on the robot may sense an image while moving the robot. Although an image may be sensed by any method, the relative position and orientation between the camera and the target object 103 in image sensing is obtained and stored in association with the sensed image. When a plurality of cameras are arranged on the scaffold, the relative position and orientation can be obtained from the shape of the scaffold. When the user holds a camera, the relative position and orientation can be obtained from a position and orientation sensor by mounting it on the camera. When the camera mounted on the robot senses an image, the relative position and orientation can be obtained using control information of the robot." And Paragraphs 0160-0161, “In step S1002, the position and orientation measurement processing unit 913 obtains (measures) the position and orientation of at least one target object among a plurality of target objects in the sensed image received from the sensor information acquisition unit 911. The obtained position and orientation is used to predict whether extraction of a target object will succeed or to grip a target object by the robot. The processing in step S1002 is the same as that in step S402, and a description thereof will not be repeated.
After obtaining the position and orientation of the gripping candidate object, the position and orientation measurement processing unit 913 sends the obtained position and orientation to the prediction unit 914 and the target object selection unit 915. If the position and orientation measurement processing unit 913 obtains the positions and orientations of a plurality of gripping candidate objects, it sends all the obtained positions and orientations to the prediction unit 914 and the target object selection unit 915.” See also Figure 10, which demonstrates that the process of sensing (imaging the environment) is done repeatedly until the object has been successfully gripped and the task is complete. Also see Paragraph 0024, “In the first embodiment, the positions and orientations of target objects heaped in a pallet are measured using the first sensor (a projector and camera) for acquiring two-dimensional information (a two-dimensional image) and three-dimensional information (a range image or a two-dimensional image for obtaining three-dimensional point group data) about target objects.”);
moving the grasping unit toward the first object (See Figure 10, step S1006, the robot executes the instructions according to the work requested. This includes moving the end effector towards the target object. Paragraph 0063, "For example, if the robot work instruction unit 121 receives the position and orientation of a target object from the position and orientation measurement processing unit 113, it generates an instruction signal to move the hand to the position and orientation in which a target object in the received position and orientation can be gripped, and grip the target object." And Paragraphs 0160-0161, “In step S1002, the position and orientation measurement processing unit 913 obtains (measures) the position and orientation of at least one target object among a plurality of target objects in the sensed image received from the sensor information acquisition unit 911. The obtained position and orientation is used to predict whether extraction of a target object will succeed or to grip a target object by the robot. The processing in step S1002 is the same as that in step S402, and a description thereof will not be repeated.
After obtaining the position and orientation of the gripping candidate object, the position and orientation measurement processing unit 913 sends the obtained position and orientation to the prediction unit 914 and the target object selection unit 915. If the position and orientation measurement processing unit 913 obtains the positions and orientations of a plurality of gripping candidate objects, it sends all the obtained positions and orientations to the prediction unit 914 and the target object selection unit 915.” also see Figure 10, step S1001 and the loop that this processing is a part of. This demonstrates that as the robot executes commands to grip the target object, the system continues to sense the object and environment and use that information to determine control of the robot as long as the end has not been reached. Also see at least paragraph 0073, “In the embodiment, the coarse position and orientation (represented by a six-dimensional vector s) of the target object to be measured is repeatedly corrected by an iterative operation using the Gauss-Newton method, which is a kind of nonlinear optimization method, so that the three-dimensional geometric model is fitted in the sensed image. Note that the optimization method for obtaining the position and orientation of a target object is not limited the Gauss-Newton method.”) … ; and
grasping the first object. (Paragraph 0044, "A plurality of cameras may be arranged by setting up a scaffold for image sensing, the user may hold a camera to sense an image, or a camera mounted on the robot may sense an image while moving the robot. Although an image may be sensed by any method, the relative position and orientation between the camera and the target object 103 in image sensing is obtained and stored in association with the sensed image. When a plurality of cameras are arranged on the scaffold, the relative position and orientation can be obtained from the shape of the scaffold. When the user holds a camera, the relative position and orientation can be obtained from a position and orientation sensor by mounting it on the camera. When the camera mounted on the robot senses an image, the relative position and orientation can be obtained using control information of the robot." See also Figure 10, which demonstrates that the process of sensing (imaging the environment) is done repeatedly until the object has been successfully gripped and the task is complete.)
Suzuki does not specifically disclose the camera capturing images of the workspace continuously for alterations to the trajectory, determining a distance between the camera and the object based on focal length and image data, or comparing the orientation and position of the grasper with that of the target object and identifying a degree of priority for the grasping of different objects according to values describing the environment including the object and end effector. However, Corkum, in the same field of endeavor of robotic control, teaches:
… based on the continuously identified relative position of the first object such that movement of the grasping unit may be adjusted accordingly (Paragraph 0017, "For example, the controller 160 can: locate an object reference frame—including an object coordinate system—relative to a target object identified in an image recorded from the camera 150; orient the object reference frame in real space relative to the end effector 140 based on the position and orientation of the target object identified in the image and based on a known offset between the camera 150 and the end effector 140; project the preplanned trajectory into the object reference frame; implement closed-loop controls to move the end effector 140 along the preplanned trajectory toward the terminus of the preplanned trajectory at which the end effector 140 may accurately engage the target object; refine the location and orientation of the object reference coordinate system as the arm moves the end effector 140 and the camera 150 closer to the target object (which may yield a higher-resolution image of the target object); and repeat this process regularly—such as at a rate of 2 Hz, 20 Hz, or for every 10-millimeter interval traversed by the end effector 140—until the end effector 140 engages the target object. By thus realigning the preplanned trajectory to the target object (e.g., to an object feature or constellation of object features) detected in the field of view of the camera 150 as the end effector 140 approaches the target object, the system 100 can achieve increased locational accuracy of the end effector 140 relative to the target object as the end effector 140 nears the target object while also accommodating wide variances in the location and orientation of the target object from its expected location and orientation and/or accommodating wide variances in the location and orientation of one unit of the target object to a next unit of the target object.") …
However, Leroux, in the same field of endeavor of robotic grasping, teaches:
calculating a distance between the first object and camera based on a focal length of the camera and an occupied ratio of an image area corresponding to the first object with respect to an entire image area; (Column 3 Line 45 – Column 4 Line 12, “We can distinguish the relationship: .DELTA..sub.m=T.sub.p.DELTA..sub.p where .DELTA.m and T.sub.p are expressed in mm, .DELTA..sub.p is the movement in pixels of the object between the two images (without dimensions) and T.sub.p represents the size of a pixel on the image of the camera 2. We can thus calculate the coordinates of the object 4: Z=F(D/(.DELTA..sub.pT.sub.p)-1)
Where
Z is the focal length of the camera 2 at the surface of the object 4;
F is the focal length of the lens (in mm);
D is the movement of the camera 2 between the taking of two images (in mm);
T.sub.p is the size of a pixel of the CCD sensor (in mm).
We can then deduce the other coordinates of the object 4: X=X.sub.pT.sub.p(Z+F)/F and Y=Y.sub.pT.sub.p(Z+F)/F, where
X is the abscissa of the object in the camera indicator (in mm);
Y is the ordinate of the object in the camera indicator (in mm);
X.sub.p is the abscissa of the object in the image (in pixels, therefore without dimension);
Y.sub.p is the ordinate of the object in the image (in pixels, therefore without dimension);
The method may be applied to chains of three or more images. Each image has characteristic points that may be paired with a point or no points of the previous image and the following image. If a point can be found by pairing on all of the images of a chain, it is more likely that it truly belongs to the object 4, which increases the reliability of the method.” Please also see figure 1 which demonstrates that this system is operating in a three dimensional environment and the arm has multiple degrees of freedom therefor the distance is a three dimensional distance. Examiner further notes that any distance may be understood to be a three dimensional distance even if one or two of the dimensions have a value of zero.)
However, Shi, in the same field of endeavor of robotics, teaches:
… the posture of a grasping unit used to grasp the objects; (Paragraph 0096, "The method 300 commences 301 and flow of the algorithm proceeds to block 302 whereat the computer processor obtains data regarding the grasping device 100. The present algorithm for planning a grasp considers the grasping device data, along with object data, to determine how the grasping device characteristics match up with object dimensions and pose, for determining how to position the grasping device for grasping the object.") …
However, Arimatsu, in the same field of endeavor of robotic grasp planning, teaches:
… that has a current posture most similar to (Paragraph 0032, “A target detecting part 22a of image processing part 22 of robot controller 20 processes an image captured by video camera 18 and detects a plurality of objects. A target selecting part 22b selects an object as a target to be picked among the plurality of objects, according to the procedure as described below. The procedure based on size information of the image of the detected object, a degree of coincidence (or a degree of matching) between the image of the detected object and the feature model, a degree of inclination of the image of the detected object in relation to the feature model, presence or absence of hiding or overlapping of the object by another object, or the combination of at least two of them. For example, as the simplest case, the highest degree of priority of picking is given to the object having the largest detected size, whereby the object positioned at the top layer within the container is picked first. Or, the object having the higher degree of matching compared to the feature model may be judged to be more correctly detected. Further, by taking both the detected size and the degree of matching into consideration, the object, which is positioned at the highest level and correctly detected, is preferentially or firstly picked. Then, as described below, a view line direction calculating part 22c calculates the direction of a view line of the camera toward the selected object, and a target height estimating part 22d estimates the position and the height of the selected object based on the size information of the image of the object. Next, a grip correction calculating part 22e calculates an amount of correction of movement which is required for the robot so as to grip the object. A data storing part 22f may store a teaching model and/or data regarding a feature model for calculating the degree of matching and/or the degree of inclination.”) …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic grasping system as taught by Suzuki with the visual servoing methods as taught by Corkum as well as the ability to determine a relative position of the object with respect to the camera using focal length and image data (pixel size) as taught by Leroux. Further, Shi teaches a method of comparing the pose of the end effector with the pose of the object to be grasped in order to determine how closely they match and how to adjust the grasping device for an effective grasp. It would be obvious to incorporate this processing into the method of selecting a gripping candidate as taught by Suzuki in order to select a candidate requiring the least movement before grasping. As it is well known in the art to plan a grasp based on the pose of the end effector and based on the pose of a plurality of target objects and to further prioritize different objects in the order of grasping, it would be obvious to one of ordinary skill in the art to combine these concepts to prioritize grasping of an object which is closest to the end effectors pose wherein the pose of the object and the end effector describes the position and orientation as disclosed by Arimatsu. Using this combined method, the system would be capable of determining the position of objects even if partially occluded and to identify an optimal candidate for the next grasping action. This would ensure highly accurate control of the robotic system and allow the system to react efficiently to unexpected movement or changes in the workspace or to the workpiece position/orientation.
Claim(s) 15 and 27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Suzuki in view of Corkum, Leroux, Shi, and Arimatsu and in further view of Shekhawat and in further view of Xi et al. (US 20200094414 A1), hereinafter Xi and Bradski et al. (US 20160089791 A1), hereinafter Bradski.
Regarding claim 15, where all the limitations of claim 9 are discussed above, Suzuki does not specifically teach a detection unit and a wireless tag that is connected to a specification for the object and allow the system to identify a relative position of the object. However, Xi, in the same field of endeavor of robotic systems with manipulators for object grasping, teaches:
15. (original) The object grasping system according to claim 9, further comprising:
a detection unit for detecting a wireless tag (Paragraph 0064, "At procedure 630, the controller 430 may process the captured image to obtain the information of the product. In certain embodiments, the captured image may include more than one of the e-package information tags 200 on different surfaces of the object 100, such that the controller 430 may obtain the thorough information for the robotic manipulation for handling the object. Once the information of the product is obtained, at procedure 640, the controller 430 may control a robotic grasping device 420 to perform a robotic manipulation for handling the object 100 based on the information of the product."), wherein the control unit identifies a specification of the object based on information from the wireless tag attached to the object by using the detection unit (Paragraph 0012, "In certain embodiments, the first area of each of the e-package information tags is designed for easy detection and robust pose estimation, and the second area of each of the e-package information tags stores the information of the physical features of the object, where the physical features of the object includes: a dimension of the object; a weight of the object; a weight distribution of the object; a property of the object; product information of the object; and the location and the orientation of the object."), and identifies a relative position of the object with respect to the grasping unit based on the specification. (See Figure 6, step 620, the information about the product (such as location and orientation as seen in paragraph 0012) is obtained and step 640 which controls the robotic grasping device to interact with the object. This implies that the system is able to determine the relative position of the object so that the distance can be covered and the grasping device may interact with the object.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic gripping system and method as taught by Suzuki with the e-package information tags and ability to read those as taught by Xi. This would allow the system to be informed on physical features of the object as well as the location and orientation of the object and thereby perform operations more effectively. The system would be able to grasp in a more ideal position based on the weight distribution of the object for example.
Regarding claim 27, where all the limitations of claim 21 are discussed above, Suzuki does not specifically teach a detection unit and a wireless tag that is connected to a specification for the object and allow the system to identify a relative position of the object. However, Xi, in the same field of endeavor of robotic systems with manipulators for object grasping, teaches:
27. (previously presented) The object grasping method according to claim 21, further comprising:
detecting a wireless tag via a detection unit (Paragraph 0064, "At procedure 630, the controller 430 may process the captured image to obtain the information of the product. In certain embodiments, the captured image may include more than one of the e-package information tags 200 on different surfaces of the object 100, such that the controller 430 may obtain the thorough information for the robotic manipulation for handling the object. Once the information of the product is obtained, at procedure 640, the controller 430 may control a robotic grasping device 420 to perform a robotic manipulation for handling the object 100 based on the information of the product.");
identifying, via a control unit a specification of the object based on information from the wireless tag attached to the object by using the detection unit (Paragraph 0012, "In certain embodiments, the first area of each of the e-package information tags is designed for easy detection and robust pose estimation, and the second area of each of the e-package information tags stores the information of the physical features of the object, where the physical features of the object includes: a dimension of the object; a weight of the object; a weight distribution of the object; a property of the object; product information of the object; and the location and the orientation of the object."), and identifying a relative position of the object with respect to the grasping unit based on the specification. (See Figure 6, step 620, the information about the product (such as location and orientation as seen in paragraph 0012) is obtained and step 640 which controls the robotic grasping device to interact with the object. This implies that the system is able to determine the relative position of the object so that the distance can be covered and the grasping device may interact with the object.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic gripping system and method as taught by Suzuki with the e-package information tags and ability to read those as taught by Xi. This would allow the system to be informed on physical features of the object as well as the location and orientation of the object and thereby perform operations more effectively. The system would be able to grasp in a more ideal position based on the weight distribution of the object for example.
Conclusion
The Examiner has cited particular paragraphs or columns and line numbers in the referencesapplied to the claims above for the convenience of the Applicant. Although the specified citations arerepresentative of the teachings of the art and are applied to specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested of the Applicant in preparing responses, to fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner. See MPEP 2141.02 [R-07.2015] VI. A prior art reference must be considered in its entirety, i.e., as a whole, including portions that would lead away from the claimed Invention. W.L. Gore & Associates, Inc. v. Garlock, Inc., 721 F.2d 1540, 220 USPQ 303 (Fed. Cir. 1983), cert, denied, 469 U.S. 851 (1984). See also MPEP §2123.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
/H.J.K./Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657
