DETAILED ACTION
This communication is a Non-Final Office Action on the Merits. Claims 1-14 as originally filed are currently pending and have been considered as follows:
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The lengthy specification has not been checked to the extent necessary to determine the
presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of
which applicant may become aware in the specification.
The specification is objected to because of the following informalities:
“so as not to interface the robot with the surroundings” in ¶17 should read “so as not to interfere the robot with the surroundings”
“deice” in ¶26 should read “device”
“griping” in ¶30 & ¶ ¶49 should read “gripping”
“interface” in [Appendix 1] should read “interfere”
“interface” in [Appendix 13] should read “interfere”
Appropriate correction is required.
Claim Objections
Claim(s) 1, 3, 5, 6, 7, 13, and 14 are objected to because of the following informalities:
“so as not to interface the robot with the surroundings,” in Claim 1 should read “so as not to interfere the robot with the surroundings,”
“capture an image of a three-dimensional image including the object and the hand” in Claim 3 should read “capture a three-dimensional image including the object and the hand”
“corresponding to one type of objects with a same shape” in Claim 5 should read "corresponding to one type of object having a same shape"
“stores shapes of a plurality types of objects” in Claim 6 should read “stores shapes of a plurality of types of objects”
“based on the output pre-gripping hand shape and the output post-gripping hand shape” in Claim 7 should read “based on the outputted pre-gripping hand shape and the outputted post-gripping hand shape”
“so as not to interface the robot with surroundings” in Claim 13 should read “so as not to interfere the robot with surroundings”
“computer readable” in Claims 13 & 14 should read “computer-readable”
“based on the image information from the image capturing device;” in Claim 13 should read “based on the image information from an image capturing device;”
“from a predetermined first position to a take-out position of the object based on the pre-gripping hand shape;” in Claim 14 should read “from a predetermined first position to the take-out position of the object based on the pre-gripping hand shape;”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 7, 11, and 14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 7 recites that the take-out position calculation unit is configured to specify “a type of the object from among the plurality of objects stored in the shape storage unit.” However, claim 6, from which claim 7 depends, recites that the shape storage unit stores “shapes of a plurality types of objects” and corresponding pre-gripping and post-gripping hand shapes. Thus, claim 6 indicates that the shape storage unit stores object shapes, object-type information, and corresponding hand-shape information, not the actual plurality of objects. It is therefore unclear whether “the plurality of objects stored in the shape storage unit” refers to actual objects, object types, object-shape data, or stored records corresponding to object types. Therefore, claim 7 is rejected.
Claim 11 recites that the operation route generation unit is configured to “grip changes in the shape and arrangement position of the housing vessel.” The term “grip changes” is unclear in this context. The operation route generation unit is a processing/control unit, and “grip” ordinarily connotes physically holding or grasping an object. It is unclear whether the claim requires the operation route generation unit to detect changes, recognize changes, obtain change information, account for changes, or generate a route based on such changes. These possible interpretations have materially different scopes. Therefore, claim 11 is rejected.
Claim 14 recites that “the process of generating the operation route of the hand includes: a pre-gripping route generation process ...; and a post-gripping route generation process ... based on the post-gripping hand shape is included.” This construction is grammatically unclear because the claim first states that the process “includes” listed processes, but then ends the second listed process with the phrase “is included.” It is unclear what subject is “included,” and it is unclear whether the process of generating the operation route includes both the pre-gripping route generation process and the post-gripping route generation process, or whether only the post-gripping route generation process is being separately stated as included. Therefore, claim 14 is rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-6 and 10-14 are rejected under 35 U.S.C. 103 as being unpatentable over Truebenbach (US Pub. No. 20190389062) in view of Komoda (US Pub. No. 20190283249).
As per Claim 1, Truebenbach discloses a method for robotic bin picking, comprising:
controlling a robot to take out a bulk-loaded object by the robot including a hand (as per Fig. 19, as per “the robot may be controlled to physically select the first candidate object. If the feasibility is not validated, at least one of a different grasping point of the first candidate object, a second path, or a second candidate object may be selected” in Abstract, as per Fig. 28)
a take-out position calculation unit configured to calculate a take-out position of an object to be captured by the robot based on image information from an image capturing device that captures an image including the object; (as per “robotic bin picking process 10 may identify 200 a list of candidate workpieces or objects to be picked up… Metrics may include likelihood of a successful pick, likelihood of a successful place, and/or suitability for placement in a particular location. As discussed above and in some embodiments, bin picking system (e.g., bin picking system 64) may include a scanning system (e.g., one or more sensors and/or scanners) configured to identify parts in a bin” in ¶161, as per “a graphical user interface that allows for validation of workpiece detection is provided. The user may validate the workpiece configuration by adding parts to the bin then triggering a scan and detection to find matches. The detection results may be rendered and displayed in a list” in ¶114, as per “f the part needs to be inspected by a camera after it is picked, and the inspection result determines whether the part is placed in e.g., place position 1 or e.g., place position 2. In order to guarantee successful motion, validation logic of robotic bin picking process 10 may confirm both of these alternatives before the part can be moved.” in ¶176)
a spatial information storage unit configured to store a movable range in which the robot is movable and an interference range in which the robot interferes with surroundings in the movable range; (as per “the user may select the Environment tab to configure the workspace obstacles. In this tab the user can load, create, edit, and/or save the set of shapes that define all of the obstacles in the workspace that may be avoided during the bin picking operation. Three shape types may be supported: sphere, capsule, and lozenge. However, numerous other shape types are also within the scope of the present disclosure. The user may load and save the collision shapes from a file on the bin picking system.” in ¶103, as per “robotic bin picking process 10 may determine 202 a path to the one or more candidate objects based upon, at least in part, a robotic environment and at least one robotic constraint. For example, robotic bin picking process 10 may define a path to the candidate objects or workpieces taking into consideration one or more aspects including, but not limited to, the workpiece shape, the environment, the bin, the end of arm tool, and/or robot link/joint limitations/constraints” in ¶162)
a shape storage unit configured to store a shape of the robot and the hand; (as per “Along with the bin picking application, the coprocessor may host the relevant files for bin picking including the STEP files for the EOAT, bin, and workpiece” in ¶86, as per “a suction cup end of arm tool (EOAT) may be connected to the controller via a e.g., 24 VDC Digital Output channel” in ¶83)
based on outputs of the take-out position calculation unit, the spatial information storage unit and the shape storage unit, (as per “robotic bin picking process 10 may determine 202 a path to the one or more candidate objects based upon, at least in part, a robotic environment and at least one robotic constraint. For example, robotic bin picking process 10 may define a path to the candidate objects or workpieces taking into consideration one or more aspects including, but not limited to, the workpiece shape, the environment, the bin, the end of arm tool, and/or robot link/joint limitations/constraints” in ¶162, as per “The sensor (e.g., a scanner) may also provide a data set describing the perceived environment including static and dynamic objects. In some embodiments, robotic bin picking process 10 may use the data set to learn the environment to determine the path and/or for collision avoidance” in ¶163, as per “etermining 202 the path to the one or more candidate objects may include using information about one or more surfaces of at least one object adjacent to the candidate object and avoiding a collision with the at least one object adjacent the candidate object. In this manner, robotic bin picking process 10 may use information about surfaces of objects around the candidate workpiece when determining a path the candidate object to avoid a collision with the objects around the candidate workpiece” in ¶167)
wherein the shape storage unit is configured to store a pre-gripping hand shape before the hand grips the object (as per “the bin picking system may be able to accept a CAD model of the end effector. The bin picking system may also work with a point cloud of an end effector. The bin picking system may have a selectable option to avoid collisions between the end effector and the bin or a non-gripped workpiece. When collision avoidance with adjacent workpieces is selected, the gripper, robot, and any gripped workpiece should not contact other workpieces during gripping. This implies that the path planning may search for some level of clearance around a target workpiece. The bin picking system may allow the definition of multiple pick points or grasps for a given workpiece. If multiple pick points or grasps for a different workpiece are definable, an indication of which grip was used may be available to the controlling program. If multiple pick points or grasps for a different workpiece are definable, there may be a hierarchy of gripping preferences” in ¶148) and a post-gripping hand shape after the hand grips the object. (as per “a graphical user interface that allows for the configuration of tool collision shapes is provided. The tool collision shapes may be configured in an editor that is like the one used for the environment collision shapes. The tool and the shapes may be rendered constantly, and the user can rotate and zoom to see the shapes as they are edited” in ¶108, as per “a graphical user interface that allows for configuring workpiece collision shapes is provided. The user may configure the collision shapes for the workpiece. These shapes are used to detect and avoid collisions between the workpiece and the environment after the workpiece has been picked” in ¶113)
Truebenbach fails to expressly disclose:
an operation route generation unit configured to generate an operation route of the hand so as not to interface the robot with the surroundings,
Komoda discloses of a grasping control device, grasping system, and computer program product, comprising:
an operation route generation unit configured to generate an operation route of the hand so as not to interface the robot with the surroundings, (as per “The route calculation unit 21 d calculates a movement route of the end effector 13 from a current position and a current posture to the grasping position and the grasping posture. In addition, the interference determination unit 21 e determines the presence or absence of interference between the manipulator 10 and the obstacle O with respect to movement of the end effector 13 on the calculated movement route” in ¶49, as per “the arithmetic processing unit 21 functions as the route calculation unit 21 d and the interference determination unit 21 e to calculate a route from the current position and the current posture to the grasping position and the grasping posture of the end effector 13 for each pattern (candidate for the usage form) included in the selected model (grasping form) and calculate the presence or absence of interference between the end effector 13 and the manipulator 10, and the obstacle O other than the object T with respect to movement of the end effector 13 and the manipulator 10 on the route (S13)” in ¶58)
In this way, Komoda operates to calculate a movement route of the end effector from a current position and posture to a grasping position and posture, and to determine the presence or absence of interference between the manipulator, end effector, and obstacle during movement along the calculated route (¶49, ¶58). Like Truebenbach, Komoda is concerned with robotic grasping and control of an end effector relative to an object while avoiding interference with surrounding obstacles.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the robotic bin-picking system of Truebenbach with the route calculation and interference determination arrangement of Komoda to enable another standard means of generating an operation route for the robot hand while avoiding interference with surroundings. Such modification also allows the system to calculate a grasping route for the end effector and determine whether the manipulator or end effector would interfere with surrounding obstacles along the route, thereby improving collision avoidance and pick reliability during robotic bin picking (¶49, ¶58).
As per Claim 2, the combination of Truebenbach and Komoda teaches or suggests all limitations of Claim 1. Truebenbach fails to expressly disclose wherein the image capturing device is configured to capture an image including the object and the hand.
See Claim 1 for teachings of Komoda. Komoda further discloses wherein the image capturing device is configured to capture an image including the object and the hand. (as per “The information acquisition unit 21 a acquires information (feature information, attribute information, and detection information) of the object T, an obstacle O, and the manipulator 10. Information on the object T, the obstacle O, and the manipulator 10 is obtained from, for example, a detection value by the sensor 17. The sensor 17 is, for example, an RGB-D sensor, a camera, a contact sensor, a distance sensor, etc.” in ¶36)
In this way, Komoda operates to calculate a movement route of the end effector from a current position and posture to a grasping position and posture, and to determine the presence or absence of interference between the manipulator, end effector, and obstacle during movement along the calculated route (¶49, ¶58). Like Truebenbach, Komoda is concerned with robotic grasping and control of an end effector relative to an object while avoiding interference with surrounding obstacles.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the robotic bin-picking system of Truebenbach with the route calculation and interference determination arrangement of Komoda to enable another standard means of generating an operation route for the robot hand while avoiding interference with surroundings. Such modification also allows the system to calculate a grasping route for the end effector and determine whether the manipulator or end effector would interfere with surrounding obstacles along the route, thereby improving collision avoidance and pick reliability during robotic bin picking (¶49, ¶58).
As per Claim 3, the combination of Truebenbach and Komoda teaches or suggests all limitations of Claim 1. Truebenbach further discloses wherein:
the image capturing device is configured to capture an image of a three-dimensional image (as per “the sensor may be a 3-D sensor. In some embodiments, the sensor may be a 2-D sensor. The re-scan may be in an area of the sensed volume where the sensor resolution is maximal. The sensor (e.g., a scanner) may also provide a data set describing the perceived environment including static and dynamic objects.” in ¶163)
the operation route generation unit is configured to generate the operation route of the hand based on an output of the take-out position calculation unit, information of the three-dimensional image captured by the image capturing device, and the pre-gripping hand shape and the post-gripping hand shape stored in the shape storage unit. (as per “robotic bin picking process 10 may determine 202 a path to the one or more candidate objects based upon, at least in part, a robotic environment and at least one robotic constraint. For example, robotic bin picking process 10 may define a path to the candidate objects or workpieces taking into consideration one or more aspects including, but not limited to, the workpiece shape, the environment, the bin, the end of arm tool, and/or robot link/joint limitations/constraints. In some embodiments, the path may be a feasible path, an optimal path, or both. For example, a feasible path may generally include a possible path to the workpiece while an optimal path may generally include a path optimized for one or more attributes (e.g., shortest time, fewest adjustments in the robotic arm, etc.)” in ¶162, as per “a graphical user interface that allows for configuring workpiece collision shapes is provided. The user may configure the collision shapes for the workpiece. These shapes are used to detect and avoid collisions between the workpiece and the environment after the workpiece has been picked.” in ¶113)
Truebenbach fails to expressly disclose:
an image including the object and the hand,
See Claim 1 for teachings of Komoda. Komoda further discloses an image including the object and the hand. (as per “The information acquisition unit 21 a acquires information (feature information, attribute information, and detection information) of the object T, an obstacle O, and the manipulator 10. Information on the object T, the obstacle O, and the manipulator 10 is obtained from, for example, a detection value by the sensor 17. The sensor 17 is, for example, an RGB-D sensor, a camera, a contact sensor, a distance sensor, etc.” in ¶36)
In this way, Komoda operates to calculate a movement route of the end effector from a current position and posture to a grasping position and posture, and to determine the presence or absence of interference between the manipulator, end effector, and obstacle during movement along the calculated route (¶49, ¶58). Like Truebenbach, Komoda is concerned with robotic grasping and control of an end effector relative to an object while avoiding interference with surrounding obstacles.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the robotic bin-picking system of Truebenbach with the route calculation and interference determination arrangement of Komoda to enable another standard means of generating an operation route for the robot hand while avoiding interference with surroundings. Such modification also allows the system to calculate a grasping route for the end effector and determine whether the manipulator or end effector would interfere with surrounding obstacles along the route, thereby improving collision avoidance and pick reliability during robotic bin picking (¶49, ¶58).
As per Claim 4, the combination of Truebenbach and Komoda teaches or suggests all limitations of Claim 1. Truebenbach further discloses wherein:
the operation route generation unit is configured to generate a pre-gripping route from a predetermined first position to the take-out position of the object based on the pre-gripping hand shape, (as per “graphical user interface that allows for training grasps and placements is provided. The user may train the grasps and placements by clicking on a grasp node in the program tree on the left and following through the Grasp page tabs from left to right. Each grasp page may allow the user to 1) define the grasp position relative to the workpiece, 2) define the grasp offset to be used when approaching the workpiece, 3) define the placement position relative to the robot base, and 4) define the placement offset to use when approaching the placement position. The user can give each grasp a unique name by clicking in the “Name” field. The user may set the grasp pick position by following the steps shown in the dialog on the “Pick Position” tab. The pick position may refer to the point on the surface of the workpiece where the EOAT will attach. The user may click the first button to move the robot to the teaching position (rescan position). Next the user may put the workpiece in the gripper and click the second button to trigger a scan. The workpiece pose relative to the EOAT may be recorded and saved as the grasp position. The user may then switch to the pick offset tab and set the offset value.” in ¶118)
generate a post-gripping route from the take-out position of the object to a predetermined second position based on the post-gripping hand shape. (as per “graphical user interface that allows a user to train the pick is provided. The user may select the “teach pick approach’ option and move the robot to the pick approach position. The approach position should not be in the part template collision zone. The user may select the “ok” option to record the position and then continue to set other positions” in ¶137, as per “controlling 206 the robot may include performing a second scan of the first candidate object, moving the first candidate object to a placement target having a fixed location with an accuracy requirement, manipulating the first candidate object and delivering the first candidate object to the placement target in accordance with the accuracy requirement. For example, the robot may pick up a candidate workpiece and move it to a placement location that may be a machine. The machine may have a fixed location with a higher accuracy requirement. Accordingly and to improve placement accuracy, robotic bin picking process 10 may scan the picked up workpiece (e.g., re-scan), manipulate the workpiece, and locate it to the machine. The re-scan operation may use the same sensor/scanner used to locate the workpiece, or an additional sensor/scanner. In some embodiments, the second scan of the candidate object may be in an area of maximum resolution of the scanner.” in ¶168)
As per Claim 5, the combination of Truebenbach and Komoda teaches or suggests all limitations of Claim 1. Truebenbach further discloses wherein the shape storage unit is configured to store a pre-gripping hand shape and a post-gripping hand shape corresponding to one type of objects with a same shape. (as per “the bin picking program configuration phase is where the user configures the bin picking system to perform a bin picking operation with a given workpiece and placement or fixture. The user may first load or create a new program configuration. Creating a new program may include, but is not limited to, configuring the tool, workpiece template, and bin followed by training grasps and placements” in ¶90, as per “a graphical user interface that allows for configuring the workpiece and loading a workpiece model is provided. The user may configure the workpiece to be picked by clicking on the “Part Template” node in the program tree. The user may load the workpiece CAD model from a file on the bin picking system. The CAD model may be converted to a mesh file for rendering and point cloud for pose detection.” in ¶112, as per “a graphical user interface that allows for configuring workpiece collision shapes is provided. The user may configure the collision shapes for the workpiece. These shapes are used to detect and avoid collisions between the workpiece and the environment after the workpiece has been picked” in ¶113)
As per Claim 6, the combination of Truebenbach and Komoda teaches or suggests all limitations of Claim 1. Truebenbach further discloses wherein the shape storage unit is configured to store shapes of a plurality types of objects with different shapes, and a plurality of pre-gripping hand shapes (as per “The user may configure the workpiece to be picked by clicking on the “Part Template” node in the program tree. The user may load the workpiece CAD model from a file on the bin picking system. The CAD model may be converted to a mesh file for rendering and point cloud for pose detection.” in ¶112, as per “The bin picking system may allow for multiple types of pick-able workpieces in the same bin. If this is the case, the bin picking system may be able to specify programmatically which type of workpiece is desired before starting the pick.” in ¶147, as per “It may be possible to recall a previously trained workpiece type and create a new bin picking program within one hour.” in ¶152) and a plurality of post-gripping hand shapes corresponding to the plurality types of objects. (as per “Along with the bin picking application, the coprocessor may host the relevant files for bin picking including the STEP files for the EOAT, bin, and workpiece.” in ¶86, as per “Each grasp page may allow the user to 1) define the grasp position relative to the workpiece, 2) define the grasp offset to be used when approaching the workpiece, 3) define the placement position relative to the robot base, and 4) define the placement offset to use when approaching the placement position.” in ¶118, as per “The user may configure the collision shapes for the workpiece. These shapes are used to detect and avoid collisions between the workpiece and the environment after the workpiece has been picked.” in ¶113, as per “The bin picking system may allow the definition of multiple pick points or grasps for a given workpiece. If multiple pick points or grasps for a different workpiece are definable, an indication of which grip was used may be available to the controlling program.” in ¶148)
As per Claim 10, the combination of Truebenbach and Komoda teaches or suggests all limitations of Claim 1. Truebenbach further discloses wherein
the object is a plurality of objects randomly placed inside a housing vessel, (as per “present disclosure are directed towards a system and method for robotic bin picking. Accordingly, the bin picking methodologies included herein may allow a robot to work with a scanning system to identify parts in a bin, pick parts from the bin, and place the picked parts at a designated location” in ¶64)
the hand sequentially takes out individual objects in the housing vessel, and the housing vessel is replaced with another housing vessel in order. (as per “the user may trigger the bin picking system to perform bin picking or stop, and monitors the progress. The bin picking system may run automatically and scan the bin prior to each pick attempt. In some embodiments, there are two anticipated user roles for the bin picking system these may include the user role and developer role.” in ¶91, as per “Setup may be agnostic to the bin size and shape. Preferably, the bin may even change in between picks, e.g. from a plastic tote to a cardboard box, without affecting system operation. The bin picking system may work with cardboard boxes that have open flaps. The bin picking system may work when there is no bin, e.g. if parts are in a pile. The bin picking system may work as a 2D bin picker as well, e.g. with parts uniformly posed on a flat surface. The bin picking system may work with workpieces as small as 1×1×0.1 cm, and as large as 30×30×30 cm. Resolution and accuracy may vary with workpiece size” in ¶146)
As per Claim 11, the combination of Truebenbach and Komoda teaches or suggests all limitations of Claim 10. Truebenbach further discloses wherein the operation route generation unit is configured to grip changes in the shape and arrangement position of the housing vessel based on the image information from the image capturing device. (as per “The bin picking system may work with imprecisely placed bins, and bins that move between cycles” in ¶151, as per “The sensor (e.g., a scanner) may also provide a data set describing the perceived environment including static and dynamic objects. In some embodiments, robotic bin picking process 10 may use the data set to learn the environment to determine the path and/or for collision avoidance” in ¶163, as per “Preferably, the bin may even change in between picks, e.g. from a plastic tote to a cardboard box, without affecting system operation” in ¶146)
Truebenbach fails to expressly disclose generating an operation route of the hand.
See Claim 10 for teachings of Komoda. Komoda further discloses generating an operation route of the hand. (as per “The route calculation unit 21 d calculates a movement route of the end effector 13 from a current position and a current posture to the grasping position and the grasping posture. In addition, the interference determination unit 21 e determines the presence or absence of interference between the manipulator 10 and the obstacle O with respect to movement of the end effector 13 on the calculated movement route.” in ¶49, as per “the arithmetic processing unit 21 functions as the route calculation unit 21 d and the interference determination unit 21 e to calculate a route from the current position and the current posture to the grasping position and the grasping posture of the end effector 13 for each pattern (candidate for the usage form) included in the selected model (grasping form) and calculate the presence or absence of interference between the end effector 13 and the manipulator 10, and the obstacle O other than the object T with respect to movement of the end effector 13 and the manipulator 10 on the route (S13).” in ¶58)
In this way, Komoda operates to calculate a movement route of the end effector from a current position and posture to a grasping position and posture, and to determine the presence or absence of interference between the manipulator, end effector, and obstacle during movement along the calculated route (¶49, ¶58). Like Truebenbach, Komoda is concerned with robotic grasping and control of an end effector relative to an object while avoiding interference with surrounding obstacles.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the robotic bin-picking system of Truebenbach with the route calculation and interference determination arrangement of Komoda to enable another standard means of generating an operation route for the robot hand while avoiding interference with surroundings. Such modification also allows the system to calculate a grasping route for the end effector and determine whether the manipulator or end effector would interfere with surrounding obstacles along the route, thereby improving collision avoidance and pick reliability during robotic bin picking (¶49, ¶58).
As per Claim 12, the combination of Truebenbach and Komoda teaches or suggests all limitations of Claim 1. Truebenbach further discloses
a robot system including a robot having a hand configured to take out a bulk-loaded object, (as per “the bin picking system (e.g., bin picking system 64) may include a robot arm (e.g., Universal Robots UR5 available from Universal Robots, etc.), a controller, a gripper, a sensor, and a coprocessor (e.g., to run the computationally expensive operations from perception and task planning)” in ¶80)
an image capturing device configured to capture an image including the object, and (as per “the sensor may any suitable sensor (e.g., a 3D sensor)” in ¶81)
a robot controller configured to control the robot so as to take out the object by the hand, (as per “present disclosure are directed towards a system and method for robotic bin picking. Accordingly, the bin picking methodologies included herein may allow a robot to work with a scanning system to identify parts in a bin, pick parts from the bin, and place the picked parts at a designated location” in ¶64, as per Fig. 28))
As per Claim 13, Truebenbach discloses a method for robotic bin picking, comprising:
computer readable non-transitory tangible medium for storing a robot control program for a robot system (as per “The instruction sets and subroutines of robotic bin picking process 10, which may be stored on storage device 16 coupled to computing device 12, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within computing device 12.” in ¶68, as per “As explained above, the invention provides both a method and corresponding equipment consisting of various modules providing the functionality for performing the steps of the method. The modules may be implemented as hardware, or may be implemented as software or firmware for execution by a computer processor. In particular, in the case of firmware or software, the invention can be provided as a computer program product including a computer readable storage structure embodying computer program code (i.e., the software or firmware) thereon for execution by the computer processor.” in ¶182)
a robot having a hand configured to take out a bulk-loaded object, (as per “Embodiments of the present disclosure are directed towards a system and method for robotic bin picking. Accordingly, the bin picking methodologies included herein may allow a robot to work with a scanning system to identify parts in a bin, pick parts from the bin, and place the picked parts at a designated location.” in ¶64, as per “the bin picking system (e.g., bin picking system 64) may include a robot arm (e.g., Universal Robots UR5 available from Universal Robots, etc.), a controller, a gripper, a sensor, and a coprocessor (e.g., to run the computationally expensive operations from perception and task planning).” in ¶80)
an image capturing device configured to capture an image including the object, (as per “the sensor may any suitable sensor (e.g., a 3D sensor).” in ¶81, as per “the Sensor interface may include the following methods and may be implemented through the Sensor class to interface with the Scanner. Method Parameters Description scan ScanData [out] Triggers a scan in the underlying hardware implementation. This method returns a Status object and populates an instance of ScanData - point cloud and time stamp.” in ¶94-¶97, as per “bin picking system (e.g., bin picking system 64) may include a scanning system (e.g., one or more sensors and/or scanners) configured to identify parts in a bin.” in ¶161)
a robot controller configured to control the robot so as to take out the object by the hand, (as per “if the validation passes, robotic bin picking process 10 may control the robot to pick up the candidate workpiece.” in ¶165, as per “the robot may pick up a candidate workpiece and move it to a placement location that may be a machine.” in ¶168)
the robot control program causing an arithmetic processing unit to execute a process of calculating a take-out position of an object to be captured by the robot based on the image information from the image capturing device; (as per “the Matcher interface includes the following methods and is implemented through the Matcher class to utilize the SDK pose estimation utility. Method Parameters Description findTarget ScanData [in] TargetData [out] Completes workpiece detection and pose estimation. Returns the description of the found target workpiece in TargetData.” in ¶95-¶100, as per “The user may load the workpiece CAD model from a file on the bin picking system. The CAD model may be converted to a mesh file for rendering and point cloud for pose detection.” in ¶112, as per “robotic bin picking process 10 may identify 200 a list of candidate workpieces or objects to be picked up.” in ¶161, as per “The pick position may refer to the point on the surface of the workpiece where the EOAT will attach.” in ¶118)
based on the image information from the image capturing device, an output of a spatial information storage unit configured to store an interference range in which the robot interferes with surroundings in a movable range, and an output of a shape storage unit configured to store a shape of the robot and the hand, (as per “the user may select the Environment tab to configure the workspace obstacles. In this tab the user can load, create, edit, and/or save the set of shapes that define all of the obstacles in the workspace that may be avoided during the bin picking operation.” in ¶103, as per “Along with the bin picking application, the coprocessor may host the relevant files for bin picking including the STEP files for the EOAT, bin, and workpiece.” in ¶86, as per “The tool collision shapes may be configured in an editor that is like the one used for the environment collision shapes.” in ¶108, as per “robotic bin picking process 10 may determine 202 a path to the one or more candidate objects based upon, at least in part, a robotic environment and at least one robotic constraint.” in ¶162)
wherein the shape storage unit is configured to store a pre-gripping hand shape before the hand grips the object, and a post-gripping hand shape after the hand grips the object. (as per “the bin picking system may be able to accept a CAD model of the end effector. The bin picking system may also work with a point cloud of an end effector.” in ¶148, as per “The user may configure the collision shapes for the workpiece. These shapes are used to detect and avoid collisions between the workpiece and the environment after the workpiece has been picked.” in ¶113, as per “When collision avoidance with adjacent workpieces is selected, the gripper, robot, and any gripped workpiece should not contact other workpieces during gripping.” in ¶148)
Truebenbach fails to expressly disclose:
a process of generating an operation route of the hand so as not to interface the robot with surroundings,
Komoda discloses of a grasping control device, grasping system, and computer program product, comprising:
a process of generating an operation route of the hand so as not to interface the robot with surroundings, (as per “The route calculation unit 21 d calculates a movement route of the end effector 13 from a current position and a current posture to the grasping position and the grasping posture. In addition, the interference determination unit 21 e determines the presence or absence of interference between the manipulator 10 and the obstacle O with respect to movement of the end effector 13 on the calculated movement route.” in ¶49, as per “the arithmetic processing unit 21 functions as the route calculation unit 21 d and the interference determination unit 21 e to calculate a route from the current position and the current posture to the grasping position and the grasping posture of the end effector 13 for each pattern (candidate for the usage form) included in the selected model (grasping form) and calculate the presence or absence of interference between the end effector 13 and the manipulator 10, and the obstacle O other than the object T with respect to movement of the end effector 13 and the manipulator 10 on the route (S13).” in ¶58)
In this way, Komoda operates to calculate a movement route of the end effector from a current position and posture to a grasping position and posture, and to determine the presence or absence of interference between the manipulator, end effector, and obstacle during movement along the calculated route (¶49, ¶58). Like Truebenbach, Komoda is concerned with robotic grasping and control of an end effector relative to an object while avoiding interference with surrounding obstacles.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the robotic bin-picking system of Truebenbach with the route calculation and interference determination arrangement of Komoda to enable another standard means of generating an operation route for the robot hand while avoiding interference with surroundings. Such modification also allows the system to calculate a grasping route for the end effector and determine whether the manipulator or end effector would interfere with surrounding obstacles along the route, thereby improving collision avoidance and pick reliability during robotic bin picking (¶49, ¶58).
As per Claim 14, the combination of Truebenbach and Komoda teaches or suggests all limitations of Claim 14. Truebenbach further discloses wherein:
the process of generating the operation route of the hand includes a pre-gripping route generation process configured to generate a pre-gripping route from a predetermined first position to a take-out position of the object based on the pre-gripping hand shape; (as per “The user may set the grasp pick position by following the steps shown in the dialog on the “Pick Position” tab. The pick position may refer to the point on the surface of the workpiece where the EOAT will attach.” in ¶118, as per “The user may train the workpiece pick position and offset by following the “Pick Position” and “Pick Offset” tabs.” in ¶119, as per “The user may select the “teach pick approach’ option and move the robot to the pick approach position. The approach position should not be in the part template collision zone. The user may select the “ok” option to record the position and then continue to set other positions.” in ¶137)
a post-gripping route generation process configured to generate a post-gripping route from the take-out position of the object to a predetermined second position based on the post-gripping hand shape is included. (as per “Each grasp page may allow the user to 1) define the grasp position relative to the workpiece, 2) define the grasp offset to be used when approaching the workpiece, 3) define the placement position relative to the robot base, and 4) define the placement offset to use when approaching the placement position.” in ¶118, as per “The user may train the workpiece place position and offset by following the “Place Position” and “Place Offset” tabs.” in ¶120, as per “controlling 206 the robot may include performing a second scan of the first candidate object, moving the first candidate object to a placement target having a fixed location with an accuracy requirement, manipulating the first candidate object and delivering the first candidate object to the placement target in accordance with the accuracy requirement.” in ¶168)
To the extent Truebenbach fails to expressly disclose generating the pre-gripping and post-gripping routes as part of an operation route of the hand so as not to interfere with surroundings, see Claim 13 for teachings of Komoda. Komoda discloses calculating a movement route of the end effector to the grasping position and determining whether the manipulator and end effector interfere with an obstacle during movement on the calculated route. (as per “The route calculation unit 21 d calculates a movement route of the end effector 13 from a current position and a current posture to the grasping position and the grasping posture. In addition, the interference determination unit 21 e determines the presence or absence of interference between the manipulator 10 and the obstacle O with respect to movement of the end effector 13 on the calculated movement route.” in ¶49)
In this way, Komoda operates to calculate a movement route of the end effector from a current position and posture to a grasping position and posture, and to determine the presence or absence of interference between the manipulator, end effector, and obstacle during movement along the calculated route (¶49, ¶58). Like Truebenbach, Komoda is concerned with robotic grasping and control of an end effector relative to an object while avoiding interference with surrounding obstacles.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the robotic bin-picking system of Truebenbach with the route calculation and interference determination arrangement of Komoda to enable another standard means of generating an operation route for the robot hand while avoiding interference with surroundings. Such modification also allows the system to calculate a grasping route for the end effector and determine whether the manipulator or end effector would interfere with surrounding obstacles along the route, thereby improving collision avoidance and pick reliability during robotic bin picking (¶49, ¶58).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Truebenbach (US Pub. No. 20190389062) in view of Komoda (US Pub. No. 20190283249) in further view of Melikian (US Pub. No. 20100021051).
As per Claim 7, the combination of Truebenbach and Komoda teaches or suggests all limitations of Claim 6. Truebenbach further discloses wherein:
configured to calculate take-out positions of the specified type of the object, (as per “The CAD model may be converted to a mesh file for rendering and point cloud for pose detection.” in ¶112, as per “The pick position may refer to the point on the surface of the workpiece where the EOAT will attach.” in ¶118, as per “robotic bin picking process 10 may identify 200 a list of candidate workpieces or objects to be picked up.” in ¶161, as per “robotic bin picking process 10 may determine 202 a path to the one or more candidate objects based upon, at least in part, a robotic environment and at least one robotic constraint.” in ¶162)
the shape storage unit is configured to output a pre-gripping hand shape and a post-gripping hand shape corresponding to the specified type of the object, (as per “Along with the bin picking application, the coprocessor may host the relevant files for bin picking including the STEP files for the EOAT, bin, and workpiece.” in ¶86, as per “Each grasp page may allow the user to 1) define the grasp position relative to the workpiece, 2) define the grasp offset to be used when approaching the workpiece, 3) define the placement position relative to the robot base, and 4) define the placement offset to use when approaching the placement position.” in ¶118, as per “The selected grasp may be rendered in the “Current View” window and its ID will be displayed left of this window.” in ¶122, as per “These shapes are used to detect and avoid collisions between the workpiece and the environment after the workpiece has been picked.” in ¶113)
the operation route generation unit is configured to generate an operation route of the hand based on the output pre-gripping hand shape and the output post-gripping hand shape.
Truebenbach fail to expressly disclose:
the take-out position calculation unit is configured to specify a type of the object from among the plurality of objects stored in the shape storage unit based on the image captured by the image capturing device,
the operation route generation unit is configured to generate an operation route of the hand based on the output pre-gripping hand shape and the output post-gripping hand shape.
See Claim 6 for teachings of Komoda. Komoda further discloses:
the operation route generation unit is configured to generate an operation route of the hand based on the output pre-gripping hand shape and the output post-gripping hand shape. (as per “The route calculation unit 21 d calculates a movement route of the end effector 13 from a current position and a current posture to the grasping position and the grasping posture. In addition, the interference determination unit 21 e determines the presence or absence of interference between the manipulator 10 and the obstacle O with respect to movement of the end effector 13 on the calculated movement route.” in ¶49, as per “the arithmetic processing unit 21 functions as the route calculation unit 21 d and the interference determination unit 21 e to calculate a route from the current position and the current posture to the grasping position and the grasping posture of the end effector 13 for each pattern…” in ¶58)
In this way, Komoda operates to calculate a movement route of the end effector from a current position and posture to a grasping position and posture, and to determine the presence or absence of interference between the manipulator, end effector, and obstacle during movement along the calculated route (¶49, ¶58). Like Truebenbach, Komoda is concerned with robotic grasping and control of an end effector relative to an object while avoiding interference with surrounding obstacles.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the robotic bin-picking system of Truebenbach with the route calculation and interference determination arrangement of Komoda to enable another standard means of generating an operation route for the robot hand while avoiding interference with surroundings. Such modification also allows the system to calculate a grasping route for the end effector and determine whether the manipulator or end effector would interfere with surrounding obstacles along the route, thereby improving collision avoidance and pick reliability during robotic bin picking (¶49, ¶58).
Truebenbach and Komoda fail to expressly disclose:
the take-out position calculation unit is configured to specify a type of the object from among the plurality of objects stored in the shape storage unit based on the image captured by the image capturing device,
Melikian discloses of an automated guidance and recognition system , comprising:
the take-out position calculation unit is configured to specify a type of the object from among the plurality of objects stored in the shape storage unit based on the image captured by the image capturing device, (as per “The visual guidance and recognition system 10 can undergo set up for one or a plurality of different work pieces. The amount of different reference pieces is limited only by system memory. Furthermore, the type of reference work pieces can vary in unlimited ways, including length, width, depth, shape, color, material, etc.” in ¶19, as per “A database is stored in the recognition controller 50 of any reference work pieces 70 that are learned during the set up process. During operation, a production work piece 170 is introduced to the manipulator 20 in an at rest position in the manner described above. A two dimensional image of the production work piece 170 is acquired by the recognition controller 50 via the camera 30. The recognition controller 50 compares the two dimensional image of the production work piece 170 with the database of reference work pieces.” in ¶32, as per “The recognition controller compares the two dimensional image of the production work piece with the database of reference work piece images. A match indicates that the recognition controller has recognized the type of work piece (S16) and can properly determine the required relevant positioning of the manipulator and production work piece so that the manipulator can accurately perform its function” in ¶42, as per ¶32 & ¶24)
In this way, Melikian operates to acquire an image of a production workpiece via a camera, compare the acquired image with a database of learned reference workpieces, and recognize the type of workpiece based on the comparison (¶32, ¶42). Like Truebenbach and Komoda, Melikian is concerned with image-guided robotic manipulation and positioning of a manipulator relative to a workpiece.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system of Truebenbach and Komoda with the image-based workpiece recognition arrangement of Melikian to enable another standard means of specifying the type of object from among stored workpiece/reference object information based on an image captured by a camera. Such modification also allows the system to automatically identify the type of workpiece being picked from captured image information before calculating the corresponding pick or take-out position, thereby improving flexibility when different object types are presented (¶32, ¶42).
Claim(s) 8 is rejected under 35 U.S.C. 103 as being unpatentable over Truebenbach (US Pub. No. 20190389062) in view of Komoda (US Pub. No. 20190283249) in further view of Holhjem (US Pub. No. 20240051136).
As per Claim 8, the combination of Truebenbach and Komoda teaches or suggests all limitations of Claim 1. Truebenbach and Komoda fail to expressly disclose:
the hand is attached to a movable portion operable with respect to the robot,
the operation route generation unit is configured to correct the pre-gripping hand shape and the post-gripping hand shape stored in the shape storage unit based on an operation of the hand with respect to the robot by the movable portion, and is configured to generate an operation route of the hand.
Holhjem discloses of motion planning, comprising:
the hand is attached to a movable portion operable with respect to the robot, (as per “the second model (or a further model as outlined in respect of certain embodiments of the present invention) may comprise a representation of an end effector. Those skilled in the art will appreciate that an end effector is a device, typically located at the end of a robot arm, that interacts with the robot's environment or external objects” in ¶58, as per “a robot arm 2 operating in an environment 4. The robot arm 2 consists of a base 6, a pair of links 8 a, 8 b, and an end effector 10 connected to each other at joints 12. The robot arm 2 is tasked with moving items from one bin 14 to a second bin 16, e.g. for sorting parts on an assembly line.” in ¶125)
the operation route generation unit is configured to correct the pre-gripping hand shape and the post-gripping hand shape stored in the shape storage unit based on an operation of the hand with respect to the robot by the movable portion, (as per “a gripper may be modelled in an open (i.e. non-gripping) position as one model and in a closed (i.e. gripping) position as another model. Due to the advantageous modular approach provided by embodiments the present invention, the appropriate model can be generated or received for the current or planned state of the end effector when carrying out motion planning. Of course, different mappings may be used for different sections of the path, such as an open gripper when moving toward an object and a closed gripper when picking up the object and moving it to a destination” in ¶60, as per “FIG. 6 is a schematic diagram illustrating the workspace elements 36 that are hit by the fingers 34 when operated in the open position, and FIG. 7 illustrates the uniquely mapped workspace elements 38 that are hit by the fingers 34 in the open position, i.e. those elements 38 that are hit only as a result of the fingers 34 being present, rather than those ‘overlapping’ elements 40 already hit due to the presence of the palm 30. Thus, only information relating to the set of these additional workspace elements 38 need be stored in the mapping, because the elements 38 already stored within the mapping for the palm 30 are otherwise redundant” in ¶144, as per “mappings for the fingers 34 in the open and closed positions can then be stored in a library in memory as ‘Part Maps’ for that part (i.e. for the gripper) alongside the Link Map(s). These can then be selected from a library as appropriate depending on the current or planned state of the gripper for a given task. Advantageously, this avoids storing two complete mappings for the entire robot when the only difference in the workspace elements hit is due to the position of the fingers 34, where all other workspace element hits (i.e. due to the palm and the other links of the robot 2) are duplicated for both positions.” in ¶146)
is configured to generate an operation route of the hand. (as per “the mappings may then be used to generate a path for the robot to follow. In other words, the combination of the appropriate mappings can be used in the same way that a single mapping for an entire robot normally is, in order to generate an appropriate plan for the robot to execute a given task” in ¶102)
In this way, Holhjem operates to model a gripper in different operating states, including an open non-gripping position and a closed gripping position, and to select the appropriate model depending on the current or planned state of the end effector when carrying out motion planning (¶60). Like Truebenbach and Komoda, Holhjem is concerned with robotic motion planning using end-effector models to generate paths while accounting for collision or workspace interference.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system of Truebenbach and Komoda with the open-state and closed-state gripper modeling arrangement of Holhjem to enable another standard means of correcting or selecting the pre-gripping hand shape and post-gripping hand shape based on operation of the hand with respect to the robot. Such modification also allows the system to use the appropriate gripper shape for different sections of the path, including an open gripper when moving toward an object and a closed gripper when moving the picked object to a destination, thereby improving collision checking and path generation (¶60, ¶102).
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Truebenbach (US Pub. No. 20190389062) in view of Komoda (US Pub. No. 20190283249) in further view of Konolige (US Pub. No. 20160016311)
As per Claim 9, the combination of Truebenbach and Komoda teaches or suggests all limitations of Claim 1. Truebenbach and Komoda fail to expressly disclose:
an object shape measurement unit configured to measure an object shape of the object based on the image information from the image capturing device; and
a post-gripping hand shape generation unit configured to generate the post-gripping hand shape based on the object shape measured by the object shape measurement unit, wherein
the operation route generation unit is configured to generate an operation route of the hand so that the robot does not interfere with surroundings based on an output of the post-gripping hand shape generation unit.
Konolige discloses of real-time determination of object metrics for trajectory planning, comprising:
an object shape measurement unit configured to measure an object shape of the object based on the image information from the image capturing device; (as per “The sensors may scan an environment containing one or more objects in order to capture visual data and/or three-dimensional (3D) depth information. Data from the scans may then be integrated into a representation of larger areas in order to provide digital environment reconstruction. In additional examples, the reconstructed environment may then be used for identifying objects to pick up, determining pick positions for objects, and/or planning collision-free trajectories for the one or more robotic arms and/or a mobile base” in ¶34, as per “the robotic device may utilize a combination of perception together with planning to guide the robot arm to pick up a box and place it where it needs to go” in ¶51, as per “As the robotic arm 102 moves, a sensor 106 on the arm may capture sensor data about the stack of boxes 220 in order to determine shapes and/or positions of individual boxes” in ¶52, as per “one or more depth or visual sensors may be mounted or otherwise positioned on a robot arm. As the robot arm moves to grip and pick up an object, the on-arm sensor(s) may collect sensor data in order to estimate the dimensions, size, and/or shape of the object” in ¶96, as per “sensors 420 and 422 may be depth or visual sensors mounted or positioned at points within the environment. In some examples, locations for sensors 420 and 422 may be chosen to receive sensor data indicating one or more dimensions of objects such as box 408 that may be undetectable by on-arm sensor 418” in ¶97)
a post-gripping hand shape generation unit configured to generate the post-gripping hand shape based on the object shape measured by the object shape measurement unit, (as per “During box picking, some information about the box being picked may be unknown until after the box has been picked up and/or moved a certain amount. For example, box weight or box depth may only be measurable by one or more sensors after the box is in the air. In some examples, these measurements may influence what trajectory should be used by the robot to transport the box. For instance, the box metrics may influence the speed at which the robot can transport the box without dropping it or whether a particular path can be used without causing a collision” in ¶25, as per “data from on-arm sensors as well as off-arm sensors placed strategically within the environment can be combined and used to analyze an object's
dimensions as it is being picked” in ¶26, as per “a triangulation depth sensor, including a laser beam line and offset camera, may be used. This type of sensor may have a low profile so that it can determine object depth after the gripper has made contact with an object and is therefore in close proximity to the object” in ¶27, as per “object properties that may influence the trajectory include object dimensions, size, shape, mass, center of gravity, and/or inertia matrix” in ¶82, as per “sensor data may indicate that box 408 has a depth corresponding to box shape 410... Accordingly, trajectory 412 may selected for robot arm 402 to use to move box 408” in ¶111)
wherein the operation route generation unit is configured to generate an operation route of the hand so that the robot does not interfere with surroundings based on an output of the post-gripping hand shape generation unit. (as per “planning a collision-free trajectory may involve determining the 3D location of objects and surfaces in the environment. A trajectory optimizer may make use of the 3D information provided by environment reconstruction to optimize paths in the presence of obstacles” in ¶62, as per “an environment may be captured as a mesh or set of 3D points. A robot arm may be represented as a convex hull of plane segments for quick collision checking. Constant or frequent updating of the environment may allow the robot arm to quickly respond to changes. In further examples, an optimizer may perform frequent continuous collision checking throughout its path” in ¶63, as per “another objective may be to determine a trajectory to move an object without causing a collision involving the object” in ¶83, as per “different trajectories may be determined in order to avoid collisions based on the predicted box sizes. For instance, it may be necessary to move a box shape 414 up higher in the air than a box shape 410 in order to avoid knocking over or otherwise colliding with other boxes within facade 406” in ¶89, as per “after receiving sensor data to determine or estimate one or more measurements of one or more properties of an object, a trajectory may be selected for an object with the determined or estimated measurements” in ¶108, as per “at a later point in time when the depth of the object has been determined, a trajectory corresponding to both the determined weight and determined depth may be selected and used from that point forward” in ¶109, as per “the robot may switch to using the new trajectory as soon as it has been determined using the received sensor data” in ¶110)
In this way, Konolige operates to use visual or depth sensor data to determine dimensions, size, or shape of an object as the robot moves to grip or pick up the object, and to select or update a trajectory based on the measured object properties (¶96, ¶108-¶111). Like Truebenbach and Komoda, Konolige is concerned with robotic perception and trajectory planning for picking and moving objects while avoiding collisions in the surrounding environment.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system of Truebenbach and Komoda with the object-shape measurement and trajectory-selection arrangement of Konolige to enable another standard means of generating or updating the post-gripping hand/object shape based on image-derived object shape information. Such modification also allows the system to determine object shape from sensor data during picking and generate an operation route that accounts for the picked object’s measured shape, thereby improving collision avoidance and trajectory selection when moving the object through the surrounding environment (¶96, ¶108-¶111).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Watanabe (US Pub. No. 20170173798) discloses an information processing apparatus, information processing method, and storage medium.
Zhu (US Pub. No. 20210308866) discloses adaptive grasp planning for bin picking.
Nagarajan (US Pub. No. 20210347040) discloses grasping of an object by a robot based on grasp strategy determined using machine learning model(s).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TYLER R ROBARGE whose telephone number is (703)756-5872. The examiner can normally be reached Monday - Friday, 8:00 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached on (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.R.R./Examiner, Art Unit 3658
/Ramon A. Mercado/Supervisory Patent Examiner, Art Unit 3658
Prosecution Insights