DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed on 09/17/2025 has been entered. Claims 1, 3-8, 10-13, 15-20 remain pending in the application.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 4, 13, 15, 16, 20 are rejected under 35 U.S.C. 103 as being unpatentable by Fan (US20220072707) in view of Shafer (US11331799) and Davidson (US20210053217) and Jeong (US20210016454) and Moreira (US20220281114) and Chitta (US20200047331).
Regarding claim 1, Fan teaches a method performed by one or more computers to cause a robot having an end effector to grasp an object in a container having one or more objects, the method comprising ([0013] disclosing the robot for picking parts from a bin. [0019] disclosing the end effector is a gripper):
determining a virtual workspace representing all or a portion of an interior of the container ([0023] disclosing simulating object piles in the bin);
determining a set of grasp proposals with associated grasping windows, wherein each grasp proposal of the set has a different respective position within the workspace ([0024] disclosing determining candidate grasps “proposal” wherein each grasp has a grasp pose “different position” and point map “grasping window”);
Fan does not teach determining, for each grasp proposal in the set of grasp proposals, a respective set of waypoints comprising a pre-grasp pose and a grasp pose within the workspace based on position values of the grasp proposal; generating, for each grasp proposal, a respective grasp proposal score based on scene features appearing within the respective window; selecting a grasp proposal from the set based on the grasp proposal scores; calculating a grasp trajectory for the selected grasp proposal based on the set of waypoints for the selected grasp proposal; controlling the end effector to grasp an object in the workspace based on the calculated grasp trajectory associated to the selected grasp proposal. Wherein a higher score is assigned to grasp proposal having fewer edges within the respective window. However, Fan teaches determining a success rate and ranking of successful grasps [0050].
Shafer teaches determining, for each grasp proposal in the set of grasp proposals, a respective set of waypoints comprising a pre-grasp pose and a grasp pose within the workspace based on position values of the grasp proposal (col 8 lines55 – col 9 line 13 disclosing determining candidate grasp poses and a pre-grasp pose based on the instances of visual features, the visual features are a contact point determined as a pixel which is interpreted as position values of the grasp proposal. See col 20 lines 9-65 disclosing the visual feature or instance of visual feature is based on grasp locations and pixels. The proposals generated all include a pre-grasp pose and multiple grasp pose candidates).
selecting a grasp proposal from the set based on the grasp proposal scores (col 6 lines 11-37 discloses the best predicted measure of success grasp pose is selected as the final grasp pose. Since the grasp instances has a grasp pose and a pre-grasp pose as cited in col 8 lines 55-67, the final pose is interpreted as being a final grasp proposal since it is associated with a pre-grasp pose);
calculating a grasp trajectory for the selected grasp proposal based on the set of waypoints for the selected grasp proposal (col 8 lines55 – col 9 line 13 discloses calculating a trajectory based on the waypoints to move the effector from the pre-grasp pose to the grasp pose); and
controlling the end effector to grasp an object in the workspace based on the calculated grasp trajectory associated to the selected grasp proposal (col 8 lines55 – col 9 line 13 discloses controlling the end effector based on the best proposal).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Fan to incorporate the teaching of Shafer of determining, for each grasp proposal in the set of grasp proposals, a respective set of waypoints comprising a pre-grasp pose and a grasp pose within the workspace based on position values of the grasp proposal; selecting a grasp proposal from the set based on the grasp proposal scores calculating a grasp trajectory for the selected grasp proposal based on the set of waypoints for the selected grasp proposal; controlling the end effector to grasp an object in the workspace based on the calculated grasp trajectory associated to the selected grasp proposal in order to determine a path to traverse the effector to the grasp path as taught by Shafer col 8 lines55 – col 9 line 13).
Fan as modified by Shafer does not teach generating, for each grasp proposal, a respective grasp proposal score based on scene features appearing within the respective window. Wherein a higher score is assigned to grasp proposal having fewer edges within the respective window.
Davidson teaches generating, for each grasp proposal, a respective grasp proposal score based on scene features appearing within the respective window ([0059]-[0060] disclosing determining for the instance “scene” that the glasses “feature” is in the gripper “window” to determine a measure of success “score”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Fan as modified by Shafer to incorporate the teaching of Davidson of generating, for each grasp proposal, a respective grasp proposal score based on scene features appearing within the respective window in order to measure the grasp outcome as taught by Davidson [0059].
Fan as modified by Shafer and Davidson does not teach Wherein a higher score is assigned to grasp proposal having fewer edges within the respective window.
Jeong teaches Wherein a higher score is assigned to grasp proposal having fewer edges within the respective window ([0006], [0045] disclosing scoring greater success to picks including picking an object near a center of a surface of the object and that does not overlap with another object, i.e., it is interpreted that the center is away from the edges of the surface “window”, thus the score is higher for the fewer edges, also since it does not overlap with other object that would include another edge, thus it will have fewer edges in the pickup window).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Fan as modified by Shafer and Davidson to incorporate the teaching of Jeong of Wherein a higher score is assigned to grasp proposal having fewer edges within the respective window in order to optimize the picking of an object as taught by Jeong [0006], [0045].
Moreira teaches determining the boundaries of a portion of a container by fitting point cloud data associated with an image of the virtual workspace captured by a camera to identify one or more two-dimensional planes, and applying one or more heuristics related to a respective orientation of each identified plane to determine boundaries ([0093] disclosing cloud points for depth information to identify a location associated with a container wall and identifying the boundary as being in a horizontal direction away from the wall thus forming a parallel boundary to the wall).
It would be obvious to combine the teaching of Moreira yielding predictable results by identifying regions where a safety region that avoids walls thus improving safety of operation of a robot gripper and avoids damage to gripper and surrounding.
Chitta teaches wherein each grasping window represents a projection of the end effectors active surface on an x-y plane ([0004]-[0007] and [0033]-[0040] and figs 3A-3J disclosing the gripper area inside the box on an xy plane as the grasping window accounting for the presence of a wall of the container).
Wherein the respective window is contained within the boundaries of the virtual space ([0004]-[0007] and [0033]-[0040] and figs 3A-3J disclosing the gripper area inside the box on an xy plane as the grasping window accounting for the presence of a wall of the container).
It would have been obvious to combine the teaching of Chitta in order to represent the full area of the gripper and determine poses within the wall thus optimizing the grasping technique and avoiding a collision with the container.
Regarding claim 3, Fan as modified by Shafer and Davidson and Jeong teaches the method of claim 1, wherein the set of grasp proposals spans a portion of the workspace’s x-y dimension (Fan ([0024] disclosing determining candidate grasps “proposal” wherein each grasp has a grasp pose “different position” and point map “grasping window” which are defined in the x,y, and z coordinates).
Regarding claim 4, Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta teaches the method of claim 3, wherein each grasp proposal has a respective z-value defined by scene topography under the associated window (Fan ([0024] disclosing determining candidate grasps “proposal” wherein each grasp has a grasp pose “different position” and point map “grasping window” which are defined in the x,y, and z coordinates. Figure 3 310 shows when a rabbit is picked from the top, the point map of the grasps is defined by the scene which is the rabbit shape, i.e., the z value is defined by the scene topography under the associated window).
Claims 13, 20 are rejected for similar reasons as claim 1, see above rejection.
Claim s 15-16 are rejected for similar reasons as claims 3-4 respectively, see above rejection.
Claims 5, 17 are rejected under 35 U.S.C. 103 as being unpatentable by Fan (US20220072707) in view of Shafer (US11331799) and Davidson (US20210053217) and Jeong (US20210016454) and Moreira (US20220281114) and Chitta (US20200047331) and Holson (US20200171653).
Regarding claim 5, Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta method of claim 1. Fan as modified by Shafer and Davidson and Moreira and Chitta does not teach wherein the set of waypoints are based on one or more safety margins.
Holson teaches wherein the set of waypoints are based on one or more safety margins ([0031] disclosing the trajectory “set of waypoints” are planned such that all parts of the robot are within a distance “safety margin” away from an object).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta to incorporate the teaching of Holson of wherein the set of waypoints are based on one or more safety margins in order to avoid collision with objects.
Claim 17 is rejected for similar reasons as claim 5, see above rejection.
Claims 6, 7, 8, 18, 19 are rejected under 35 U.S.C. 103 as being unpatentable by Fan (US20220072707) in view of Shafer (US11331799) and Davidson (US20210053217) and Jeong (US20210016454) and Moreira (US20220281114) and Chitta (US20200047331) and Humayun (US20220016766, from IDS) and Oka (US20200391385).
Regarding claim 6, Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta teaches the method of claim 1, Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta does not teach wherein the set of waypoints comprises an anticipatory pose outside the workspace.
Humayun teaches wherein the set of waypoints comprises an anticipatory pose ([0100] disclosing the waypoints includes a current position of the end effector to a pre-grasp position to a grasp position, the current position is interpreted as an anticipatory position).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta to incorporate the teaching of Humayun of wherein the set of waypoints are based on one or more safety margins in order to control movement along a trajectory from a current position of the robot as taught by Humayun [0100].
Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta and Humayun does not teach the waypoint is outside the workspace.
Oka teaches waypoint is outside the workspace ([0063] disclosing moving the manipulators to the inside of the container to pickup objects making sure the moving route does not contact the wall surface of the container).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta and Humayun to incorporate the teaching of Oka of waypoint is outside the workspace in order to define a route that avoids colliding with the container wall as taught by Oka [0063].
Regarding claim 7, Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta and Humayun and Oka teaches the method of claim 6. Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta and Humayun and Oka further teaches further comprising:
Specifically, Humayun teaches modeling movement of the end effector within the physical scene from the anticipatory pose to the pre-grasp pose and from the pre-grasp pose to the grasp pose ([0100]-[0101] disclosing planning and executing the grasp trajectory from a current position to the pre-grasp position and then to the grasp position and labeling the execution as successful or not, the execution by the robot in the physical scene to the movement is interpreted as modeling since it is to determine an outcome of the plan and to teach the robot);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta and Humayun and Oka to incorporate the teaching of Humayun of modeling movement of the end effector within the physical scene from the anticipatory pose to the pre-grasp pose and from the pre-grasp pose to the grasp pose in order to determine a successful or failure attempt to grasp an object as taught by Humayun [0100]-[0101].
optionally; determining that the end effector is simulated to collide with the workspace; and in response, updating the pre-grasp pose and anticipatory pose in the set of waypoints (the current limitations comes after optionally and thus does not have patentable weight).
Regarding claim 8, Fan as modified by Shafer and Davidson and Moreira and Chitta and Jeong and Humayun and Oka teaches the method of claim 7, further comprising re-scoring the grasp proposals after updating the set of waypoints (claim 8 depends on the optional limitations of claim 7 and thus do not have patentable weight).
Claims 18-19 are rejected for similar reasons as claims 6-7, see above rejection.
Claims 10 are rejected under 35 U.S.C. 103 as being unpatentable by Fan (US20220072707) in view of Shafer (US11331799) and Davidson (US20210053217) and Jeong (US20210016454) and Moreira (US20220281114) and Chitta (US20200047331) and Yongxiang (US20220388162).
Regarding claim 10, Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta teaches the method of claim 1, wherein generating a grasp proposal score based on the scene features appearing within the respective window (Davidson [0059]-[0060] disclosing determining for the instance “scene” that the glasses “feature” is in the gripper “window” to determine a measure of success “score”).
Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta comprises assigning higher scores to grasp proposals that are higher on the z-axis.
yongxian teaches comprises assigning higher scores to grasp proposals that are higher on the z-axis ([0031] disclosing the objects on top of the pile are good candidates to pick and will have a high quality, which is interpreted as being scored higher).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta to incorporate the teaching of yongxian of comprises assigning higher scores to grasp proposals that are higher on the z-axis in order to improve the quality of the grasp as taught by Yongxian [0031]. Fan teaches scoring grasp proposals, Yongxian teaches high quality grasp proposals to objects higher on the z-axia, It would have been obvious to one of ordinary skill in the art to have modified the teaching of Fan as modified by Yongxian to incorporate a higher score for grasp proposals higher on the z-axis which would yield predictable results.
Claims 11 are rejected under 35 U.S.C. 103 as being unpatentable by Fan (US20220072707) in view of Shafer (US11331799) and Davidson (US20210053217) and Jeong (US20210016454) and Moreira (US20220281114) and Chitta (US20200047331) and Sun (US20220289502)
Regarding claim 11, Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta teaches the method of claim 1, wherein generating a grasp proposal score based on the scene features appearing within the respective window Davidson [0059]-[0060] disclosing determining for the instance “scene” that the glasses “feature” is in the gripper “window” to determine a measure of success “score”).
Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta does not teach comprises assigning higher scores to grasp proposals that are closer to a front of the container.
Sun teaches comprises assigning higher scores to grasp proposals that are closer to a front of the container ([0063]-[0070] disclosing the item to a front of a chute has a higher success rate, i.e., the grasp proposal to pick the item in a front of the container which would cause a grasp to happen at a front of the container is scored higher).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta to incorporate the teaching of Sun of comprises assigning higher scores to grasp proposals that are closer to a front of the container in order to improve the success of picking up the item as taught by Sun [0063]-[0070].
Claims 12 are rejected under 35 U.S.C. 103 as being unpatentable by Fan (US20220072707) in view of Shafer (US11331799) and Davidson (US20210053217) and Jeong (US20210016454) and Moreira (US20220281114) and Chitta (US20200047331) and Aiglstorfer (US20210047115).
Regarding claim 12, Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta teaches method of claim 1. Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta further teaches further comprising rotating each grasp pose by a fixed degree increment.
Specifically, Davidson teaches comprising rotating each grasp pose by a fixed degree increment ([0007] disclosing new grasp poses within a variance “fixed degree increment”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Fan as modified by Shafer and Davidson and Jeong and Moreira and Chitta to incorporate the teaching of Davidson of comprising rotating each grasp pose by a fixed degree increment in order to generate new successful grasp attempts as taught by Davidson [0007].
Fan as modified by Shafer and Davidson and Jeong does not teach until the grasp pose does not collide with the container.
Aiglstorfer teaches until the grasp pose does not collide with the container ([0077]-[0080] disclosing the gripping pose is changed with a no collision with the side wall of the container).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Fan as modified by Shafer and Davidson and Jeong to and Moreira and Chitta incorporate the teaching of Aiglstorfer of until the grasp pose does not collide with the container in order to avoid collision with a container as taught by Aiglstorfer [0077]-[0080].
Response to Arguments
Applicant’s arguments filed on 09/17/2025 have been fully considered but they are not persuasive.
Applicant’s arguments with respect to the amended independent claims have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. The newly cited prior arts on the final rejection, specifically, Moreira teaches the cloud points to define boundaries within a container and Chitta teaches defining windows within walls of a container, see full rejection above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to
applicant's disclosure. The prior art cited in PTO-892 and not mentioned above disclose related devices and methods.
US20210170609 discloses not grasping object near edges.
US20200316782 discloses cutting objects into planes to determine graspable region.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMAD O EL SAYAH whose telephone number is (571)270-7734. The examiner can normally be reached on M-Th 6:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached on (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMAD O EL SAYAH/Examiner, Art Unit 3664B