Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This communication is responsive to Application No. 18/255,542 and amendments filed on 8/5/2025.
3. Claims 1-5, 8-9, and 25 are presented for examination.
Information Disclosure Statement
4. The information disclosure statements (IDS) submitted on 7/11/2023, 7/11/2024, and 1/14/2025 have been fully considered by the Examiner.
Response to Arguments
5. Applicant's arguments filed 8/5/2025 with respect to the objection to claim 5 for minor informalities have been fully considered but they are not persuasive.
The Examiner notes that claim 5 has not been amended to remove the duplicated instance of “one or more” recited in line 5 of claim 5. As such, the objection to claim 5 is maintained, in which will be described further below.
6. Applicant’s arguments, see page 6, filed 8/5/2025, with respect to the objection to claim 17 for minor informalities have been fully considered and are persuasive. The objection to claim 17 of 4/8/2025 has been withdrawn.
7. Applicant’s arguments, see page 7, filed 8/5/2025, with respect to the rejection of claims 1-6 and 10-24 under 35 U.S.C. 101 have been fully considered and are persuasive. The rejection of claims 1-6 and 10-24 under 35 U.S.C. 101 of 4/8/2025 has been withdrawn.
8. Applicant’s arguments with respect to the rejection of claim(s) 1-24 under 35 U.S.C. 103 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Regarding independent claim 1, the Examiner agrees that the combination of US 20210069904 A1 to Duan and US 20180295783 A1 to Alexander fails to teach all of the limitations to amended claim 1. However, in light of the amendments and the Applicant’s remarks, an updated search was conducted, and a new ground of rejection concerning claim 1 has been determined, in which will be described later.
Regarding dependent claims 2-5 and 8-9, as all of these claims depend from claim 1, are still rejected, in which will be described later.
Regarding claims 6-7 and 10-24, these claims have been cancelled, and thus, are withdrawn from further consideration.
Claim Objections
9. Claim 5 is objected to because of the following informalities:
Regarding Claim 5, the term "based on the one or more one or more grasp locations" recited in lines 5-6 of the claim is objected to for its unnecessary duplication of the term "one or more.".
Appropriate correction is required.
Claim Rejections - 35 USC § 103
10. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
11. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
12. Claim(s) 1, 2, 8, 9, and 25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Duan et al. (US 20210069904 A1 hereinafter Duan) in view of Amano et al. (US 20220092330 A1 hereinafter Amano).
Regarding Claim 1, Duan teaches a control system comprising: a grasp control system for a robot ([0034] via “FIG. 1 illustrates an example of commercial environment 100 having a robotic arm for picking and perturbing items in a bin in accordance with some embodiments of the present technology.”), ([0082] via “For example, software 915 may include program instructions for implementing a robot control process, ….”); and
a data processing apparatus for grasp generation ([0045] via “The ranking model may include one or more deep neural nets that have been trained for ranking the probability of success in pick-up options. The ranking model may provide a set of proposed pick-up strategies including grasps and orientations within the range of the 6DOF robotic arm.”), wherein the data processing apparatus is configured to:
obtain image data, comprising depth data, representative of an image of an object captured by a camera ([0046] via “In step 405, the machine assesses the bin to collect information on the current state of items in the bin. The machine may use visual sensing, imaging, or any other type of sensing helpful for identifying items in the bin. In some examples, the machine takes one or more images using at least one camera to see the arrangement of items in the bin.”), ([0047] via “During step 410, the machine may further reason about the geometry of distinct objects, the depth or distance of objects with respect to the machine or bin, the material properties of distinct objects, or similar qualities and features including combinations thereof.”);
provide the image data to a grasping model comprising a neural network trained to predict a plurality of outcomes associated with a grasping operation independently based on images of graspable objects ([0046] via “In step 405, the machine assesses the bin to collect information on the current state of items in the bin. The machine may use visual sensing, imaging, or any other type of sensing helpful for identifying items in the bin.”), ([0047] via “In step 410, the machine uses data collected in step 405 to reason about the situation in the bin. In some embodiments, this reasoning includes segmentation. Via segmentation, the machine may reason about or identify distinct objects within the bin. In some examples, segmentation is implemented within a machine learning algorithm, such as a deep neural net, wherein the machine may learn how to effectively identify distinct objects and subsequently improve at segmentation.”), ([0048] via “In step 415, the machine reasons about the items and points with the highest probability of picking success. Reasoning performed by the machine may be based on knowledge from previous picking experience or attempts. … In certain embodiments, the machine uses a ranking model for both objects and places. Using the ranking model, the machine can rank the items in view by the probability that the machine could successfully pick up each item. Similarly, the machine may rank points on various objects according to the probability of a successful pick-up at that point. Based on the rankings, the machine may produce a set of proposed grasps and orientations to attempt to pick the items. The reasoning performed in step 415 may, similarly to step 410, be implemented in a machine learning structure. Using a trained deep neural net, the machine can learn from previous failures and continually improve at identifying and choosing picking strategies based on the information provided.”);
obtain a plurality of pixelwise predictions corresponding to the plurality of outcomes, each pixelwise prediction being a representation of pixelwise probability values corresponding to a given outcome of the plurality of outcomes associated with the grasping operation ([0067] via “After failing to pick up the marker, in step 815, the robotic arm transfers the knowledge gained from the failure and attempts to generalize it to other areas of the marker. … The failure at the edge of the marker cap may be generalized to other pixels or regions that contain elements similar to point 1, such as all pixels comprising the edge of the cap. In some examples, the edge of the marker may have infinite points, so if the machine only learned to avoid the single point it attempted in step 805, it may continue to try other points along the edge of the marker, although they are just as likely to cause the machine to fail to pick up the marker. Thus, any pixels or groups of pixels that appear similar to that of the failure point may be indicated as having a high probability of failure. Other pixels less similar to point 1
may be left unscored or scored with a low probability of failure until the machine learns otherwise.”); and
output the aggregated pixelwise outcome prediction for selection therefrom of one or more pixels on which to base generation of one or more grasp poses to grasp the object ([0068] via “After rating the other pixels according to their probability of failure, the robotic arm attempts to pick up the marker again, this time at point 2, in step 820. Point 2 is not the same as point 1. In the present example, point 2 is a position that is not already predicted to have a high probability of failure.”), ([0070] via “Finally, in step 835 the robotic arm attempts to pick the marker at point 3, wherein point 3 is different from points 1 and 2. In step 840, the robotic arm successfully picks the marker. In the present example, the robotic arm succeeds at a task after only two failed attempts. However, in other examples, a machine may succeed on the first try, second try, fourth try, or on any other attempt. Furthermore, a machine may continue learning about probability of failure after a successful attempt.”), (Note: The Examiner interprets that in combination with Amano, which teaches obtaining an aggregated pixelwise outcome prediction below, that this limitation is satisfied.);
wherein the data processing apparatus includes a pose generator ([0039] via “In step 210, the machine uses data collected in step 205 to determine a picking strategy.”), ([0048] via “In certain embodiments, the machine uses a ranking model for both objects and places. Using the ranking model, the machine can rank the items in view by the probability that the machine could successfully pick up each item. Similarly, the machine may rank points on various objects according to the probability of a successful pick-up at that point. Based on the rankings, the machine may produce a set of proposed grasps and orientations to attempt to pick the items.”), wherein the pose generator is configured to:
obtain one or more grasp locations corresponding to one or more pixels selected from the aggregated pixelwise outcome prediction based on one or more corresponding probability values in the aggregated pixelwise outcome prediction ([0067] via “After failing to pick up the marker, in step 815, the robotic arm transfers the knowledge gained from the failure and attempts to generalize it to other areas of the marker. … The failure at the edge of the marker cap may be generalized to other pixels or regions that contain elements similar to point 1, such as all pixels comprising the edge of the cap. In some examples, the edge of the marker may have infinite points, so if the machine only learned to avoid the single point it attempted in step 805, it may continue to try other points along the edge of the marker, although they are just as likely to cause the machine to fail to pick up the marker. Thus, any pixels or groups of pixels that appear similar to that of the failure point may be indicated as having a high probability of failure. Other pixels less similar to point 1 may be left unscored or scored with a low probability of failure until the machine learns otherwise.”), ([0068] via “After rating the other pixels according to their probability of failure, the robotic arm attempts to pick up the marker again, this time at point 2, in step 820. Point 2 is not the same as point 1. In the present example, point 2 is a position that is not already predicted to have a high probability of failure.”); and
determine one or more grasp poses based on the one or more one or more grasp locations ([0048] via “In certain embodiments, the machine uses a ranking model for both objects and places. Using the ranking model, the machine can rank the items in view by the probability that the machine could successfully pick up each item. Similarly, the machine may rank points on various objects according to the probability of a successful pick-up at that point. Based on the rankings, the machine may produce a set of proposed grasps and orientations to attempt to pick the items.”);
wherein the grasp control system includes a controller ([0081] via “Storage system 910
may comprise additional elements, such as a controller, capable of communicating with processing system 925 or possibly other systems.”) configured to obtain the one or more grasp poses from the pose generator and control a robotic manipulator to grasp the object based on the one or more grasp poses ([0048] via “Based on the rankings, the machine may produce a set of proposed grasps and orientations to attempt to pick the items.”), ([0049] via “In step 420, the machine attempts to pick up an item by the highest ranked (i.e., most promising) point in the bin and move the item to a new location. In some examples, the new location may be a different bin. … If the machine successfully picks up the item and moves it to its new location, the machine may continue down the rankings and attempt to pick up an item by the next highest ranked point in the bin.”), ([0075] via “Alternatively, the robotic device may be controlled by a trained neural net implemented entirely in software on an external computing system, or may be implemented as a combination of the two across one or more devices.”).
Duan is silent on wherein the data processing apparatus is configured to: aggregate the plurality of pixelwise predictions to obtain an aggregated pixelwise outcome prediction.
However, Amano teaches to aggregate the plurality of pixelwise predictions to obtain an aggregated pixelwise outcome prediction ([0029] via “In the model creation process of FIG. 4
(S120), image processing device 30 first matches the feature points P extracted from different two-dimensional images Gi (S300) and counts the number of times the matched feature points P appear in each the two-dimensional image Gi (S310). … In FIG. 7, the feature points P (e.g., Ps) at the four corners of the main face of workpiece W in two-dimensional image G0
match with the feature points P at the four corners of the main face of workpiece W in two-dimensional images G1 to G8, respectively, and since they appear in each of two-dimensional images G0 to G8, the number of appearances is counted as having a value of 9.”), ([0033] via “Next, image processing device 30 projects an image of three-dimensional shape model M on two-dimensional image Ga based on the set approximate positions Pr (S520, see FIG. 11B). … Next, image processing device 30 performs a position adjustment so that the degree of overlap between the projected image of three-dimensional shape model M and workpiece W in two-dimensional image Ga increases (S530), identifies the position at which the degree of overlap is at a maximum (x, y, z, Rx, Ry, Rz) as the three-dimensional position of the feature point P, recognizes the position and orientation of workpiece W (S540), and terminates the matching process. For example, in S530, S540, image processing device 30 acquires the brightness difference between the pixel of interest and the overlapping pixel while adjusting the position of the projected image on a pixel-by-pixel basis, and identifies the position where the degree of overlap is at a maximum by detecting the position where the rate of change of the interpolated waveform is 0 on a sub-pixel basis. ... In this manner, image processing device 30 can recognize the position and orientation of workpiece W from two-dimensional image Ga and three-dimensional shape model M. In addition, since control device 18 controls robot arm 22 based on the recognized position and orientation of workpiece W, it is possible to properly pick up workpiece W.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Amano wherein the data processing apparatus is configured to: aggregate the plurality of pixelwise predictions to obtain an aggregated pixelwise outcome prediction. Doing so increases the accuracy in determining a position of the object, improving the accuracy in properly picking up the object, as stated by Amano ([0035] via “In work robot 20 of the first embodiment described above, image processing device 30 recognizes the position and orientation of workpiece W by matching the feature points P of two-dimensional image Ga of workpiece W captured by camera 24 with the feature points Pm of three-dimensional shape model M. Therefore, since image processing device 30 does not have to perform the process for all points in two-dimensional image Ga and can properly perform matching based on the feature descriptor f of the feature point P, the position and orientation of workpiece W can be accurately recognized while suppressing the processing load. Further, work robot 20 can improve work accuracy by properly picking up workpiece W.”).
Regarding Claim 2, modified reference Duan teaches the control system of claim 1, wherein the data processing apparatus is configured to implement the grasping model ([0070] via “Finally, in step 835 the robotic arm attempts to pick the marker at point 3, wherein point 3 is different from points 1 and 2. In step 840, the robotic arm successfully picks the marker. In the present example, the robotic arm succeeds at a task after only two failed attempts. However, in other examples, a machine may succeed on the first try, second try, fourth try, or on any other attempt.”).
Regarding Claim 8, modified reference Duan teaches the control system of claim 1, further comprising the robotic manipulator ([0034] via “FIG. 1 illustrates an example of commercial environment 100 having a robotic arm for picking and perturbing items in a bin in accordance with some embodiments of the present technology.”).
Regarding Claim 9, modified reference Duan teaches the control system of claim 8, wherein the robotic manipulator comprises an end effector for grasping the object ([0034] via “Robotic arm comprises vision system 110, picking element 115, and perturbation element 120.”), ([0035] via “Picking element 115 may comprise one or more picking mechanisms for grabbing items in bin 125. Picking mechanisms may include a suction mechanism, a gripping mechanism, a robotic hand, a pinching mechanism, a magnet, or any other picking mechanism that may be used in accordance with the present disclosure.”).
Regarding Claim 25, modified reference Duan teaches the control system of claim 9, wherein the end effector comprises at least one of a jaw gripper, a finger gripper, a magnetic or electromagnetic gripper, a Bernoulli gripper, a vacuum suction cup, an electrostatic gripper, a van der Waals gripper, a capillary gripper, a cryogenic gripper, an ultrasonic gripper, or a laser gripper ([0034] via “Robotic arm comprises vision system 110, picking element 115, and perturbation element 120.”), ([0035] via “Picking element 115 may comprise one or more picking mechanisms for grabbing items in bin 125. Picking mechanisms may include a suction mechanism, a gripping mechanism, a robotic hand, a pinching mechanism, a magnet, or any other picking mechanism that may be used in accordance with the present disclosure.”).
13. Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Duan et al. (US 20210069904 A1 hereinafter Duan) in view of Amano et al. (US 20220092330 A1 hereinafter Amano), and further in view of Menon et al. (US 20210122056 A1 hereinafter Menon).
Regarding Claim 3, modified reference Duan teaches the control system of claim 1, wherein the plurality of outcomes associated with the grasping operation comprises: a successful grasp of the object ([0045] via “The output or outputs of unified reasoning module 310 serves as input to ranking and decision module 315. Ranking and decision module 315
may process the unified model provided by unified reasoning module 310 to produce a ranking of potential pick-up points. The ranking may include a ranking of items according to probability of successful pick-up and a ranking of points on one item or multiple items according to the probabilities of successful pick-up.”).
Duan is silent on wherein the plurality of outcomes associated with the grasping operation comprises at least one or more of: a successful scan of the object; a successful placement of the object; a successful subsequent grasp of another object; an avoidance of grasping another object with the object; or an avoidance of stopping the grasping operation.
However, Menon teaches wherein the plurality of outcomes associated with the grasping operation comprises at least one or more of: a successful scan of the object; a successful placement of the object; a successful subsequent grasp of another object; an avoidance of grasping another object with the object; or an avoidance of stopping the grasping operation ([0107] via “The probability that the grasping strategy will result in a successful grasp of the item may be based on one more grasping factors, such as contextual information about the environment, …, etc. Contextual information about the environment includes the existence of other items near or adjacent to the item, the amount that the other items near or adjacent to the item hinder an ability of a robotic arm to grasp the item, whether more items are continuously being added to a workspace area, etc.”), (Note: The Examiner interprets this citation of Menon to teach “an avoidance of grasping another object.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Menon wherein the plurality of outcomes associated with the grasping operation comprises at least one or more of: a successful scan of the object; a successful placement of the object; a successful subsequent grasp of another object; an avoidance of grasping another object with the object; or an avoidance of stopping the grasping operation. Doing so encourages the robot to not choose grasp locations where other surrounding objects would interfere with the robot’s ability to pick, as stated above by Menon.
14. Claim(s) 4 and 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Duan et al. (US 20210069904 A1 hereinafter Duan) in view of Amano et al. (US 20220092330 A1 hereinafter Amano), and further in view of Kalakrishnan et al. (US 10861184 B1 hereinafter Kalakrishnan).
Regarding Claim 4, modified reference Duan teaches the control system of claim 1, further comprising a grasp control system for a robot ([0034] via “FIG. 1 illustrates an example of commercial environment 100 having a robotic arm for picking and perturbing items in a bin in accordance with some embodiments of the present technology.”), ([0082] via “For example, software 915 may include program instructions for implementing a robot control process, ….”).
Duan is silent on wherein the grasp control system is configured to: obtain the aggregated pixelwise outcome prediction and one or more pixelwise heuristic maps, each representative of pixelwise heuristic values corresponding to a given heuristic; and combine the aggregated pixelwise outcome prediction and the one or more pixelwise heuristic maps to obtain a combined pixelwise map.
However, Kalakrishnan teaches to: obtain the aggregated pixelwise outcome prediction and one or more pixelwise heuristic maps, each representative of pixelwise heuristic values corresponding to a given heuristic; and combine the aggregated pixelwise outcome prediction and the one or more pixelwise heuristic maps to obtain a combined pixelwise map (Col. 6 lines 48-62, where “The system provides the image as an input to a neural network subsystem to generate heat maps for one or more feature points of the object (320). The neural network subsystem is trained to receive images of objects and to generate respective heat maps for each of the feature points of the object in the image. A heat map for a feature point maps each region of the image to a score representing a likelihood that the region corresponds to the feature point. The system applies a differentiable transformation on each heat map to generate one or more feature coordinates for each of the one or more feature points (330). The transformation being differentiable allows the system to be trained end-to-end using stochastic gradient descent with backpropagation, a technique that requires computing partial derivatives.”), (Note: The Examiner interprets the original generated heatmap as the aggregated pixelwise prediction and the differentiable transformation of the original heatmap as applying the heuristic information to it to obtain a combined pixelwise map.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Kalakrishnan wherein the grasp control system is configured to: obtain the aggregated pixelwise outcome prediction and one or more pixelwise heuristic maps, each representative of pixelwise heuristic values corresponding to a given heuristic; and combine the aggregated pixelwise outcome prediction and the one or more pixelwise heuristic maps to obtain a combined pixelwise map. Doing so allows the pixelwise map to be trained end-to-end, as stated above by Kalakrishnan.
Regarding Claim 5, modified reference Duan teaches the control system data processing apparatus of claim 4, comprising a pose generator configured to: obtain one or more grasp locations corresponding to one or more pixels sampled from the combined pixelwise map ([0067] via “After failing to pick up the marker, in step 815, the robotic arm transfers the knowledge gained from the failure and attempts to generalize it to other areas of the marker. … The failure at the edge of the marker cap may be generalized to other pixels or regions that contain elements similar to point 1, such as all pixels comprising the edge of the cap. In some examples, the edge of the marker may have infinite points, so if the machine only learned to avoid the single point it attempted in step 805, it may continue to try other points along the edge of the marker, although they are just as likely to cause the machine to fail to pick up the marker. Thus, any pixels or groups of pixels that appear similar to that of the failure point may be indicated as having a high probability of failure. Other pixels less similar to point 1
may be left unscored or scored with a low probability of failure until the machine learns otherwise.”), ([0068] via “After rating the other pixels according to their probability of failure, the robotic arm attempts to pick up the marker again, this time at point 2, in step 820. Point 2
is not the same as point 1. In the present example, point 2 is a position that is not already predicted to have a high probability of failure.”); and
determine one or more grasp poses based on the one or more one or more grasp locations ([0048] via “In certain embodiments, the machine uses a ranking model for both objects and places. Using the ranking model, the machine can rank the items in view by the probability that the machine could successfully pick up each item. Similarly, the machine may rank points on various objects according to the probability of a successful pick-up at that point. Based on the rankings, the machine may produce a set of proposed grasps and orientations to attempt to pick the items.”).
Examiner’s Note
15. The Examiner has cited particular paragraphs or columns and line numbers in the
references applied to the claims above for the convenience of the Applicant. Although the
specified citations are representative of the teachings of the art and are applied to specific
limitations within the individual claim, other passages and figures may apply as well. It is
respectfully requested of the Applicant in preparing responses, to fully consider the references
in their entirety as potentially teaching all or part of the claimed invention, as well as the
context of the passage as taught by the prior art or disclosed by the Examiner. See MPEP
2141.02 [R-07.2015] VI. A prior art reference must be considered in its entirety, i.e., as a whole,
including portions that would lead away from the claimed Invention. W.L. Gore & Associates,
Inc. v. Garlock, Inc., 721 F.2d 1540, 220 USPQ 303 (Fed. Cir. 1983), cert, denied, 469 U.S. 851
(1984). See also MPEP §2123.
Conclusion
16. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
17. Any inquiry concerning this communication or earlier communications from the
examiner should be directed to BYRON X KASPER whose telephone number is (571)272 -3895.
The examiner can normally be reached Monday - Friday 8 am - 5 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing
using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is
encouraged to use the USPTO Automated Interview Request (AIR) at
http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s
supervisor, Adam Mott can be reached on (571) 270-5376. The fax phone number for the
organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be
obtained from Patent Center. Unpublished application information in Patent Center is available
to registered users. To file and manage patent submissions in Patent Center, visit:
https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for
more information about Patent Center and https://www.uspto.gov/patents/docx for
information about filing in DOCX format. For additional questions, contact the Electronic
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO
Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BYRON XAVIER KASPER/Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657