DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Status of Claims
Claims 1 and 10 have been amended. Claims 1-17 are now pending.
Effective Filing Date
Examiner acknowledges that the instant application claims domestic benefit to provisionally filed application 63675066, filed 07/24/2024. This provisionally filed application has been reviewed and provides sufficient support for the instant application’s claims. As such, the effective filing date of the application is 07/24/2024.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 6-11, and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Sun et al. (US 20210122586 A1), hereinafter Sun in view of Shafer (US 11331799 B1), hereinafter Shafer.
Regarding claim 1, Sun discloses:
A method for logging and analyzing robotic picking data for a robotic pick-and-place system, comprising:
attempting a pick with a robotic gripper of the pick-and-place system (See at least [0073]: “At 322, the system attempts to grasp one or more items using the strategy determined at 320. For example, the end effector of a robotic arm may be moved to a position adjacent to the item(s) to be grasped, according to the determined strategy, and the end effector may be operated according to the determined strategy to attempt to grasp the item(s).”)
determining whether the pick was successfully executed (see at least [0074]: “At 324 a determination is made as to whether the grasp was successful. For example, image data and/or force (weight), pressure, proximity, and/or other sensor data may be used to determine whether the item was grasped successfully. If so, at 326 the item(s) is/are moved to the conveyor. If not, processing returns to step 320, at which a new strategy to grasp the item is determined (if available).”)
using the determination of the pick success to compute a throughput of the robotic pick-and-place system; and storing the computed throughput in the analytics log (see at least [0102]: “In the example shown in FIG. 5C, at 562, collective throughput, local singulation station observed and/or estimated (locally scheduled) throughput, and overall and station-specific error rates are monitored. At 564, the conveyor speed and local station speeds are adjusted to maximize collective throughput net or errors. At 566, conveyor slots are allocated to respective stations to maximize net throughput. While in this example conveyor slots are allocated/assigned explicitly, in some embodiments station speeds are controlled so as to ensure downstream stations have slots available to place items, without (necessarily) pre-assigning specific slots to specific stations. Processing continues (562, 564, 566) while any station has items remaining to be picked/placed (568).”)
Sun does not explicitly disclose, but Shafer teaches:
logging information about the pick with data logging and analysis logic in an analytics log, the that stores a plurality of per-pick records, each per-pick record comprising at least determination of the pick success for a respective pick (see at least col. 7, lines 32-42: “In some implementations, grasps attempted using the above techniques are “labeled” as successful or unsuccessful. For example, they can be automatically labeled using automated technique(s) that determine whether a grasp is successful based on an end effector “closing extent” when attempting a grasp (e.g., grasp unsuccessful if close all the way, otherwise successful), end effector torque reading(s) when attempting a grasp (e.g., torque spike when partially closed indicates success), and/or based on capturing additional end effector vision sensor data after the grasp and “lifting” (e.g., to determine if an object still in field of view).”)
It would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention, with a reasonable expectation for success, to combine the invention of Sun with the per-pick records as taught by Shafer. This is because as stated in col. 7, line 43 – col. 8 line 7 of Shafer: “In those implementations, the end effector vision data and the utilized grasp pose can be stored, along with the grasp success label. This data can be used to generate corresponding training instance(s), and a machine learning model trained based on the training instance(s). The machine learning model, once trained, can be used to process end effector vision data (or features determined based thereon) and predict a final grasp pose. For example, the machine learning model can be used to process vision data and predict x & y coordinates (and optionally z) of grasp point(s) (e.g., in a “vision data frame”) and optionally to predict an end effector “rotation” value (i.e., about the z axis) and/or other orientation value(s). As another example, the machine learning model can be used to process vision data (or features based thereon) and a candidate grasp pose and generate a value that indicates likelihood of successful grasp using the candidate grasp pose and in view of the vision data (or features based thereon). Once the ML model is trained, it can be used in predicting grasp poses in lieu of (or in addition to) utilizing the pre-stored instances of visual features described herein. As one non-limiting example, multiple candidate poses can be determined utilizing the pre-stored instances of visual features, then one of the candidate poses selected using a trained machine learning model that predicts a grasp success measure based on processing a corresponding grasp pose and a corresponding instance of end effector vision data (or features based thereon). For instance, each of the multiple candidate poses can be processed, using the machine learning model and along with corresponding vision data, and the one with the best resulting grasp success measure selected for utilization in generating a grasp path.”
Regarding claim 2, the combination of Sun and Shafer teaches:
The method of claim 1.
Sun further discloses wherein the information further comprises information used by one or more filtering or sorting rules to select a target object for the pick (see at least [0059]: “In various embodiments, an arbitrary mix of items to be singulated may include parcels, packages, and/or letters of a variety of shapes and sizes. Some may be standard packages one or more attributes of which may be known, others may be unknown. Image data is used, in various embodiments, to discern individual items (e.g., via image segmentation). The boundaries of partially occluded items may be estimated, e.g., by recognizing an item as a standard or known type and/or extending visible item boundaries to logical estimated extents (e.g., two edges extrapolated to meet at an occluded corner). In some embodiments, a degree of overlap (i.e., occlusion by other items) is estimated for each item, and the degree of overlap is taken into consideration in selecting a next item to attempt to grasp. For example, for each item a score may be computed to estimate the probability of grasp success, and in some embodiments the score is determined at least in part by the degree of overlap/occlusion by other items. Less occluded items may be more likely to be selected, for example, other considerations being equal.”)
Regarding claim 6, the combination of Sun and Shafer teaches:
The method of claim 2.
Sun further discloses wherein the information comprises a number of available picks as determined by the filtering or sorting rules at a time of the pick (see at least [0108]: “In various embodiments, the system may take a photo and identify two (or more) objects to pick. The system picks and moves the first one; then, instead of doing a full scene re-compute to find a next package to pick, the system simply looks at whether the second package is disturbed. If not, the system picks it without doing a full recompute of the scene, which typically would save a lot of time. In various embodiments, the time savings is in one or more of sensor read latency, image acquisition time, system compute time, re-segmentation, masking, package pile ordering computations, and finally control and planning. If the second item is not where expected, the system does a full scene re-compute to find a next package to pick.”)
Regarding claim 7, the combination of Sun and Shafer teaches:
The method of claim 1.
Sun further discloses wherein the information further comprises an amount of time required to detect the target object (see at least [0108]: “In various embodiments, the system may take a photo and identify two (or more) objects to pick. The system picks and moves the first one; then, instead of doing a full scene re-compute to find a next package to pick, the system simply looks at whether the second package is disturbed. If not, the system picks it without doing a full recompute of the scene, which typically would save a lot of time. In various embodiments, the time savings is in one or more of sensor read latency, image acquisition time, system compute time, re-segmentation, masking, package pile ordering computations, and finally control and planning. If the second item is not where expected, the system does a full scene re-compute to find a next package to pick.”)
Regarding claim 8, the combination of Sun and Shafer teaches:
The method of claim 1.
Sun further discloses wherein the throughput represents an average pick rate over a predetermined period of time (see at least [0066-0067]: “While in the example shown in FIG. 2B each station has one robotic arm, in various embodiments two or more robots may be deployed at a station, operated under control of an associated control computer, such as control computer 212 in the example shown in FIG. 2B, in a manner that avoids the robots interfering with each other's operation and movement and which maximizes their collective throughput, including by avoiding and/or managing contention to pick and place the same item. In various embodiments, a scheduler coordinates operation of a plurality of robots, e.g., one or more robots working at each of a plurality of stations, to achieve desired throughput without conflict between robots, such as one robot placing an item in a location the scheduler has assigned to another robot.”)
Regarding claim 9, the combination of Sun and Shafer teaches:
The method of claim 1.
Sun further discloses:
A system comprising:
a robotic arm; a conveyor for conveying objects to the robotic arm; a sensor; (see at least Fig. 2A.)
and a processor configured to perform the method of claim 1 (see at least [0032]: “The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor.”)
Regarding claim 10, Sun discloses:
A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:
attempt a pick with a robotic gripper of the pick-and-place system (See at least [0073]: “At 322, the system attempts to grasp one or more items using the strategy determined at 320. For example, the end effector of a robotic arm may be moved to a position adjacent to the item(s) to be grasped, according to the determined strategy, and the end effector may be operated according to the determined strategy to attempt to grasp the item(s).”)
determine whether the pick was successfully executed (see at least [0074]: “At 324 a determination is made as to whether the grasp was successful. For example, image data and/or force (weight), pressure, proximity, and/or other sensor data may be used to determine whether the item was grasped successfully. If so, at 326 the item(s) is/are moved to the conveyor. If not, processing returns to step 320, at which a new strategy to grasp the item is determined (if available).”)
use the determination of the pick success to compute a throughput of the robotic pick-and-place system; and store the computed throughput in the analytics log (see at least [0102]: “In the example shown in FIG. 5C, at 562, collective throughput, local singulation station observed and/or estimated (locally scheduled) throughput, and overall and station-specific error rates are monitored. At 564, the conveyor speed and local station speeds are adjusted to maximize collective throughput net or errors. At 566, conveyor slots are allocated to respective stations to maximize net throughput. While in this example conveyor slots are allocated/assigned explicitly, in some embodiments station speeds are controlled so as to ensure downstream stations have slots available to place items, without (necessarily) pre-assigning specific slots to specific stations. Processing continues (562, 564, 566) while any station has items remaining to be picked/placed (568).”)
Sun does not explicitly disclose, but Shafer teaches:
log information about the pick with data logging and analysis logic in an analytics log, the that stores a plurality of per-pick records, each per-pick record comprising at least determination of the pick success for a respective pick (see at least col. 7, lines 32-42: “In some implementations, grasps attempted using the above techniques are “labeled” as successful or unsuccessful. For example, they can be automatically labeled using automated technique(s) that determine whether a grasp is successful based on an end effector “closing extent” when attempting a grasp (e.g., grasp unsuccessful if close all the way, otherwise successful), end effector torque reading(s) when attempting a grasp (e.g., torque spike when partially closed indicates success), and/or based on capturing additional end effector vision sensor data after the grasp and “lifting” (e.g., to determine if an object still in field of view).”)
It would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention, with a reasonable expectation for success, to combine the invention of Sun with the per-pick records as taught by Shafer. This is because as stated in col. 7, line 43 – col. 8 line 7 of Shafer: “In those implementations, the end effector vision data and the utilized grasp pose can be stored, along with the grasp success label. This data can be used to generate corresponding training instance(s), and a machine learning model trained based on the training instance(s). The machine learning model, once trained, can be used to process end effector vision data (or features determined based thereon) and predict a final grasp pose. For example, the machine learning model can be used to process vision data and predict x & y coordinates (and optionally z) of grasp point(s) (e.g., in a “vision data frame”) and optionally to predict an end effector “rotation” value (i.e., about the z axis) and/or other orientation value(s). As another example, the machine learning model can be used to process vision data (or features based thereon) and a candidate grasp pose and generate a value that indicates likelihood of successful grasp using the candidate grasp pose and in view of the vision data (or features based thereon). Once the ML model is trained, it can be used in predicting grasp poses in lieu of (or in addition to) utilizing the pre-stored instances of visual features described herein. As one non-limiting example, multiple candidate poses can be determined utilizing the pre-stored instances of visual features, then one of the candidate poses selected using a trained machine learning model that predicts a grasp success measure based on processing a corresponding grasp pose and a corresponding instance of end effector vision data (or features based thereon). For instance, each of the multiple candidate poses can be processed, using the machine learning model and along with corresponding vision data, and the one with the best resulting grasp success measure selected for utilization in generating a grasp path.”
Regarding claim 11, the combination of Sun and Shafer teaches:
The computer-readable storage medium of claim 10.
Sun further discloses wherein the information further comprises information used by one or more filtering or sorting rules to select a target object for the pick (see at least [0059]: “In various embodiments, an arbitrary mix of items to be singulated may include parcels, packages, and/or letters of a variety of shapes and sizes. Some may be standard packages one or more attributes of which may be known, others may be unknown. Image data is used, in various embodiments, to discern individual items (e.g., via image segmentation). The boundaries of partially occluded items may be estimated, e.g., by recognizing an item as a standard or known type and/or extending visible item boundaries to logical estimated extents (e.g., two edges extrapolated to meet at an occluded corner). In some embodiments, a degree of overlap (i.e., occlusion by other items) is estimated for each item, and the degree of overlap is taken into consideration in selecting a next item to attempt to grasp. For example, for each item a score may be computed to estimate the probability of grasp success, and in some embodiments the score is determined at least in part by the degree of overlap/occlusion by other items. Less occluded items may be more likely to be selected, for example, other considerations being equal.”)
Regarding claim 15, the combination of Sun and Shafer teaches:
The computer-readable storage medium of claim 11.
Sun further discloses wherein the information comprises a number of available picks as determined by the filtering or sorting rules at a time of the pick (see at least [0108]: “In various embodiments, the system may take a photo and identify two (or more) objects to pick. The system picks and moves the first one; then, instead of doing a full scene re-compute to find a next package to pick, the system simply looks at whether the second package is disturbed. If not, the system picks it without doing a full recompute of the scene, which typically would save a lot of time. In various embodiments, the time savings is in one or more of sensor read latency, image acquisition time, system compute time, re-segmentation, masking, package pile ordering computations, and finally control and planning. If the second item is not where expected, the system does a full scene re-compute to find a next package to pick.”)
Regarding claim 16, the combination of Sun and Shafer teaches:
The computer-readable storage medium of claim 10.
Sun further discloses wherein the information further comprises an amount of time required to detect the target object (see at least [0108]: “In various embodiments, the system may take a photo and identify two (or more) objects to pick. The system picks and moves the first one; then, instead of doing a full scene re-compute to find a next package to pick, the system simply looks at whether the second package is disturbed. If not, the system picks it without doing a full recompute of the scene, which typically would save a lot of time. In various embodiments, the time savings is in one or more of sensor read latency, image acquisition time, system compute time, re-segmentation, masking, package pile ordering computations, and finally control and planning. If the second item is not where expected, the system does a full scene re-compute to find a next package to pick.”)
Regarding claim 17, the combination of Sun and Shafer teaches:
The computer-readable storage medium of claim 10.
Sun further discloses wherein the throughput represents an average pick rate over a predetermined period of time (see at least [0066-0067]: “While in the example shown in FIG. 2B each station has one robotic arm, in various embodiments two or more robots may be deployed at a station, operated under control of an associated control computer, such as control computer 212 in the example shown in FIG. 2B, in a manner that avoids the robots interfering with each other's operation and movement and which maximizes their collective throughput, including by avoiding and/or managing contention to pick and place the same item. In various embodiments, a scheduler coordinates operation of a plurality of robots, e.g., one or more robots working at each of a plurality of stations, to achieve desired throughput without conflict between robots, such as one robot placing an item in a location the scheduler has assigned to another robot.”)
Claims 3-5 and 12-14 are rejected under 35 U.S.C. 103 as being unpatentable over Sun and Shafer in view of Chavez et al. (US 20240033937 A1), hereinafter Chavez.
Regarding claim 3, the combination of Sun and Shafer teaches:
The method of claim 2.
Sun further discloses wherein the method further comprises and displaying an effect of the proposed change on a user interface (see at least [0076]: “FIG. 4A is a diagram illustrating normal vector computation and display in an embodiment of a robotic singulation system. In various embodiments, item boundaries and normal vectors are determined and a visualization of the item boundaries and normal vectors is generated and displayed by a control computer comprising a robotic singulation system as disclosed herein, such as control computer 212 of FIGS. 2A and 2B.” See further [0079]: “In some embodiments, additional information not shown in FIG. 4A may be displayed. For example, in some embodiments, for each item a best grasp strategy and associated probability of grasp success are determined and displayed adjacent to the item.”)
Sun does not explicitly disclose, but Chavez, in an analogous field of endeavor teaches:
receiving a proposed change to the filtering or sorting rules; determining how the proposed change would affect the selection of the target object (see at least [0087]: “At 404, corresponding scores of successful grasp are determined for each of the grasping strategies. A score of a successful grasp of a feature may be based on a probability that the grasping strategy will result in a successful grasp of the feature. Probabilities are determined for the different combinations of gripping tools and gripping locations. The probability that the grasping strategy will result in a successful grasp of the feature may be based on one more factors, such as contextual information about the environment, historical grasp information for the environment, an angle at which a robotic arm is to grasp the feature (to avoid collision with other objects), a height at which a robotic arm is to grasp the feature (to prevent collision at the top of the gripper), grip width, orientation of surface normal at grasp points, the amount of the feature that is capable of being grasped, etc. Contextual information about the environment includes the existence of other objects near or adjacent to the object, the amount that the other objects near or adjacent to the object hinder an ability of a robotic arm to grasp the feature, whether more objects are continuously being added to a workspace, etc.”)
It would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention, with a reasonable expectation for success, to combine the invention of Sun with the methods taught by Chavez because as stated in [0023] of Chavez: “The robotic system is able to increase the throughput of the robotic system by associating objects with different shapes and using grasping strategies for each of the different shapes. Such a technique is adaptive for any set of objects and does not require the robotic system to be programmed for a particular set of objects prior to picking and placing the objects.”
Regarding claim 4, the combination of Sun, Shafer and Chavez teaches the method of claim 3.
Sun further discloses:
wherein determining how the proposed change would affect the selection of the target object comprises:
modeling a state of the robotic pick-and-place system at a time of the pick using the proposed change to simulate one or more picks starting from the modeled state, determining an updated throughput using the proposed change, and storing the updated throughput in the analytics log (see at least [0103-0104]: “FIGS. 6A through 6C illustrate an example of item flow through a feeder chute in an embodiment of a robotic singulation system. In various embodiments, the flow of items through a chute or other receptacle is modeled. The flow model is used in various embodiments to determine strategies to grasp items from the flow. In some embodiments, modeled and/or observed flow may be used to perform one or more of the following: to determine a grasp strategy and/or plan to grasp an item at a future location to which it is expected to flow; to determine grasp strategies for each of a plurality of items, and to determine and implement a plan to grasp a succession of items, each to be grasped at a corresponding future position determined at least in part based on the flow model; to ensure a robotic arm is in a position to avoid obscuring a view of an item at a future moment at a location in which the items is anticipated based on the model to be located and planned to be picked from; and to wait, e.g., for a computed (based on the model) or prescribed amount of time, to allow for the flow to become more stable (e.g., slower moving, items moving mostly in a uniform direction, minimal change or low rate of change of orientation, etc.). Referring to FIGS. 6A through 6C, in the example shown, the flow model shows a currently mostly stable arrangement of items which the model indicates will continue to be relatively stable/uniform as items 602 and 604 are picked from the flow/pile. In various embodiments, the model information illustrated in FIGS. 6A through 6C would be used, potentially with other information (e.g., available grasp strategies, required/assigned station throughput, etc.), to determine and implement a plan to pick and place items 602 and 604 in succession, each from a location at which the model indicates it is expected to be at the time it is scheduled to be grasped.”)
Regarding claim 5, the combination of Sun, Shafer and Chavez teaches the method of claim 4.
Sun further discloses:
wherein simulating the one or more picks comprises simulating a plurality of picks by a plurality of robotic arms of the robotic pick-and-place system (see at least [0103-0104]: “FIGS. 6A through 6C illustrate an example of item flow through a feeder chute in an embodiment of a robotic singulation system. In various embodiments, the flow of items through a chute or other receptacle is modeled. The flow model is used in various embodiments to determine strategies to grasp items from the flow. In some embodiments, modeled and/or observed flow may be used to perform one or more of the following: to determine a grasp strategy and/or plan to grasp an item at a future location to which it is expected to flow; to determine grasp strategies for each of a plurality of items, and to determine and implement a plan to grasp a succession of items, each to be grasped at a corresponding future position determined at least in part based on the flow model; to ensure a robotic arm is in a position to avoid obscuring a view of an item at a future moment at a location in which the items is anticipated based on the model to be located and planned to be picked from; and to wait, e.g., for a computed (based on the model) or prescribed amount of time, to allow for the flow to become more stable (e.g., slower moving, items moving mostly in a uniform direction, minimal change or low rate of change of orientation, etc.). Referring to FIGS. 6A through 6C, in the example shown, the flow model shows a currently mostly stable arrangement of items which the model indicates will continue to be relatively stable/uniform as items 602 and 604 are picked from the flow/pile. In various embodiments, the model information illustrated in FIGS. 6A through 6C would be used, potentially with other information (e.g., available grasp strategies, required/assigned station throughput, etc.), to determine and implement a plan to pick and place items 602 and 604 in succession, each from a location at which the model indicates it is expected to be at the time it is scheduled to be grasped.”)
Regarding claim 12, the combination of Sun and Shafer teaches:
The computer-readable storage medium of claim 11.
Sun further discloses displaying an effect of the proposed change on a user interface (see at least [0076]: “FIG. 4A is a diagram illustrating normal vector computation and display in an embodiment of a robotic singulation system. In various embodiments, item boundaries and normal vectors are determined and a visualization of the item boundaries and normal vectors is generated and displayed by a control computer comprising a robotic singulation system as disclosed herein, such as control computer 212 of FIGS. 2A and 2B.” See further [0079]: “In some embodiments, additional information not shown in FIG. 4A may be displayed. For example, in some embodiments, for each item a best grasp strategy and associated probability of grasp success are determined and displayed adjacent to the item.”)
Sun does not explicitly disclose, but Chavez, in an analogous field of endeavor teaches:
receive a proposed change to the filtering or sorting rules; determine how the proposed change would affect the selection of the target object (see at least [0087]: “At 404, corresponding scores of successful grasp are determined for each of the grasping strategies. A score of a successful grasp of a feature may be based on a probability that the grasping strategy will result in a successful grasp of the feature. Probabilities are determined for the different combinations of gripping tools and gripping locations. The probability that the grasping strategy will result in a successful grasp of the feature may be based on one more factors, such as contextual information about the environment, historical grasp information for the environment, an angle at which a robotic arm is to grasp the feature (to avoid collision with other objects), a height at which a robotic arm is to grasp the feature (to prevent collision at the top of the gripper), grip width, orientation of surface normal at grasp points, the amount of the feature that is capable of being grasped, etc. Contextual information about the environment includes the existence of other objects near or adjacent to the object, the amount that the other objects near or adjacent to the object hinder an ability of a robotic arm to grasp the feature, whether more objects are continuously being added to a workspace, etc.”)
It would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention, with a reasonable expectation for success, to combine the invention of Sun with the methods taught by Chavez because as stated in [0023] of Chavez: “The robotic system is able to increase the throughput of the robotic system by associating objects with different shapes and using grasping strategies for each of the different shapes. Such a technique is adaptive for any set of objects and does not require the robotic system to be programmed for a particular set of objects prior to picking and placing the objects.”
Regarding claim 13, the combination of Sun, Shafer and Chavez teaches the computer-readable storage medium of claim 12.
Sun further discloses:
wherein determining how the proposed change would affect the selection of the target object comprises:
model a state of the robotic pick-and-place system at a time of the pick; using the proposed change to simulate one or more picks start from the modeled state; determine an updated throughput using the proposed change; and store the updated throughput in the analytics log. (see at least [0103-0104]: “FIGS. 6A through 6C illustrate an example of item flow through a feeder chute in an embodiment of a robotic singulation system. In various embodiments, the flow of items through a chute or other receptacle is modeled. The flow model is used in various embodiments to determine strategies to grasp items from the flow. In some embodiments, modeled and/or observed flow may be used to perform one or more of the following: to determine a grasp strategy and/or plan to grasp an item at a future location to which it is expected to flow; to determine grasp strategies for each of a plurality of items, and to determine and implement a plan to grasp a succession of items, each to be grasped at a corresponding future position determined at least in part based on the flow model; to ensure a robotic arm is in a position to avoid obscuring a view of an item at a future moment at a location in which the items is anticipated based on the model to be located and planned to be picked from; and to wait, e.g., for a computed (based on the model) or prescribed amount of time, to allow for the flow to become more stable (e.g., slower moving, items moving mostly in a uniform direction, minimal change or low rate of change of orientation, etc.). Referring to FIGS. 6A through 6C, in the example shown, the flow model shows a currently mostly stable arrangement of items which the model indicates will continue to be relatively stable/uniform as items 602 and 604 are picked from the flow/pile. In various embodiments, the model information illustrated in FIGS. 6A through 6C would be used, potentially with other information (e.g., available grasp strategies, required/assigned station throughput, etc.), to determine and implement a plan to pick and place items 602 and 604 in succession, each from a location at which the model indicates it is expected to be at the time it is scheduled to be grasped.”)
Regarding claim 14, the combination of Sun, Shafer and Chavez teaches the computer-readable storage medium of claim 13.
Sun further discloses:
wherein simulate the one or more picks comprises simulating a plurality of picks by a plurality of robotic arms of the robotic pick-and-place system (see at least [0103-0104]: “FIGS. 6A through 6C illustrate an example of item flow through a feeder chute in an embodiment of a robotic singulation system. In various embodiments, the flow of items through a chute or other receptacle is modeled. The flow model is used in various embodiments to determine strategies to grasp items from the flow. In some embodiments, modeled and/or observed flow may be used to perform one or more of the following: to determine a grasp strategy and/or plan to grasp an item at a future location to which it is expected to flow; to determine grasp strategies for each of a plurality of items, and to determine and implement a plan to grasp a succession of items, each to be grasped at a corresponding future position determined at least in part based on the flow model; to ensure a robotic arm is in a position to avoid obscuring a view of an item at a future moment at a location in which the items is anticipated based on the model to be located and planned to be picked from; and to wait, e.g., for a computed (based on the model) or prescribed amount of time, to allow for the flow to become more stable (e.g., slower moving, items moving mostly in a uniform direction, minimal change or low rate of change of orientation, etc.). Referring to FIGS. 6A through 6C, in the example shown, the flow model shows a currently mostly stable arrangement of items which the model indicates will continue to be relatively stable/uniform as items 602 and 604 are picked from the flow/pile. In various embodiments, the model information illustrated in FIGS. 6A through 6C would be used, potentially with other information (e.g., available grasp strategies, required/assigned station throughput, etc.), to determine and implement a plan to pick and place items 602 and 604 in succession, each from a location at which the model indicates it is expected to be at the time it is scheduled to be grasped.”)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ELIZABETH NELESKI whose telephone number is (571)272-6064. The examiner can normally be reached 10 - 6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, THOMAS WORDEN can be reached at (571) 272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/E.R.N./Examiner, Art Unit 3658
/THOMAS E WORDEN/Supervisory Patent Examiner, Art Unit 3658
Prosecution Insights