Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Pursuant to communications filed on 02 March 2026, amendments and/or arguments have been submitted and placed in the application file. Claims 7 and 13-15 have been cancelled, claims 19-21 have been added, therefore claims 1-6, 8-12 and 16-21 are currently pending in the instant application.
Response to Arguments
Applicant’s arguments, see page 8, filed 02 March 2026, with respect to the Double Patenting Rejections have been fully considered and are persuasive in light of the submitted and approved Terminal Disclaimers. Accordingly, the double patenting rejections have been withdrawn.
Applicant's arguments filed 02 March 2026 have been fully considered but they are not persuasive. Examiner notes wherein Applicant’s arguments primarily focus on the newly amended limitations of independent claim 1, which include similar limitations previously provided in now cancelled dependent claims 7 and 14-15, as indicated below.
Applicant argues on page 9 of the Remarks,
In its rejection of claim 7, the Examiner does not point to any identical disclosure (or teaching or suggestion) of "select[ing] the grasp location as a location of the object relevant to the work objective". By way of this Response, Applicant has copied the features of original claim 7 into independent claim 1 and canceled claim 7. Additionally, Applicant has added greater emphasis on the grasp selection being relevant to the work objective by amending independent claim 1 to recite "select[ing], by the robot controller and from the library of grasp primitives, a grasp primitive based at least in part on both the work objective and at least one three-dimensional shape in the platonic representation of the object." This amendment also necessitated an amendment to claim 9 for consistency.
Stubbs teaches, at column 15, lines 9-15, generating predictably successful grasps based on the primitive shapes that are representing the item. Stubbs further teaches, at column 21, lines 6-18, selecting contact points for the selected grasp based on one or more factors that are being prioritized by the system, such as probability of success, lower energy use, less manipulation of the item being required before and/or after picking the item up, or speed or efficiency of execution.
Applicant argues on page 9 of the Remarks,
In its rejection of claim 15, the Examiner does not point to any identical disclosure (or teaching or suggestion) of "adjust[ing] control of the end effector based n the further sensor data [to] cause the robot controller to optimize actuation of at least one member of the end effector to increase grasp effectiveness". By way of this Response, Applicant has copied the features of original claim 15 into independent claim 1 and canceled claim 15. And to add further context to such features, Applicant has also copied in the features of original claim 14 and canceled claim 14.
Stubbs teaches, at column 5, lines 43-57, “The sensing information may also be used as feedback to adjust the grasps used by the end of arm tool 126”. Stubbs further teaches, at column 21, lines 19-38, “At 1212, the process 1200 updates the set of contact points and the probabilities of success. This may include updating based at least in part on the feedback. For example, the feedback may indicate that the one or more contact points, while selected with high probabilities of computed, actual, or simulated success, are not ideal for grasping the item. This may be because the robotic manipulator was unable to grasp the item, the item slipped in the grasp, or any other indicator of success. This real-world feedback can be used to reduce (or increase) the probability of success for the one or more contact points” This modified probability of success will be used in future iterations at step 1208 of Fig. 12, described at column 21, lines 6-18, to select contact points that have been determined to have the highest probability of success. This iterative process of determining and using the grasps with the highest probability of success will optimize the grasps over the multiple iterations.
Applicant additionally argues on pages 9-10 of the Remarks, wherein claim 11 is in condition for allowance based on it’s dependence upon claim 1, however, for the same reasons provided herein, with respect to claim 1 above, claim 11 remains rejected as indicated below for at least its dependence upon a rejected base claim.
Accordingly, for the reasons provided above, with respect to the referenced sections of Stubbs, Applicant’s arguments are unpersuasive and remain rejected as indicated below. Examiner notes wherein the below rejection has been augmented to better clarify the rejection(s) in view of the prior art, in light of Applicant’s amendments and/or arguments.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-6, 8-10, 12 and 16-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Stubbs et al (US 9,694,494 B1, hereinafter Stubbs).
Regarding claim 1, Stubbs teaches a method for operating a robot system including a robot body, at least one sensor, and a robot controller including at least one processor and at least one non-transitory processor-readable storage medium storing a library of three-dimensional shapes and a library of grasp primitives, the method comprising:
accessing, by the at least one processor, a work objective of the robot system (Figures 2 & 12; at least as in column 21, lines 6-18, wherein “the one or more contact points associated with the grasp that has the highest probability of success may be selected. In some examples, the selection of the one or more contact points may be constrained by some other factor in place of or in addition to probability of success. For example, if certain contact points would result in lower energy use, would require less manipulation of the item before and/or after picking the item up, or would be executed more quickly or efficiently, this may impact the selection of the one or more contact points”);
capturing, by the at least one sensor, sensor data about an object (Figures 3 & 6; at least as in column 9, lines 31-43, wherein “the item identification module 302 receives sensing information captured by a robotic arm and identifies an item based on the sensing information.” and further as in column 14, line 22-column 15, line 2, specifically at least wherein “the one or more scans may be three-dimensional and may be captured by an imaging device such as a camera or a video camera.”);
accessing, by the robot controller, a platonic representation of the object comprising a set of at least one three-dimensional shape from the library of the three-dimensional shapes, the platonic representation of the object based at least part on the sensor data (Figures 3 & 6; at least as in column 9, lines 44-52, wherein “The feature identification module 304 may be configured to identify features of items identified by the item identification module 302. This can include, for example, analyzing a model of an item and a set of grasping surfaces corresponding to a feature of the item to identify other features of the item.” and further as in column 15, lines 3-17, wherein “The primitive model 604 may be generated based at least in part on the model 602. The primitive model 604 may include one or more primitive shapes representative of the model 602 and the item. Thus, as illustrated, the primitive model 604 may include a combination of a cuboid as a torso, four cylinders as arms and legs, four spheres as hands and feet, a sphere as a head, and two cylinders as ears.”);
selecting, by the robot controller and from the library of grasp primitives, a grasp primitive based at least in part on both the work objective and at least one three-dimensional shape in the platonic representation of the object (Figures 3 & 6; at least as in column 9, line 53-column 10, line 15, “The shape determination module 306 may be configured to determine one or more primitive shapes based on one or more grasping surfaces. For example, after a robotic manipulator has been taught to pick up an item by a feature of the item (e.g., a handle of a coffee mug), the shape determination module 306 may function to generate a primitive shape that corresponds to the handle. The primitive shape may be selected from a set of primitive shapes to closely correspond to the part. The set of primitive shapes may include shapes such as cuboids, cylinders, pyramids, spheres, cones, toroids, and any other suitable primitive shapes.” and further wherein “primitive shapes generated by the shape determination module 306 may be stored together with their associated grasps in the grasp database 236, together with the arm tools in the arm tool database 234 which can pick up an item having the primitive shapes, and/or together with the items in the item database 232 from which the primitive shapes were derived” and column 15, lines 3-17, specifically wherein, “The primitive model 604 may include one or more primitive shapes representative of the model 602 and the item. Thus, as illustrated, the primitive model 604 may include a combination of a cuboid as a torso, four cylinders as arms and legs, four spheres as hands and feet, a sphere as a head, and two cylinders as ears. Using the primitive model 604 and information from a grasp database that includes grasps based on similar primitive shapes, a set of predictably successful grasps may be generated for picking up the item at any of the locations represented by primitive shapes (e.g., ears, head, torso, arms and legs, or hands and feet). These predictably successful grasps may be shared with a robotic manipulator for validation and execution.”); and
controlling, by the robot controller, an end effector of the robot body to apply the grasp primitive to grasp the object at a grasp location (Figures 2, 3 & 6; at least as in column 6, line 55-column 7, line 2 wherein “The grasp execution engine 222 may be configured to instruct one or more robotic manipulators to execute a set of grasps that have been generated and validated” and as in column 10, lines 22-37, wherein “the human operator may use the grasp generation module 310 to teach a robotic arm how to manipulate an end of arm tool to pick up an item. The grasp generation module 310 may then save the characteristics of the taught grasp in the grasp database 236 or in some other location. The grasp generation module 310 may access the taught grasp and other successful grasps in order to generate other potential grasps for grasping the same item or other similar items using the same end of arm tool or other end of arm tools” and further as in at least column 15, lines 3-17, wherein “Using the primitive model 604 and information from a grasp database that includes grasps based on similar primitive shapes, a set of predictably successful grasps may be generated for picking up the item at any of the locations represented by primitive shapes (e.g., ears, head, torso, arms and legs, or hands and feet). These predictably successful grasps may be shared with a robotic manipulator for validation and execution.”);
as the end effector is controlled to apply the grasp primitive, capturing further sensor data indicative of engagement between the end effector and the object being different from expected engagement between the end effector and the at least one three-dimensional shape upon which the selection of the grasp primitive is at least partially based (at least as in column 5, lines 43-57, wherein “The sensors can include sensors configured to detect pressure, force, weight, light, objects, slippage, and any other information that may be used to control and/or monitor the operation of the robotic manipulator 110, including the end of arm tool 126. The sensors may be in communication with the management device 212. In this manner, the management device 212 may control the operation of the robotic manipulator 110 and the end of arm tool 126…The sensing information may also be used as feedback to adjust the grasps used by the end of arm tool 126, and to generate new grasps, to validate grasps, and to determine quality values for grasps, which may be a numerical value based at least in part on one or more objective factors”, and further as in column 21, lines 19-38, wherein “At 1210, the process 1200 receives feedback from a robotic arm that grasps the item at the one or more contact points. This may be the one or more selected contact points. The feedback may come in any suitable form. For example, the feedback may include sensing information detected by any suitable sensor associated with the robotic arm (e.g., slippage, force exerted by suction, fingers, etc., whether the item was even capable of being picked up, and any other suitable information). At 1212, the process 1200 updates the set of contact points and the probabilities of success. This may include updating based at least in part on the feedback. For example, the feedback may indicate that the one or more contact points, while selected with high probabilities of computed, actual, or simulated success, are not ideal for grasping the item. This may be because the robotic manipulator was unable to grasp the item, the item slipped in the grasp, or any other indicator of success. This real-world feedback can be used to reduce (or increase) the probability of success for the one or more contact points”); and
optimizing, based on the further sensor data, actuation of at least one member of the end effector to increase grasp effectiveness (at least as in column 5, lines 52-54, wherein “The sensing information may also be used as feedback to adjust the grasps used by the end of arm tool 126, and to generate new grasps” and further as in column 21, lines 6-18, wherein “At 1208, the process 1200 selects one or more contact points from the set of contact points. For example, the one or more contact points associated with the grasp that has the highest probability of success may be selected” and further as in lines 28-38, wherein “At 1212, the process 1200 updates the set of contact points and the probabilities of success. This may include updating based at least in part on the feedback. For example, the feedback may indicate that the one or more contact points, while selected with high probabilities of computed, actual, or simulated success, are not ideal for grasping the item. This may be because the robotic manipulator was unable to grasp the item, the item slipped in the grasp, or any other indicator of success. This real-world feedback can be used to reduce (or increase) the probability of success for the one or more contact points”).
Regarding claim 2, Stubbs teaches the method further comprising: identifying, by the at least one processor, the object, wherein accessing the platonic representation of the object comprises accessing a three-dimensional model of the object from a database, the three-dimensional model including the platonic representation of the object (Figures 2, 3 & 6; at least as in column 6, line 55-column 7, line 2 wherein “The grasp execution engine 222 may be configured to instruct one or more robotic manipulators to execute a set of grasps that have been generated and validated” and as in column 10, lines 22-37, wherein “the human operator may use the grasp generation module 310 to teach a robotic arm how to manipulate an end of arm tool to pick up an item. The grasp generation module 310 may then save the characteristics of the taught grasp in the grasp database 236 or in some other location. The grasp generation module 310 may access the taught grasp and other successful grasps in order to generate other potential grasps for grasping the same item or other similar items using the same end of arm tool or other end of arm tools” and further as in at least column 15, lines 3-17, wherein “Using the primitive model 604 and information from a grasp database that includes grasps based on similar primitive shapes, a set of predictably successful grasps may be generated for picking up the item at any of the locations represented by primitive shapes (e.g., ears, head, torso, arms and legs, or hands and feet). These predictably successful grasps may be shared with a robotic manipulator for validation and execution.”).
Regarding claim 3, Stubbs further teaches wherein accessing the platonic representation of the object comprises generating the at least one platonic representation of the object, by approximating the object with the set of at least one three-dimensional shape (Figures 3 & 6; at least as in column 9, line 53-column 10, line 15, wherein “The shape determination module 306 may be configured to determine one or more primitive shapes based on one or more grasping surfaces” and further as in at least column 14, line 22-column 15, line 17, wherein “The primitive model 604 may be generated based at least in part on the model 602. The primitive model 604 may include one or more primitive shapes representative of the model 602 and the item. Thus, as illustrated, the primitive model 604 may include a combination of a cuboid as a torso, four cylinders as arms and legs, four spheres as hands and feet, a sphere as a head, and two cylinders as ears. Using the primitive model 604 and information from a grasp database that includes grasps based on similar primitive shapes, a set of predictably successful grasps may be generated for picking up the item at any of the locations represented by primitive shapes (e.g., ears, head, torso, arms and legs, or hands and feet).”).
Regarding claim 4, Stubbs further teaches wherein generating the at least one platonic representation of the object comprises:
identifying at least one portion of the object suitable for representation by respective three-dimensional shapes (Figures 3 & 5; at least as in column 9, lines 44-52, wherein “The feature identification module 304 may be configured to identify features of items identified by the item identification module 302. This can include, for example, analyzing a model of an item and a set of grasping surfaces corresponding to a feature of the item to identify other features of the item. In some examples, the feature identification module 304 may be capable of processing a surface model of an item to identify features that can be used for grasping the item.” and as in column 12, line 62-column 13, line 7, wherein “grasp information identifying characteristics of how the feature 514 was grasped may be saved. This information may include a set of contact points on the feature 514 where the end of arm tool contacted the feature 514. These contact points may correspond to one or more grasping surfaces. For example, if the end of arm tool grasped the feature 514 with a two-finger grabber there may be two grasping surfaces disposed opposite each other. At the state 504, the grasping surfaces may correspond to the shape of the feature 514. At the state 506, the grasping surfaces, which correspond to the shape of the feature 514, may be bounded by a primitive shape 516.”); and
for each portion of the at least one portion: accessing a geometric three-dimensional shape model which is similar in shape to the portion (Figures 3 & 5; at least as in column 9, lines 44-52, wherein “The feature identification module 304 may be configured to identify features of items identified by the item identification module 302. This can include, for example, analyzing a model of an item and a set of grasping surfaces corresponding to a feature of the item to identify other features of the item. In some examples, the feature identification module 304 may be capable of processing a surface model of an item to identify features that can be used for grasping the item.” and as in column 12, line 62-column 13, line 7, wherein “grasp information identifying characteristics of how the feature 514 was grasped may be saved. This information may include a set of contact points on the feature 514 where the end of arm tool contacted the feature 514. These contact points may correspond to one or more grasping surfaces. For example, if the end of arm tool grasped the feature 514 with a two-finger grabber there may be two grasping surfaces disposed opposite each other. At the state 504, the grasping surfaces may correspond to the shape of the feature 514. At the state 506, the grasping surfaces, which correspond to the shape of the feature 514, may be bounded by a primitive shape 516.”); and
transforming the accessed geometric three-dimensional shape model to fit the portion (Figures 3 & 5; at least as in column 9, lines 44-52, wherein “The feature identification module 304 may be configured to identify features of items identified by the item identification module 302. This can include, for example, analyzing a model of an item and a set of grasping surfaces corresponding to a feature of the item to identify other features of the item. In some examples, the feature identification module 304 may be capable of processing a surface model of an item to identify features that can be used for grasping the item.” and as in column 12, line 62-column 13, line 7, wherein “The primitive shape may be selected from a set of primitive shapes and in a manner that attempts to most closely approximate the grasping surfaces, which, in this example, correspond to the shape of the feature 514. In some examples, parts of the feature 514, which do not correspond to grasping surfaces may be ignored and/or not included in the primitive shape 516. For example, tag 518 is illustrated in the state 506 as being located outside of the primitive shape 516 and is therefore ignored. The primitive shape 516 may function as a bounding shape that includes the grasping surfaces.”).
Regarding claim 5, Stubbs further teaches wherein, for each portion of the at least one portion, transforming the accessed three-dimensional geometric shape model to fit the portion comprises: transforming a size of the geometric three-dimensional shape model in at least one dimension to fit the size of the geometric three-dimensional shape model to the portion (Figure 5; at least as in column 13, lines 27-54, specifically wherein “the size of the primitive shape 516 is expanded and contracted to create different versions 516(1)-516(N) of the primitive shape 516” and further as in column 14, line 22-column 15, line 17, wherein “The primitive model 604 may include one or more primitive shapes representative of the model 602 and the item. Thus, as illustrated, the primitive model 604 may include a combination of a cuboid as a torso, four cylinders as arms and legs, four spheres as hands and feet, a sphere as a head, and two cylinders as ears. Using the primitive model 604 and information from a grasp database that includes grasps based on similar primitive shapes, a set of predictably successful grasps may be generated for picking up the item at any of the locations represented by primitive shapes (e.g., ears, head, torso, arms and legs, or hands and feet)”); transforming a position of the geometric three-dimensional shape model to align with a position of the portion (Figure 5; at least as in column 13, lines 27-54, specifically wherein “the size of the primitive shape 516 is expanded and contracted to create different versions 516(1)-516(N) of the primitive shape 516” and further as in column 14, line 22-column 15, line 17, wherein “The primitive model 604 may include one or more primitive shapes representative of the model 602 and the item. Thus, as illustrated, the primitive model 604 may include a combination of a cuboid as a torso, four cylinders as arms and legs, four spheres as hands and feet, a sphere as a head, and two cylinders as ears. Using the primitive model 604 and information from a grasp database that includes grasps based on similar primitive shapes, a set of predictably successful grasps may be generated for picking up the item at any of the locations represented by primitive shapes (e.g., ears, head, torso, arms and legs, or hands and feet)”); or rotating the geometric three-dimensional shape model to fit the geometric model to an orientation of the portion (Figure 5; at least as in column 13, lines 27-54, specifically wherein “the size of the primitive shape 516 is expanded and contracted to create different versions 516(1)-516(N) of the primitive shape 516” and further as in column 14, line 22-column 15, line 17, wherein “The orientation of the first item may be varied with respect to the imaging device in each three-dimensional scan in the set of three-dimensional scans.”).
Regarding claim 6, Stubbs teaches the method further comprising selecting, by the robot controller, the grasp location of the object (Figures 2, 3 & 6; at least as in column 6, line 55-column 7, line 2 wherein “The grasp execution engine 222 may be configured to instruct one or more robotic manipulators to execute a set of grasps that have been generated and validated” and as in column 10, lines 22-37, wherein “the human operator may use the grasp generation module 310 to teach a robotic arm how to manipulate an end of arm tool to pick up an item. The grasp generation module 310 may then save the characteristics of the taught grasp in the grasp database 236 or in some other location. The grasp generation module 310 may access the taught grasp and other successful grasps in order to generate other potential grasps for grasping the same item or other similar items using the same end of arm tool or other end of arm tools” and further as in at least column 15, lines 3-17, wherein “Using the primitive model 604 and information from a grasp database that includes grasps based on similar primitive shapes, a set of predictably successful grasps may be generated for picking up the item at any of the locations represented by primitive shapes (e.g., ears, head, torso, arms and legs, or hands and feet). These predictably successful grasps may be shared with a robotic manipulator for validation and execution.”).
Regarding claim 8, Stubbs teaches the method further comprising:
identifying, by the robot controller based on the sensor data, at least one graspable feature of the object (Figures 3 & 5; at least as in column 9, lines 44-52, wherein “The feature identification module 304 may be configured to identify features of items identified by the item identification module 302. This can include, for example, analyzing a model of an item and a set of grasping surfaces corresponding to a feature of the item to identify other features of the item. In some examples, the feature identification module 304 may be capable of processing a surface model of an item to identify features that can be used for grasping the item.” and as in column 12, line 62-column 13, line 7, wherein “grasp information identifying characteristics of how the feature 514 was grasped may be saved. This information may include a set of contact points on the feature 514 where the end of arm tool contacted the feature 514. These contact points may correspond to one or more grasping surfaces. For example, if the end of arm tool grasped the feature 514 with a two-finger grabber there may be two grasping surfaces disposed opposite each other. At the state 504, the grasping surfaces may correspond to the shape of the feature 514. At the state 506, the grasping surfaces, which correspond to the shape of the feature 514, may be bounded by a primitive shape 516.”); and
selecting, by the robot controller, one or more of the at least one graspable feature as the grasp location of the object (Figures 3 & 5; at least as in column 9, lines 44-52, wherein “The feature identification module 304 may be configured to identify features of items identified by the item identification module 302. This can include, for example, analyzing a model of an item and a set of grasping surfaces corresponding to a feature of the item to identify other features of the item. In some examples, the feature identification module 304 may be capable of processing a surface model of an item to identify features that can be used for grasping the item.” and as in column 12, line 62-column 13, line 7, wherein “The primitive shape may be selected from a set of primitive shapes and in a manner that attempts to most closely approximate the grasping surfaces, which, in this example, correspond to the shape of the feature 514. In some examples, parts of the feature 514, which do not correspond to grasping surfaces may be ignored and/or not included in the primitive shape 516. For example, tag 518 is illustrated in the state 506 as being located outside of the primitive shape 516 and is therefore ignored. The primitive shape 516 may function as a bounding shape that includes the grasping surfaces.”).
Regarding claim 9, Stubbs teaches the method further comprising:
evaluating, by the robot controller, grasp-effectiveness for a plurality of grasp primitive-location pairs, each grasp primitive-location pair including a respective three-dimensional shape in the platonic representation of the object and a respective grasp primitive from the library of grasp primitives (Figures 2, 3 & 12; at least as in column 9, lines 5-15, specifically at least wherein “the success and failure of the grasps from the grasp database 236 under actual conditions may be used to update the grasps from the grasp database”, and further as in column 10, lines 38-65, and further as in column 14, line 22-column 15, line 17, wherein “The primitive model 604 may include one or more primitive shapes representative of the model 602 and the item. Thus, as illustrated, the primitive model 604 may include a combination of a cuboid as a torso, four cylinders as arms and legs, four spheres as hands and feet, a sphere as a head, and two cylinders as ears. Using the primitive model 604 and information from a grasp database that includes grasps based on similar primitive shapes, a set of predictably successful grasps may be generated for picking up the item at any of the locations represented by primitive shapes (e.g., ears, head, torso, arms and legs, or hands and feet)”); and
selecting, by the robot controller, the grasp location, suited to achieve the work objective, as a location of the three-dimensional shape in a grasp primitive-location pair having a grasp-effectiveness which exceeds a threshold, wherein selecting the grasp primitive comprises selecting the grasp primitive as a grasp primitive in the primitive-location pair having the highest grasp-effectiveness (Figures 2, 3 & 12; at least as in as in column 10, lines 38-65, and further as in column 14, line 22-column 15, line 17, wherein “The primitive model 604 may include one or more primitive shapes representative of the model 602 and the item. Thus, as illustrated, the primitive model 604 may include a combination of a cuboid as a torso, four cylinders as arms and legs, four spheres as hands and feet, a sphere as a head, and two cylinders as ears. Using the primitive model 604 and information from a grasp database that includes grasps based on similar primitive shapes, a set of predictably successful grasps may be generated for picking up the item at any of the locations represented by primitive shapes (e.g., ears, head, torso, arms and legs, or hands and feet). These predictably successful grasps may be shared with a robotic manipulator for validation and execution.” and further column 20, line 35-column 21, line 42, specifically as shown in Figure 12).
Regarding claim 10, Stubbs further teaches wherein evaluating grasp-effectiveness for a plurality of grasp primitive-location pairs comprises, for each grasp primitive-location pair:
simulating grasping of the respective three-dimensional shape in the platonic representation of the object, by applying the respective grasp primitive (Figures 1-3 & 12; at least as in column 8, lines 39-62, column 10, line 38-column 11, line 26 and column 20, line 35-column 21, line 42, specifically as shown in Figure 12); and
generating a grasp-effectiveness score indicative of effectiveness of simulated grasping (Figures 1-3 & 12; at least as in column 8, lines 39-62, column 10, line 38-column 11, line 26 and column 20, line 35-column 21, line 42, specifically as shown in Figure 12).
Regarding claim 12, Stubbs further teaches wherein capturing sensor data about the object comprises capturing sensor data by at least one sensor selected from a group of sensors consisting of: an image sensor operable to capture image data; an audio sensor operable to capture audio data; a tactile sensor operable to capture tactile data; a haptic sensor which captures haptic data; an actuator sensor which captures actuator data indicating a state of a corresponding actuator; an inertial sensor which captures inertial data; a proprioceptive sensor which captures proprioceptive data indicating a position, movement, or force applied for a corresponding actuatable member of the robot body; and a position encoder which captures position data about at least one joint or appendage of the robot body (Figures 1-3 & 6; at least as in column 5, lines 36-57, wherein “The robotic manipulator 110 may include any suitable type and number of sensors disposed throughout the robotic manipulator 110 (e.g., sensors in the base, in the arm, in joints in the arm, in the end of arm tool 126, or in any other suitable location). The sensors can include sensors configured to detect pressure, force, weight, light, objects, slippage, and any other information that may be used to control and/or monitor the operation of the robotic manipulator 110, including the end of arm tool 126. The sensors may be in communication with the management device 212. In this manner, the management device 212 may control the operation of the robotic manipulator 110 and the end of arm tool 126 based at least in part on sensing information received from the sensors” and as in column 9, lines 31-43, wherein “the item identification module 302 receives sensing information captured by a robotic arm and identifies an item based on the sensing information.” and further as in column 14, line 22-column 15, line 2, specifically at least wherein “the one or more scans may be three-dimensional and may be captured by an imaging device such as a camera or a video camera.”).
Regarding claim 16, Stubbs further teaches wherein: the robot body carries the at least one sensor and the robot controller; and capturing the sensor data, accessing the platonic representation of the object, selecting a grasp primitive, and controlling the end effector are performed at the robot body (Figures 2-3; at least as in column 5, lines 8-52, wherein “the manipulator system 206 may include the management device 212 in electrical and/or network communication with the robotic manipulator 110. The robotic manipulator 110 may include any suitable type and number of sensors disposed throughout the robotic manipulator 110” and further as in at least column 9, lines 31-64, wherein “the item identification module 302 receives sensing information captured by a robotic arm and identifies an item based on the sensing information.”).
Regarding claim 17, Stubbs further teaches wherein:
the robot system further includes a remote device remote from the robot body, and a communication interface which communicatively couples the remote device and the robot body (Figures 1-3; at least as in column 5, line 8-column 6, line 59, specifically as shown in at least Figure 2);
the robot body carries the at least one sensor (Figures 2-3; at least as in column 5, lines 8-52, wherein “the manipulator system 206 may include the management device 212 in electrical and/or network communication with the robotic manipulator 110. The robotic manipulator 110 may include any suitable type and number of sensors disposed throughout the robotic manipulator 110” and further as in at least column 9, lines 31-64, wherein “the item identification module 302 receives sensing information captured by a robotic arm and identifies an item based on the sensing information.”);
the remote device includes the robot controller; capturing the sensor data is performed at the robot body (Figures 1-3; at least as in column 5, line 8-column 6, line 59, specifically as shown in at least Figure 2);
the method further comprises transmitting, by a communication interface, the sensor data from the robot body to the remote device (Figures 1-3; at least as in column 5, line 8-column 6, line 59, specifically as shown in at least Figure 2);
accessing the platonic representation of the object, selecting a grasp primitive, and controlling the end effector are performed at the remote device (Figures 1-3, 6 & 12; at least as in column 10, line 38-column 11, line 26, column 14, line 22-column 15, line 17 and column 20, line 35-column 21, line 42, specifically as shown in Figure 12); and
controlling the end effector comprises the robot controller preparing and sending control instructions to the robot body via the communication interface (Figures 1-3, 6 & 12; at least as in column 10, line 38-column 11, line 26, column 14, line 22-column 15, line 17 and column 20, line 35-column 21, line 42, specifically as shown in Figure 12).
Regarding claim 18, Stubbs further teaches wherein:
the robot system further includes a remote device remote from the robot body, and a communication interface which communicatively couples the remote device and the robot body; the robot body carries the at least one sensor, a first processor of the at least one processor, and a first non-transitory processor-readable storage medium of the at least one non-transitory processor-readable storage medium (Figures 1-3; at least as in column 5, line 8-column 6, line 59, specifically as shown in at least Figure 2);
the remote device includes a second processor of the at least one processor, and a second non-transitory processor-readable storage medium of the at least one non-transitory processor-readable storage medium (Figures 1-3; at least as in column 5, line 8-column 6, line 59, specifically as shown in at least Figure 2);
capturing the sensor data and controlling the end effector are performed at the robot body (Figures 2-3; at least as in column 5, lines 8-52, wherein “the manipulator system 206 may include the management device 212 in electrical and/or network communication with the robotic manipulator 110. The robotic manipulator 110 may include any suitable type and number of sensors disposed throughout the robotic manipulator 110” and further as in at least column 9, lines 31-64, wherein “the item identification module 302 receives sensing information captured by a robotic arm and identifies an item based on the sensing information.”);
accessing the platonic representation of the object and selecting the grasp primitive are performed at the remote device (Figures 1-3, 6 & 12; at least as in column 10, line 38-column 11, line 26, column 14, line 22-column 15, line 17 and column 20, line 35-column 21, line 42, specifically as shown in Figure 12); and
the method further comprises transmitting, by a communication interface, the sensor data from the robot body to the remote device (Figures 1-3, 6 & 12; at least as in column 10, line 38-column 11, line 26, column 14, line 22-column 15, line 17 and column 20, line 35-column 21, line 42, specifically as shown in Figure 12); and
the method further comprises transmitting, by the communication interface, data indicating the grasp primitive and the platonic representation of the object to the robot body from the remote device (Figures 1-3, 6 & 12; at least as in column 10, line 38-column 11, line 26, column 14, line 22-column 15, line 17 and column 20, line 35-column 21, line 42, specifically as shown in Figure 12).
Regarding claim 19, Stubbs further teaches wherein:
the at least one sensor includes a tactile sensor carried by the end effector (at least as in column 5, lines 43-57, wherein “The sensors can include sensors configured to detect pressure, force, weight, light, objects, slippage, and any other information that may be used to control and/or monitor the operation of the robotic manipulator 110, including the end of arm tool 126);
capturing further sensor data indicative of engagement between the end effector and the object being different from expected engagement between the end effector and the at least one three-dimensional shape upon which the selection of the grasp primitive is at least partially based includes capturing, by the tactile sensor, tactile data indicative of engagement between the end effector and the object being different from expected engagement between the end effector and the at least one three-dimensional shape upon which the selection of the grasp primitive is at least partially based (at least as in column 5, lines 43-57, wherein “The sensors can include sensors configured to detect pressure, force, weight, light, objects, slippage, and any other information that may be used to control and/or monitor the operation of the robotic manipulator 110, including the end of arm tool 126. The sensors may be in communication with the management device 212. In this manner, the management device 212 may control the operation of the robotic manipulator 110 and the end of arm tool 126…The sensing information may also be used as feedback to adjust the grasps used by the end of arm tool 126, and to generate new grasps, to validate grasps, and to determine quality values for grasps, which may be a numerical value based at least in part on one or more objective factors”, and further as in column 21, lines 19-38, wherein “At 1210, the process 1200 receives feedback from a robotic arm that grasps the item at the one or more contact points. This may be the one or more selected contact points. The feedback may come in any suitable form. For example, the feedback may include sensing information detected by any suitable sensor associated with the robotic arm (e.g., slippage, force exerted by suction, fingers, etc., whether the item was even capable of being picked up, and any other suitable information). At 1212, the process 1200 updates the set of contact points and the probabilities of success. This may include updating based at least in part on the feedback. For example, the feedback may indicate that the one or more contact points, while selected with high probabilities of computed, actual, or simulated success, are not ideal for grasping the item. This may be because the robotic manipulator was unable to grasp the item, the item slipped in the grasp, or any other indicator of success. This real-world feedback can be used to reduce (or increase) the probability of success for the one or more contact points”); and
optimizing, based on the further sensor data, actuation of at least one member of the end effector to increase grasp effectiveness includes optimizing, based on the tactile data, actuation of at least one member of the end effector to increase grasp effectiveness on the object (at least as in column 5, lines 52-54, wherein “The sensing information may also be used as feedback to adjust the grasps used by the end of arm tool 126, and to generate new grasps” and further as in column 21, lines 6-18, wherein “At 1208, the process 1200 selects one or more contact points from the set of contact points. For example, the one or more contact points associated with the grasp that has the highest probability of success may be selected” and further as in lines 28-38, wherein “At 1212, the process 1200 updates the set of contact points and the probabilities of success. This may include updating based at least in part on the feedback. For example, the feedback may indicate that the one or more contact points, while selected with high probabilities of computed, actual, or simulated success, are not ideal for grasping the item. This may be because the robotic manipulator was unable to grasp the item, the item slipped in the grasp, or any other indicator of success. This real-world feedback can be used to reduce (or increase) the probability of success for the one or more contact points”).
Regarding claim 20, Stubbs further teaches wherein:
the at least one sensor further includes an image sensor (Figure 6; at least as in column 14, lines 33-42, wherein “the model 602 may be based on one or more three-dimensional scans of the item. In some examples, the one or more scans may be used to identify features of the item that are flexible and rigid. For example, the one or more scans may be captured from different perspectives and compared to identify features that moved between scans and those that did not move. In some examples, the one or more scans may be three-dimensional and may be captured by an imaging device such as a camera or video camera”);
capturing, by at least one sensor, sensor data about an object includes capturing, by the image sensor, image data about the object (Figure 6; at least as in column 14, lines 33-42, wherein “the model 602 may be based on one or more three-dimensional scans of the item. In some examples, the one or more scans may be used to identify features of the item that are flexible and rigid. For example, the one or more scans may be captured from different perspectives and compared to identify features that moved between scans and those that did not move. In some examples, the one or more scans may be three-dimensional and may be captured by an imaging device such as a camera or video camera”); and
accessing, by the robot controller, the platonic representation of the object based at least in part on the sensor data includes accessing, by the robot controller, the platonic representation of the object based on the image data (Figure 6; at least as in column 14, lines 32-35, wherein “the grasp management engine 220 may function to generate the model 602 and/or the primitive model 604. For example, the model 602 may be based on one or more three-dimensional scans of the item” and further as in column 15, lines 3-9, wherein “The primitive model 604 may be generated based at least in part on the model 602. The primitive model 604 may include one or more primitive shapes representative of the model 602 and the item. Thus, as illustrated, the primitive model 604 may include a combination of a cuboid as a torso, four cylinders as arms and legs, four spheres as hands and feet, a sphere as a head, and two cylinders as ears”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Stubbs et al (US 9,694,494 B1, hereinafter Stubbs) in view of Humayun et al (US 11,559,885 B2, hereinafter Humayun).
The teachings of Stubbs have been discussed above.
Regarding claim 11, as noted above, Stubbs teaches the method further comprising: selecting, by the robot controller, the grasp location as a grasp area of the object, wherein selecting the grasp primitive comprises selecting the grasp primitive based on the at least one three-dimensional shape in the platonic representation of the object which at least approximately corresponds to the grasp location (Figures 2, 3 & 12; at least as in as in column 10, lines 38-65, and further as in column 14, line 22-column 15, line 17, wherein “The primitive model 604 may include one or more primitive shapes representative of the model 602 and the item. Thus, as illustrated, the primitive model 604 may include a combination of a cuboid as a torso, four cylinders as arms and legs, four spheres as hands and feet, a sphere as a head, and two cylinders as ears. Using the primitive model 604 and information from a grasp database that includes grasps based on similar primitive shapes, a set of predictably successful grasps may be generated for picking up the item at any of the locations represented by primitive shapes (e.g., ears, head, torso, arms and legs, or hands and feet). These predictably successful grasps may be shared with a robotic manipulator for validation and execution.” and further column 20, line 35-column 21, line 42, specifically as shown in Figure 12). That said, Stubbs is silent specifically regarding wherein the method further includes, “accessing, by the robot controller, a grasp heatmap for the object, the grasp heatmap indicative of grasp areas of the object”.
Humayun, in the same field of endeavor, teaches a robotic system and corresponding method for grasping one or more objects, wherein the system method include at least, “capturing an image of a scene during runtime (e.g., same field of view as training image, difference field of view training image); determining a graspability map (e.g., mask, heatmap, etc.) for the image using the trained graspability network; and selecting a grasp based on the graspability map (e.g., where a grasp can be an image feature)” (Figures 6-7; at least as in column 11, lines 13-31, column 14, lines 23-44 and column 15, lines 15-40). Therefore, it would have been obvious to one of ordinary skill in the art at the effective filing date of the instant invention, to modify the teachings of Stubbs, to include Humayun’s teaching of utilizing a grasp heatmap in determining/selecting a grasp primitive, since Humayun teaches wherein employing such grasping strategies provides a system/method for selecting more accurate and successful grasps by said robot, thereby providing a more dynamic, accurate and efficient robotic grasping system/method.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Stubbs et al (US 9,694,494 B1, hereinafter Stubbs) in view of Matsuda (US 2024/0100695 A1).
The teachings of Stubbs have been discussed above.
Regarding claim 21, Stubbs is silent specifically regarding wherein optimizing, based on the further sensor data, actuation of at least one member of the end effector to increase grasp effectiveness includes optimizing, based on the further sensor data, actuation of at least one member of the end effector to grasp the object more tightly.
Matsuda, in the same field of endeavor, teaches controlling a grasping force of a robotic hand in accordance with the amount of displacement measured by a slip detection unit (at least as in paragraphs 0170-0174 and 0180-0184). Matsuda further teaches wherein increasing the grasping force is an appropriate response to an object slipping. This would allow such robotic grasping systems to increase a grasping force when a grasped object is slipping so as to hold the object more securely and prevent further slipping (at least as in paragraphs 0008 and 0077-0081). Therefore, it would have been obvious to one of ordinary skill in the art at the effective filing date of the instant invention to include, with the method as taught by Stubbs, (re claim 21) optimizing, based on the further sensor data, actuation of at least one member of the end effector to increase grasp effectiveness includes optimizing, based on the further sensor data, actuation of at least one member of the end effector to grasp the object more tightly, with a reasonable expectation of success, since Matsuda teaches controlling a grasping force of a robotic hand in accordance with the amount of displacement measured by a slip detection unit, and increasing grasping force is an appropriate response to an object slipping. This would allow such robotic grasping systems/methods to increase a grasping force when a grasped object is slipping so as to hold the object more securely and prevent further slipping.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See attached PTO-892 – Notice of References Cited form. Examiner additionally notes the following prior art references, in the same field of endeavor as the instant invention, and also reads on several of the currently provided claim limitations above;
US 2022/0402128 A1, issued to Lian et al, which is directed towards a robotic grasping system that utilizes generated models of imaged objects to determine a grasp strategy for said robotic grasping system.
US 2011/0010009 A1, issued to Saito, which is directed towards teaching a robot a grasping action of an object by a robot, by employing a primitive shape model to one or more parts of said object and determining a corresponding grasping configuration for said robot.
US 2022/0379484 A1, issued to Tremblay et al, which is directed towards utilizing heatmaps for a grasp determination of an object, and further wherein a robot may use the grasp determination to manipulate the object.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN L SAMPLE whose telephone number is (571)270-5925. The examiner can normally be reached Monday-Friday 7:00am-4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Mott can be reached at (571)270-5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONATHAN L SAMPLE/Primary Examiner, Art Unit 3657