DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/02/2026 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 36-59 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 36-37, 39-41, 47, 49, 51-53, and 55-59 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Keraly et al. (US 20230092690 A, hereinafter Keraly).
Regarding claim 36, Keraly teaches:
A processing station for use in an automated assembly line comprising a plurality of modular processing stations (at least as in paragraph 0029, wherein “robotic work cell 100 may be easily modified to perform any number of alternative designated processes on a wide range of object types”), the processing station comprising:
a frame having a fixed dimension (at least as in paragraph 0053, wherein “Robotic work cell 100C includes a generalized object separating mechanism 110C, a pick-and-place robot 130C, and an object processing robot 150C that are fixedly connected to a rigid base plate 101C (e.g., a plate metal sheet)”);
a loading area coupled to the frame presenting a plurality of workpieces, the loading area comprising a workpiece re-orienting mechanism capable of adjusting an orientation of workpieces in the loading area (at least as in paragraph 0032, “object separating mechanism 110 includes a horizontal surface 111 and a dynamic manipulator 112 that cooperatively rearrange bottles received from 3D cluster 90-3D into 2D arrangement 90-2D”; at least as in paragraph at least as in paragraph 0053 & 0075, wherein object separating mechanism is fixedly connected to a rigid base plate);
a pick-and-place robot coupled to the frame and controllable to move a workpiece from the plurality of workpieces presented in the loading area to a moveable receiving area (at least as in paragraph 0034, “Pick-and-place robot mechanism 130 is configured to move target bottle 91T(t1) from horizontal surface 111 to designated hand-off location HOL in a manner that facilitates a hand-off operation (described below) using location data X1,Y1 and rotational orientation data θ1”; at least as in paragraph at least as in paragraph 0053 & 0075, wherein pick-and-place robot is fixedly connected to a rigid base plate);
an imaging system coupled to the frame capturing images of the plurality of workpieces (at least as in paragraph 0033, “Vision system 120 is a vision-based sensing system configured to sequentially identify (select) individual target bottles/objects from the bottles/objects 91(t1) forming 2D arrangement 90-2D(t1) and to generate data that facilitates the removal of each selected target bottle/object from horizontal surface 111 and subsequent delivery to a designated hand-off location HOL by pick-and-place robot 130”; at least as in paragraph 0075, wherein a protective “fish bowl” frame is formed above base plate to support cameras utilized by the associated vision systems); and
a station controller (at least as in paragraph 0056, “Shared power/control resources (control unit) 180C, which is depicted as a computer for brevity, is configured using known techniques to manage power distribution and to coordinate the various operations that achieve the desired bulk bottle (or other object) processing described herein”) configured to:
process one or more of the captured images to identify one or more pickable workpieces in the loading area (at least as in paragraph 0039, “Referring to block 220, a target object is then selected from the objects forming the 2D arrangement, and then location and rotational orientation data operably describing the location and rotational orientation of the target object are generated”; at least as in paragraph 0049, “vision system 120A then identifies individual objects (e.g., by way of comparing the current image data CID with stored image data SID that operably describes the objects) and selects (identifies and designates) one of the stationary objects as the next target object and generates the associated location and rotational orientation data”); and
operate the pick-and-place robot to transfer the identified one or more pickable workpieces to the moveable receiving area (at least as in paragraph 0049, “As indicated by block 220-2, vision system 120A then transmits the location and rotational orientation data to the pick-and-place robot mechanism as described additional detail below to facilitate the pick-up/removal and transfer of the selected target object to the hand-off location”; at least as in paragraph 0040, “Referring to block 230, the location data is then used to remove (pick-up) the selected target object from the horizontal surface”).
Regarding claim 37, Keraly further teaches:
The processing station of claim 36, further comprising:
a section of a conveyor, the conveyor connected between the processing station and at least one adjacent modular processing station of the plurality of modular processing stations of the automated assembly line to move one or more movers between adjacent stations, each of the one or more movers comprising one or more of a bed, a pallet, or a shuttle (at least as in paragraph 0075, “two exit ports 102E-1 and 102E-2 are cut or otherwise defined in base plate 101E to facilitate removal of fully processed objects released from robotic end-tools 160D-1 and 160D-2, respectively (i.e., such that the released fully processed objects fall through exit ports 102E-1 and 102E-2 and into one or more output bins)”; at least as in paragraph 0030, ““91(t0)” indicates bottles 91 at an initial time t0 when disposed in 3D cluster 90-3D”; at least as in paragraph 0047, “Gating mechanism 318A is configured to selectively pass portions (groups) of objects from a bulk supply (e.g., from hopper 101, shown in FIG. 1) by way of an intervening feed chute 317A onto horizontal surface 111A”; at least as in paragraph 0032, 0045, & Fig. 1, wherein the random 3D cluster of bottles received at initial time t0 is from an output bin 105).
Regarding claim 39, Keraly further teaches:
The processing station of claim 36, wherein the loading area comprises one or more of:
a dial-based feeder;
a bowl feeder;
a vibratory feeder;
a linear feeder;
a conveyor feeder;
a hopper; or a shaker tray (at least as in paragraph 0047, “Object separation mechanism 110A generally includes a conveyor belt mechanism 310A, a stationary frame 316A and a gating mechanism 318A”).
Regarding claim 40, Keraly further teaches:
The processing station of claim 36, wherein the fixed dimension of the frame is common to at least one of the plurality of modular processing stations (at least as in paragraph 0075, “two work cell units 100D-1 and 100D-2, each configured in accordance with robotic work cell 100D described above, are operably connected to an integral sheet-metal base plate 101E in an inverted mirror arrangement and are independently controlled by at least one control unit 180E that is at least partially disposed below base plate 101E”).
Regarding claim 41, Keraly further teaches:
The processing station of claim 36, further comprising at least one of:
a second loading area;
a second receiving area;
a second imaging system; and
a second pick-and-place robot (at least as in paragraph 0075, “work cell unit 100D-2 includes a second object separating mechanism 110B-2, a second pick-and-place robot mechanism 130C-2, a second 4-axis robot mechanism 150C-2 holding a second carousel-type robotic end-tool 160D-2, a second processing device 170D-2 and a second positioning structure 190D-2 that are also connected to base plate 101E”).
Regarding claim 47, Keraly further teaches:
The processing station of Claim 36, further comprising a second loading area, and
wherein the imaging system is operable to capture an image of a first set of workpieces loaded onto the loading area and a second image of a second set of workpieces loaded onto the second loading area, and wherein the pick-and-place robot is operable to retrieve one or more pickable workpieces from the loading area while the station controller applies a machine-learning model to the second image to identify one or more pickable workpieces from the second loading area (at least as in paragraph 0048, “FIG. 3B depicts an object separation mechanism 110B according to a presently preferred practical embodiment including two separating units 110A-1 and 110A-2 disposed in a parallel side-by-side arrangement”; at least as in paragraph 0039, 0052, & Figs. 5A-5B, wherein the work cell captures images of the still bottles on the first conveyor and transfers the target bottles, while the vision system determines through image processing techniques whether a target bottle is acquirable or sufficiently separated on the second conveyor unit).
Regarding claim 49, Keraly further teaches:
The processing station of Claim 36, wherein the workpiece re-orienting mechanism is capable of one or more of vibrating, spinning, or blowing air to re-orient workpieces in the loading area (at least as in paragraph 0049, “conveyor belt drive motor 312A is rapidly toggled (actuated) between opposing drive directions such that horizontal surface 111A moves back and forth (i.e., rapidly and repeatedly changes between movement in the +X direction and movement in the −X direction). The rapid back and forth movement of horizontal surface 111A is transferred by to the previously dispensed objects by way of frictional contact, thereby causing separation between the dispensed objects”).
Regarding claim 51, Keraly further teaches:
The processing station of claim 36, wherein the imaging system captures images of the workpieces at a second area different from the loading area to achieve a higher processing rate (at least as in paragraph 0048, “FIG. 3B depicts an object separation mechanism 110B according to a presently preferred practical embodiment including two separating units 110A-1 and 110A-2 disposed in a parallel side-by-side arrangement”; at least as in paragraph 0039, 0052, & Figs. 5A-5B, wherein the work cell captures images of the still bottles on the first conveyor and transfers the target bottles, while the vision system determines through image processing techniques whether a target bottle is acquirable or sufficiently separated on a second, different conveyor which “achieves increased bottle/object processing rates by avoiding dynamic manipulation delays”).
Regarding claim 52, Keraly further teaches:
The processing station of claim 37, wherein the second area comprises at least one of an imaging location and a second loading area (at least as in paragraph 0075, “work cell unit 100D-2 includes a second object separating mechanism 110B-2, a second pick-and-place robot mechanism 130C-2, a second 4-axis robot mechanism 150C-2 holding a second carousel-type robotic end-tool 160D-2, a second processing device 170D-2 and a second positioning structure 190D-2 that are also connected to base plate 101E”; Examiner notes wherein “the second area” is being interpreted as “a second area” in accordance with proper antecedent basis).
Regarding claim 53, Keraly teaches:
An automated assembly line (Fig. 13, a robotic work cell 100E) comprising:
a conveyor comprising one or more one or more movers comprising one or more of a bed, a pallet, or a shuttle a plurality of modular processing stations arranged in proximity to the conveyor, at least one of the plurality of modular processing stations is a feeding station comprising: (at least as in paragraph 0075, “a robotic work cell 100E according to another exemplary specific embodiment in which two work cell units 100D-1 and 100D-2, each configured in accordance with robotic work cell 100D described above, are operably connected to an integral sheet-metal base plate 101E in an inverted mirror arrangement and are independently controlled by at least one control unit 180E… two exit ports 102E-1 and 102E-2 are cut or otherwise defined in base plate 101E to facilitate removal of fully processed objects released from robotic end-tools 160D-1 and 160D-2, respectively (i.e., such that the released fully processed objects fall through exit ports 102E-1 and 102E-2 and into one or more output bins)”
a frame having a fixed dimension (at least as in paragraph 0053, wherein “Robotic work cell 100C includes a generalized object separating mechanism 110C, a pick-and-place robot 130C, and an object processing robot 150C that are fixedly connected to a rigid base plate 101C (e.g., a plate metal sheet)”);
a loading area coupled to the frame presenting a plurality of workpieces, the loading area comprising a workpiece re-orienting mechanism capable of adjusting an orientation of workpieces in the loading area (at least as in paragraph 0032, “object separating mechanism 110 includes a horizontal surface 111 and a dynamic manipulator 112 that cooperatively rearrange bottles received from 3D cluster 90-3D into 2D arrangement 90-2D”; at least as in paragraph at least as in paragraph 0053 & 0075, wherein object separating mechanism is fixedly connected to a rigid base plate);
a pick-and-place robot coupled to the frame and controllable to move a workpiece from the plurality of workpieces presented in the loading area to a moveable receiving area (at least as in paragraph 0034, “Pick-and-place robot mechanism 130 is configured to move target bottle 91T(t1) from horizontal surface 111 to designated hand-off location HOL in a manner that facilitates a hand-off operation (described below) using location data X1,Y1 and rotational orientation data θ1”; at least as in paragraph at least as in paragraph 0053 & 0075, wherein pick-and-place robot is fixedly connected to a rigid base plate);
an imaging system coupled to the frame capturing images of the loading area (at least as in paragraph 0033, “Vision system 120 is a vision-based sensing system configured to sequentially identify (select) individual target bottles/objects from the bottles/objects 91(t1) forming 2D arrangement 90-2D(t1) and to generate data that facilitates the removal of each selected target bottle/object from horizontal surface 111 and subsequent delivery to a designated hand-off location HOL by pick-and-place robot 130”; at least as in paragraph 0075, wherein a protective “fish bowl” frame is formed above base plate to support cameras utilized by the associated vision systems); and
(at least as in paragraph 0056, “Shared power/control resources (control unit) 180C, which is depicted as a computer for brevity, is configured using known techniques to manage power distribution and to coordinate the various operations that achieve the desired bulk bottle (or other object) processing described herein”) configured to:
process one or more of the captured images to identify one or more pickable workpieces in the loading area (at least as in paragraph 0039, “Referring to block 220, a target object is then selected from the objects forming the 2D arrangement, and then location and rotational orientation data operably describing the location and rotational orientation of the target object are generated”; at least as in paragraph 0049, “vision system 120A then identifies individual objects (e.g., by way of comparing the current image data CID with stored image data SID that operably describes the objects) and selects (identifies and designates) one of the stationary objects as the next target object and generates the associated location and rotational orientation data”); and
operate the pick-and-place robot to transfer the identified one or more pickable workpieces to the moveable receiving area (at least as in paragraph 0049, “As indicated by block 220-2, vision system 120A then transmits the location and rotational orientation data to the pick-and-place robot mechanism as described additional detail below to facilitate the pick-up/removal and transfer of the selected target object to the hand-off location”; at least as in paragraph 0040, “Referring to block 230, the location data is then used to remove (pick-up) the selected target object from the horizontal surface”).
Regarding claim 55, Keraly further teaches:
The automated assembly line of claim 53, wherein the feeding station further comprises:
a second loading area, and wherein the imaging system is operable to capture an image of a first set of workpieces loaded onto the loading area and a second image of a second set of workpieces loaded onto the second loading area, and wherein the pick-and-place robot is operable to retrieve one or more pickable workpieces from the loading area while the station controller applies a machine-learning model to the second image to identify one or more pickable workpieces from the second loading area (at least as in paragraph 0048, “FIG. 3B depicts an object separation mechanism 110B according to a presently preferred practical embodiment including two separating units 110A-1 and 110A-2 disposed in a parallel side-by-side arrangement”; at least as in paragraph 0039, 0052, & Figs. 5A-5B, wherein the work cell captures images of the still bottles on the first conveyor and transfers the target bottles, while the vision system determines through image processing techniques whether a target bottle is acquirable or sufficiently separated on the second conveyor unit).
Regarding claim 56, Keraly further teaches:
The automated assembly line of claim 53, wherein the fixed dimension of the frame of the feeding station is common to at least one of the plurality of modular processing stations (at least as in paragraph 0075, “two work cell units 100D-1 and 100D-2, each configured in accordance with robotic work cell 100D described above, are operably connected to an integral sheet-metal base plate 101E in an inverted mirror arrangement and are independently controlled by at least one control unit 180E that is at least partially disposed below base plate 101E”).
Regarding claim 57, Keraly further teaches:
The automated assembly line of claim 53, wherein the loading area of the feeding station comprises one or more of:
a dial-based feeder;
a bowl feeder;
a vibratory feeder;
a linear feeder;
a conveyor feeder;
a hopper; or a shaker tray (at least as in paragraph 0047, “Object separation mechanism 110A generally includes a conveyor belt mechanism 310A, a stationary frame 316A and a gating mechanism 318A”).
Regarding claim 58, Keraly further teaches:
The processing station of claim 53, wherein the imaging system captures images of the workpieces at a second area different from the loading area to achieve a higher processing rate (at least as in paragraph 0048, “FIG. 3B depicts an object separation mechanism 110B according to a presently preferred practical embodiment including two separating units 110A-1 and 110A-2 disposed in a parallel side-by-side arrangement”; at least as in paragraph 0039, 0052, & Figs. 5A-5B, wherein the work cell captures images of the still bottles on the first conveyor and transfers the target bottles, while the vision system determines through image processing techniques whether a target bottle is acquirable or sufficiently separated on a second, different conveyor which “achieves increased bottle/object processing rates by avoiding dynamic manipulation delays”).
Regarding claim 59, Keraly further teaches:
The processing station of claim 53, wherein the second area comprises at least one of an imaging location and a second loading area (at least as in paragraph 0075, “work cell unit 100D-2 includes a second object separating mechanism 110B-2, a second pick-and-place robot mechanism 130C-2, a second 4-axis robot mechanism 150C-2 holding a second carousel-type robotic end-tool 160D-2, a second processing device 170D-2 and a second positioning structure 190D-2 that are also connected to base plate 101E”; Examiner notes wherein “the second area” is being interpreted as “a second area” in accordance with proper antecedent basis).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 38 and 54 is/are rejected under 35 U.S.C. 103 as being unpatentable over Keraly in view of Wellman et al. (US 20160167228 A, hereinafter Wellman).
Regarding claim 38, Keraly teaches the processing station of claim 37 but does not explicitly teach wherein the moveable receiving area is part of the one or more movers on the conveyor.
However, Wellman, in the same field of endeavor of robot manipulator control for grasping operations, specifically teaches wherein the moveable receiving area is part of the one or more movers on the conveyor (at least as in paragraph 0091, wherein “The robotic arm 912 may receive instructions to perform a particular grasping strategy to move an inventory item 940 from a tray 918 and into the release area 906, e.g., into a box 922 positioned on a shelf 924A-D of a stadium shelving unit 920… Conveyance mechanisms 928 may be provided on any of the shelves 924A-D of the stadium shelving unit 920 to move filled boxes from the release area 906 to the packing area 908, e.g., to within reach of a worker 926 in the packing area 908 for subsequent operations such as packing a box 922 with completed orders for shipping”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Humayun, to include Wellman’s teaching of a robotic arm picking station with a release area, since Wellman teaches wherein the grasping control system increases efficiency and throughput by improving the system’s capability to effectively move items by identifying target items to be grasped.
Regarding claim 54, Keraly teaches the automated assembly line of claim 53 but does not explicitly teach wherein the moveable receiving area of the feeding station is part of the one or more movers on the conveyor.
However, Wellman, in the same field of endeavor of robot manipulator control for grasping operations, specifically teaches wherein the moveable receiving area of the feeding station is part of the one or more movers on the conveyor (at least as in paragraph 0091, wherein “The robotic arm 912 may receive instructions to perform a particular grasping strategy to move an inventory item 940 from a tray 918 and into the release area 906, e.g., into a box 922 positioned on a shelf 924A-D of a stadium shelving unit 920… Conveyance mechanisms 928 may be provided on any of the shelves 924A-D of the stadium shelving unit 920 to move filled boxes from the release area 906 to the packing area 908, e.g., to within reach of a worker 926 in the packing area 908 for subsequent operations such as packing a box 922 with completed orders for shipping”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Humayun, to include Wellman’s teaching of a robotic arm picking station with a release area, since Wellman teaches wherein the grasping control system increases efficiency and throughput by improving the system’s capability to effectively move items by identifying target items to be grasped.
Claim(s) 42-45 is/are rejected under 35 U.S.C. 103 as being unpatentable over Keraly in view of Humayun et al. (US 20220016766 A1, hereinafter Humayun).
Regarding claim 42, Keraly further teaches:
The processing station of Claim 36, wherein the station controller is configured to apply a machine-learning model to the one or more captured images to identify the one or more pickable workpieces(at least as in paragraph 0039, “Referring to block 220, a target object is then selected from the objects forming the 2D arrangement, and then location and rotational orientation data operably describing the location and rotational orientation of the target object are generated. This process is described above with reference to the operations of vision system 120 and is described in further detail below with reference to FIGS. 4, 5A and 5B. Although the present invention may be implemented using vision system software that utilizes known image processing techniques (e.g., projection, background subtraction, object segmentation, identification, and Kalman filtering) to generate stored image data SID that operably visually describes target objects, operation of the robotic work cells described herein can be enhanced by way of utilizing more sophisticated software capable of detecting individual objects even if they are touching or overlapping, which could further increase work cell processing speeds”).
But Keraly does not explicitly disclose the machine-learning model being generated based on a set of training images in which one or more related workpieces were identified as pickable based on one or more of a position and an orientation of each related workpiece shown within a training image of the set of training images.
However, Humayun in the same field of endeavor of computer vision system for grasping an object, specifically teaches the machine-learning model being generated based on a set of training images in which one or more related workpieces were identified as pickable based on one or more of a position and an orientation of each related workpiece shown within a training image of the set of training images (at least as in paragraph 0033, wherein “The object detector functions to detect objects and/or other information in images… the object detector can determine: individual instances of one or more object types, object parameters for each object (e.g., pose, principal axis, occlusion, etc.), total object count, and/or other object information”; at least as in paragraph 0034, wherein “The object detector can be a neural network”; at least as in paragraph 0023, wherein “The object detectors can be trained using synthetic data (and/or annotated real-world data) and subsequently used to guide real-world training data generation”; at least as in paragraph 0064, wherein “training images can be labelled with a plurality of grasp outcomes (e.g., a grasp outcome for each of a plurality of grasp points), and/or otherwise suitably labelled. The images can optionally be a labelled with: object parameters associated with the grasp point/pixel (e.g., as determined by the object detector and/or grasp planner), such as: a surface normal vector, a face tag, an object principal axis pose; an end effector pose (e.g., as determined by a grasp planner; an index associated therewith, such as an index along a kinematic branch for the robotic arm; in joint space, in cartesian space, etc.), and/or any other suitable label parameters”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Keraly, to include Humayun’s teaching of training object detectors using real-world data and to generate real-world training data generation, since Humayun teaches wherein the generated training data is from a real-world scene, the graspability network can be trained and tested on data from the same distribution, thus improving the accuracy of the grasping system, reducing computation time, and decreasing required computational bandwidth.
Regarding claim 43, in view of the above combination of Keraly and Humayun, Keraly further teaches:
The processing station of Claim 42, wherein the station controller is configured to identify a region of interest within the one or more captured images, (at least as in paragraph 0033, “vision system 120 utilizes a camera or other device configured to capture current image data CID from a capture image region 122 including horizontal surface 111, identifies individual bottles/objects 91(t1) by comparing current image data CID with stored image data SID using software executed by a processor 125, and then selects a target bottle 91T(t1) from other identified bottles/objects 91(t1) using known techniques. In the embodiment depicted in FIG. 1, current image data CID includes image data that allows processor 125 to distinguish upside-down bottles (i.e., bottles of 2D arrangement 90-2D(t1) having their rear surface facing upward) from frontside-up bottles (i.e., bottles of 2D arrangement 90-2D(t1) having their front surface facing upward), and target bottle 91T(t1) is exclusively selected from identified frontside-up bottles”).
But Keraly does not explicitly disclose the region of interest comprising an engagement portion of the one or more pickable workpieces for an end-of- arm-tooling component of the pick-and-place robot to engage the one or more pickable workpieces.
However, Humayun in the same field of endeavor of computer vision system for grasping an object, specifically teaches the region of interest comprising an engagement portion of the one or more pickable workpieces for an end-of- arm-tooling component of the pick-and-place robot to engage the one or more pickable workpieces (at least as in paragraph 0085, wherein “The trained graspability network receives the image (e.g., RGB, RGB-D) as an input, and can additionally or alternatively include object detector parameters as an additional input (an example is shown in FIG. 7). The trained graspability network can output: a graspability map (e.g., including a dense map of success probabilities, object parameter values, and/or robotic end effector parameter values) and/or a success probability for multiple objects' grasps (e.g., multiple points/pixels), and/or any other suitable outputs”; at least as in paragraph 0070 wherein “The graspability map preferably includes a grasp success probability for each image feature (e.g., pixel (i, j), superpixel, pixel block, pixel set, etc.), but can alternatively include a grasp failure probability, a grasp score, object parameters (e.g., wherein the network is trained based on the object parameter values for the grasp points; such as object surface normals), end effector parameters (e.g., wherein the network is trained based on the robotic manipulator parameters for the training grasps; such as gripper pose, gripper force, etc.), a confidence score (e.g., for the grasp score, grasp probability, object parameter, end effector parameter, etc.), and/or any other suitable information for any other suitable portion of the image (examples shown in FIG. 4 and FIG. 5). The image feature can depict a physical region: smaller than, larger than, substantially similar to, or otherwise related to the robotic effector's grasping area”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Keraly, to include Humayun’s teaching of the graspability network generating graspability scores for pixels and/or a graspability map for an image of the scene wherein the grasp(s) can be selected based on the graspability scores, since Humayun teaches wherein the graspability network facilitates rapid target selection from a dense object scene (e.g., including numerous occluded objects) without requiring explicit detection of different object instances in the scene, thus improving the accuracy of the grasping system, reducing computation time, and decreasing required computational bandwidth.
Regarding claim 44, Keraly further teaches:
The processing station of Claim 36, wherein an end-of-arm-tooling component of the pick-and-place robot comprises a vacuum having a vacuum cup size, (at least as in paragraph 0054, “six-axis robot 130C is implemented using a mini six-axis robot (e.g., model LR Mate 200iD provided by Fanuc America Corporation of Rochester Hills, Mich., USA), and includes a suction cup 137C operably connected to distal end portion 135C and controlled by a vacuum system 140C, which supplies suction (vacuum) force S to suction cup 137C during periods described below with reference to FIG. 7 to facilitate the pick-and-place operations described herein”).
But Keraly does not explicitly disclose and wherein an engagement portion of the one or more pickable workpieces comprises a surface area that can accommodate the vacuum cup size.
However, Humayun in the same field of endeavor of computer vision system for grasping an object, specifically teaches and wherein an engagement portion of the one or more pickable workpieces comprises a surface area that can accommodate the vacuum cup size (at least as in paragraph 0029, wherein “In a first example, the end effector is a suction gripper”; at least as in paragraph 0061, wherein “when the end effector is a suction gripper, a pressure measurement device can measure the pressure. When the pressure change is above a threshold, the grasp point can be labelled as a grasp success and otherwise labelled as a grasp failure. If the pressure change is above a threshold for less than a predetermined period (e.g., before an instruction to drop the object), then the grasp point can be labelled as a grasp failure (e.g., the object was grasped and dropped)”; at least as in paragraph 0070, wherein “The image feature can depict a physical region: smaller than, larger than, substantially similar to, or otherwise related to the robotic effector's grasping area”; at least as in paragraph 0086, wherein “The grasp can be a pixel (or point associated therewith) and/or a set thereof (e.g., contiguous pixel set cooperatively representing a physical region substantially similar to the robotic manipulator's grasping area)”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Keraly, to include Humayun’s teaching of the graspability network generating graspability scores for pixels, a graspability map for an image of the scene wherein the grasp(s) can be selected based on the graspability scores, and auxiliary scene information, since Humayun teaches wherein the graspability network facilitates rapid target selection from a dense object scene (e.g., including numerous occluded objects) without requiring explicit detection of different object instances in the scene and the grasps can be further selected based on the auxiliary data (e.g., the grasps identified from the heatmap can be prioritized based on the corresponding object poses), thus improving the accuracy of the grasping system, reducing computation time, and decreasing required computational bandwidth.
Regarding claim 45, Keraly further teaches:
The processing station of Claim 36, wherein an end-of-arm-tooling component of the pick-and-place robot comprises a gripper having a gripper size and a gripper stroke (at least as in paragraph 0035, “Gripper mechanism 163 includes a pair of opposing gripper structures 164 that are operably configured to grasp and maintain control of the target bottle when moved from designated hand-off location HOL”)…
But Keraly does not explicitly disclose wherein an engagement portion of the one or more pickable workpieces comprises edge portions that can accommodate the gripper size and the gripper stroke.
However, Humayun in the same field of endeavor of computer vision system for grasping an object, specifically teaches the engagement portion of the one or more pickable workpieces comprises edge portions that can accommodate the gripper size and gripper stroke (at least as in paragraph 0029, wherein “In a second example, the end effector is a claw gripper (e.g., dual prong, tri-prong, etc.)”; at least as in paragraph 0062, wherein “The grasp point can be labelled as a grasp failure when: … the finger gripper is open beyond a predetermined width… The grasp point can be labelled as a grasp success when the force between fingers is above a predetermined threshold, if the gripper is open to within a predetermined width (e.g., associated with the width of an object), and/or any other suitable condition”; at least as in paragraph 0070, wherein “The image feature can depict a physical region: smaller than, larger than, substantially similar to, or otherwise related to the robotic effector's grasping area”; at least as in paragraph 0086, wherein “The grasp can be a pixel (or point associated therewith) and/or a set thereof (e.g., contiguous pixel set cooperatively representing a physical region substantially similar to the robotic manipulator's grasping area)”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Keraly, to include Humayun’s teaching of the graspability network generating graspability scores for pixels, a graspability map for an image of the scene wherein the grasp(s) can be selected based on the graspability scores, and auxiliary scene information, since Humayun teaches wherein the graspability network facilitates rapid target selection from a dense object scene (e.g., including numerous occluded objects) without requiring explicit detection of different object instances in the scene and the grasps can be further selected based on the auxiliary data (e.g., the grasps identified from the heatmap can be prioritized based on the corresponding object poses), thus improving the accuracy of the grasping system, reducing computation time, and decreasing required computational bandwidth.
Claim(s) 46 is/are rejected under 35 U.S.C. 103 as being unpatentable over Keraly in view of Li (US 20230297068 A1).
Regarding claim 46, Keraly teaches the processing station of Claim 36 but does not explicitly teach wherein the pick-and-place robot comprises a plurality of end-of-arm-tooling components, and wherein the station controller is configured to select an end-of-arm-tooling component from amongst the plurality of end-of-arm-tooling components for retrieving the one or more pickable workpieces.
However, Li, in the same field of endeavor of a robot system configured to, based on various pick-up conditions for grasping operations, grasp a plurality of objects, generate training data, and train a machine learning model, specifically teaches wherein the pick-and-place robot comprises a plurality of end-of-arm-tooling components, and wherein the station controller is configured to select an end-of-arm-tooling component from amongst the plurality of end-of-arm-tooling components for retrieving the one or more pickable workpieces (at least as in paragraph 0052, wherein “The receiving unit 110 may receive the pick-up condition, which includes the information on the type of pick-up hand 31, the shape and size of the portion contacting the workpiece 50, etc., input by the user via the input unit 12, and may store the pick-up condition in the later-described storage unit 14. That is, the receiving unit 110 may receive information and store such information in the storage unit 14, the information including information on whether the pick-up hand 31 is of the air suction type or the gripping type, information on the shape and size of a suction pad contact portion where the pick-up hand 31 contacts the workpiece 50, information on the number of suction pads, information on the interval and distribution of a plurality of pads in a case where the pick-up hand 31 has the plurality of suction pads, and information on the shape and size of a portion where a gripping finger of the pick-up hand 31 contacts the workpiece 50, the number of gripping fingers, and the interval and distribution of the gripping fingers in a case where the pick-up hand 31 is of the gripping type”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Keraly, to include Li’s teaching of a manipulator system configured to determine candidates for the pick-position of the workpiece based on the pick-up conditions, since Li teaches wherein the manipulator system with a pick-up condition including the type of pick-up hand avoids collision with a surrounding obstacle thus improving operation safety and efficiency.
Claim(s) 50 is/are rejected under 35 U.S.C. 103 as being unpatentable over Keraly in view of Sun et al. (US 20210024297 A1, hereinafter Sun).
Regarding claim 50, Keraly teaches the processing station of Claim 36 but does not explicitly teach further comprising a workpiece purge mechanism operable to remove unpickable workpieces from the loading area.
However, Sun, in the same field of endeavor of automatic product inspection systems, and more particularly to a system and method for sorting moving objects, specifically teaches:
further comprising a workpiece purge mechanism operable to remove unpickable workpieces from the loading area (at least as in paragraph 0038, “the microprocessor unit 120 is able to control the object sorting device 13 to apply an object sorting process to the plurality of objects 3 that are delivered by the belt conveyor 2 according to the plurality of classified object images, thereby sorting the plurality of objects 3 into at least two object group consisting of a normal object group and a defective object group. For example, the object sorting unit 131 is a robot arm, and is controlled by the control unit 130 so as to sort the plurality of objects 3 into a plurality of normal objects 3a and a plurality of defective objects 3b. As FIG. 1 shows, the plurality of normal objects 3a are eventually delivered into a normal object collecting device 15 that is disposed near an end side of the belt conveyor 2. Moreover, the plurality of defective objects 3b are consequently delivered into a defective object collecting device 16 that is disposed near an end side of the belt conveyor 2.”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Keraly, to include Sun's teaching of automated defective object sorting, since Sun teaches wherein the system automates the quality control and sorting procedure and standardizes the rules for quality control inspection, thus improving efficiency and consistency.
Claim(s) 48 is/are rejected under 35 U.S.C. 103 as being unpatentable over Keraly in view of Skyum et al. (US 20230150777 A1, hereinafter Skyum).
Regarding claim 48, Keraly teaches the processing station of Claim 36 but does not explicitly teach wherein the loading area is moveable between an imaging location and a picking location, and wherein the loading area remains at the imaging location while the imaging system captures the one or more captured images and moves to the picking location prior to the pick-and-place robot engaging the one or more pickable workpieces.
However, Skyum discloses a robot system with a pick and place robot arranged to pick an object from a continuously moving feeding conveyor transporting a stream of objects in bulk. Skyum specifically teaches wherein “wherein the loading area is moveable between an imaging location and a picking location, and wherein the loading area remains at the imaging location while the imaging system captures the one or more captured images and moves to the picking location prior to the pick-and-place robot engaging the one or more pickable workpieces” (at least as in Fig. 1 & paragraph 0106, wherein “the robotic actuator RA is a gantry type actuator, i.e. it has a fixed part with one set of elongated elements mounted on the ground in one end adjacent to the feeding conveyor FC and in the opposite end adjacent to the induction I1, so as to allow a movable part of the robotic actuator RA to move along the elongated elements to move a controllable gripper G between a gripping area GA where to pick up and object G_O on the feeding conveyor FC and to a target area TA on the induction I1”; at least as in paragraph 0109, wherein “The basic input to the pick and place robot RA, G, CS is a sensor system with a 3D camera CM mounted on a fixed position to provide a 3D image IM of an image area IMA upstream of the position of the pick and place robot RA, G”; at least as in paragraph 0110, wherein “The image area IMA is preferably located at least a minimum distance upstream of the gripping area GA”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Humayun, to include Skyum’s teaching of a robot system configured to pick and place objects from a gripping area separate from an image area, since Skyum teaches wherein the robot system picks and places objects from bulk with a high rate of success and at a high throughput.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RICARDO I VISCARRA whose telephone number is (571)270-0154. The examiner can normally be reached M-F 9-12 & 2-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Mott can be reached on (571) 270-5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RICARDO I VISCARRA/Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights