DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: processing device in claim 6. The term “processing device” shall be interpreted as one of a class of printing devices capable of applying identifying indicia in accordance with page 8, lines 29-32 of the specification.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 4 and 8 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 4 recites, in relevant part, “wherein the depth information is preferably determined with the aid of a point cloud.” The term “preferably” assigns a uncertainty to the limitation such that it is unclear whether or not the depth information MUST be determined with the aid of a point cloud, or if this is an optional feature.
Claim 8 is worded awkwardly, and Examiner is unable to determine precisely what is meant to be claimed. Examiner believes that the claim is intended to convey the subject matter discussed on page 9, lines 8-14 of the specification where each workpiece is checked for potential collisions before a picking operation, and only workpieces which are not at risk of a collision with either the container or another work piece are selected for picking. Claim 8 is thus examined in light of this interpretation, but the claim remains indefinite until amended for clarity.
Claim 1 recites the limitation "placement area" in line 25. There is insufficient antecedent basis for this limitation in the claim. Examiner proceeds on the assumption that “placement area” refers to the “storage area” introduced earlier in the claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 9, and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shapiro (DE-102022209788-A1) in view of Ban (US-20060104788-A1) and Xu (US 20150224650 A1).
Claim 1
Shapiro teaches
an automation module with
a removal area, a storage area,
(Shapiro - [0032] At least the stacking area 22a of the work area 22 is located in the field of view 25 of a depth camera 28a and a 2D optical image camera 28b.
In one example, the secondary area 22b of the work area 22 is also in the field of view 25 of the depth camera 28a and the 2D optical image camera 28b.)
EXAMINER NOTE: The stacking area 22a corresponds to the removal area, and the secondary area 22b corresponds to the storage area.
a base gantry, … a robot device arranged on the base gantry and having a gripper that can be moved into a closed position and an open position,
(Shapiro - [0029] The robot manipulator 26 is configured to move an end effector 27 within a configuration space that includes the stacking area 22a and the secondary area 22b.
The robot manipulator 26 can be selected from a variety of means, e.g. B. a robot arm palletizer, a four-axis Cartesian portal palletizer or any other means that can remove or add stackable objects 24 at a time from or onto a stack.)
EXAMINER NOTE: The robot may be a cartesian portal palletizer, which is a gantry-type robot.
(Shapiro - [0030] The robot manipulator 26 includes an end effector 27 which can be positioned to grasp a stackable object 24a-d that is located either in the stacking area 22a or in the secondary area 22b of the work area 22.
The end effector can be, for example, a suction cup, a claw, or another end effector that can move stackable objects.
a camera device … directed toward the removal area,
(Shapiro - [0032] At least the stacking area 22a of the work area 22 is located in the field of view 25 of a depth camera 28a and a 2D optical image camera 28b.
In one example, the secondary area 22b of the work area 22 is also in the field of view 25 of the depth camera 28a and the 2D optical image camera 28b.)
and a control device,
(Shapiro - [0012] According to a fourth aspect, a computer program element is provided that includes a set of machine-readable instructions which, when executed by a processor, cause a computer to perform the steps of the computer.)
wherein the method comprises the following steps:
a. providing a blank carrier equipped with an open top and outer sides in the removal area, wherein the blank carrier holds a plurality of workpieces,
(Shapiro - [0028] As shown, the stacking area 22a of the work area 22 consists of a container 23 with sides having a total height of Δz units.)
EXAMINER NOTE: See Annotated Fig. 2 below. The top of the container is open. The container is located in stacking area 22a (removal area)
PNG
media_image1.png
522
740
media_image1.png
Greyscale
b. capturing an image of the position of the blank carrier and the workpieces arranged in the blank carrier using the camera device,
(Shapiro - [0082] Fig. 12 shows an example of segmentation from a partial grasping sequence.
Segmentation masks generated from the first and second image subsets are marked with the reference numbers 90 and 91.
[0083] Fig. 13 shows an example of Box Detection Bounding Boxes from a partial grasping sequence of a grasping proposal.
Box Detection Bounding Boxes are represented by the reference numbers 92 and 93.)
c. determining position information of the workpieces in relation to each other and in relation to the outer sides, taking into account the position image, wherein the position information comprises individual positions of the workpieces in the blank carrier,
(Shapiro - [0049] According to one embodiment, the method further comprises: capturing at least one container side wall 23 in the stacking area 22a in the first depth map and removing it from a configuration space of the robot manipulator, a part of an exclusion area 80, 81 of the stacking area 22a partially defined by the at least one container side wall, so that the gripping proposal P1 does not require the robot manipulator to move a stackable object 24 within the exclusion area 80, 81.
[0054] According to one embodiment, the gripping proposal P1 is generated by scanning the point cloud representation of the work area 22 (initial depth map) using the dimension and/or shape attribute data of the stackable objects 24, and designating at least one region in the point cloud representation and/or the first depth map as the prospective gripping proposal P1, wherein the position of the prospective gripping proposal P1 correlates with the position of a stackable object 24 in the work area 22.
A gripping suggestion is represented as a data structure in, for example, a database, which provides the identity of a region in, for example, XYZ coordinates of the work area that a robot manipulator 26 is to remove.
The gripping suggestion P1-P3 can include a data structure that refers to previous and subsequent gripping points in the work area.
The one or more gripping suggestions can be automatically compiled in machine instructions for controlling the end effector of a robot manipulator 26.)
d. selecting a suitable workpiece to be gripped from the workpieces arranged in the blank carrier, taking into account the position information,
(Shapiro - [0052] According to one embodiment, the method further comprises the calculation of at least one stack location hypothesis 82, 83, 84 using the initial depth map, wherein the at least one stack location hypothesis 82, 83, 84 defines a region of the stack area 22a that contains the stack.
During stack localization, the depth map (and/or the associated point cloud representation) is analyzed to identify objects above the floor of workspace 22.
One or more hypotheses are generated and the optimal hypothesis is selected.)
e. controlling the robot device by means of the control device by transmitting the position of the selected workpiece, so that the gripper is moved toward the workpiece, the workpiece is gripped by means of the gripper, and the gripped workpiece is removed from the blank carrier,
(Shapiro - [0054] According to one embodiment, the gripping proposal P1 is generated by scanning the point cloud representation of the work area 22 (initial depth map) using the dimension and/or shape attribute data of the stackable objects 24, and designating at least one region in the point cloud representation and/or the first depth map as the prospective gripping proposal P1, wherein the position of the prospective gripping proposal P1 correlates with the position of a stackable object 24 in the work area 22.
A gripping suggestion is represented as a data structure in, for example, a database, which provides the identity of a region in, for example, XYZ coordinates of the work area that a robot manipulator 26 is to remove.
The gripping suggestion P1-P3 can include a data structure that refers to previous and subsequent gripping points in the work area.
The one or more gripping suggestions can be automatically compiled in machine instructions for controlling the end effector of a robot manipulator 26.)
f. placing the gripped workpiece at a placement position selected from a group of predetermined placement positions in the placement area,
(Shapiro - [0036] As explained, the device 50 receives an optical image and a depth map during operation before a stackable object is manipulated.
These images form the first subgroup of training images.
The device 50 instructs the robot manipulator 26 to grasp the stackable object 24a using the end effector 27 based on a gripping proposal created according to the procedure to be discussed below.
The stackable object 24a is removed from the container according to the gripping suggestion and e.g. B. stored in the annex 22b.)
wherein workpieces placed at the placement positions are aligned with each other according to a predetermined arrangement.
(Shapiro - [0063] The method based on the first aspect is applied iteratively according to the calculated grasping suggestion.
Fig. 4 schematically shows side views for generating the first 30 and second 32 image subsets from a container with a stack of stackable objects with two levels, two stackable objects in the x-direction and one stackable object in the y-direction, for a total of four stackable objects.
For example, such a container 23 can be unpacked using a gripping proposal with five stages of a robot container unpacking process. (For clarity, robot manipulator 26 is not shown). Stackable objects that appear at the bottom layer of the container are hatched to represent the depth difference.
The depth maps D1-D5 and the visual images RGB 1-5 are received by a camera 28, which is essentially positioned above the container 23.
The camera 28 in Fig. 4 shows the stacking area 23a of the working area 22 and also the secondary area 22b, but this is not absolutely necessary.
[0064] Therefore, the sequence of depth maps D1-D5 and the visual images RGB 1-5 represent picture frames that exhibit differences in the spatial configuration of the stacking area 23a and/or the secondary area 22b is a spatial configuration according to a gripping proposal executed by a robot manipulator 26.)
EXAMINER NOTE: See Fig. 4. The objects are placed one on top of another according to the gripping proposal (predetermined arrangement).
PNG
media_image2.png
748
542
media_image2.png
Greyscale
Note that Ban also teaches many of the aspects of claim 1. Ban teaches
a robot device … having a gripper that can be moved into a closed position and an open position,
a camera device … directed toward the removal area
(Ban - [0027] … The video camera 36 is also used to obtain two dimensional characteristics of a measured object (i.e., the workpiece W).
[0028] … As shown in the drawing, the light projector 34 and the video camera 36, both composing the sensor head section of a three-dimensional visual sensor, are attached to the flange portion 44 at the distal end of the arm of the robot mechanical section 40, together with the hand 42 as a tool or an end-effector.
[0050] Step S11: While maintaining the hand 42 at the basic holding orientation, the robot mechanical section 40 is moved to a holding position (i.e., the position at which the hand 42 is operated to open or close).
A control device
(Ban - [0028] The object picking system 30 is also provided with a robot controller 38)
providing a blank carrier equipped with an open top and outer sides in the removal area, wherein the blank carrier holds a plurality of workpieces,
(Ban - [0026] FIG. 2 typically shows the entire configuration of an object picking system 30 according to an embodiment of the present invention and having the above-described basic configuration. The object picking system 30 according to the illustrated embodiment is a robot system including a hand and a visual sensor, and is capable of picking up a plurality of workpieces W stacked irregularly and randomly in a basket-type container 32 one by one from the container 32.)
capturing an image of the position of the blank carrier and the workpieces arranged in the blank carrier using the camera device … determining position information of the workpieces
(Ban - [0038] Step S1: The robot mechanical section 40 is moved to a measuring position. The measuring position is a previously taught position, such as a position above the center of the container 32 containing a plurality of workpieces W, where the image of several workpieces W can be captured in the field of view of the video camera 36.
[0042] Step S5: In the image of the workpiece W.sub.Q corresponding to the whole model M, regions in which the portions of the workpiece corresponding respectively to the partial models m1, m2 (i.e., inspected portions) can be included (usually, regions defined by the rectangular regions shown in FIG. 3A plus predetermined margins: hereinafter referred to as "existence regions") are calculated. In this step, reference is first made to the data of the position and rotation of the whole model M as previously taught (FIG. 3B), so as to estimate the deviation of the position and orientation of the workpiece W.sub.Q, corresponding to the whole model M, from the whole model M in the image.)
selecting a suitable workpiece to be gripped from the workpieces arranged in the blank carrier, taking into account the position information
e. controlling the robot device by means of the control device by transmitting the position of the selected workpiece, so that the gripper is moved toward the workpiece, the workpiece is gripped by means of the gripper, and the gripped workpiece is removed from the blank carrier
(Ban - [0009] To accomplish the above object, the present invention provides an object picking system comprising … a determining section determining, in the image of the object to be picked detected by the detecting section, whether an inspected portion of the object to be picked, … is concealed by another object, based on the appearance information of the reference object stored in the storage section; a control section deciding a picking motion for the object to be picked and outputting a control signal of the picking motion, based on a determination result of the determining section …)
Shapiro does not divulge much detail regarding the workpieces which are moved by their robot. However, Ban teaches
… plurality of workpieces, each of which has a gripping area,
wherein the workpieces are each arranged with their gripping area facing the top side and are otherwise arranged randomly in relation to each other and in relation to the outer sides,
(Ban - [0068] FIG. 7 is a top plan view showing a plurality of workpieces W1, W2, W3, . . . , each provided with two holes H1 and H2 of different sizes, stacked irregularly and randomly. When the workpiece having the above configuration, such as a mechanical component having holes, is to be held, it has been generally performed to insert fingers of the hand 42 (FIG. 2) into the hole H1 or H2 of the workpiece and to shift the fingers so as to exert outward pressure from inside the hole. In order to conduct the workpiece picking operation on such workpieces after detecting the mutual concealing of the workpieces, whole and partial models are first taught in the same way as in the above-described operation example.)
Shapiro's invention is directed to picking and placing "stackable objects." The workpieces picked by Ban are also stacked, albeit irregularly. Ban utilizes many of the same elements as Shapiro, which indicates that Shapiro's system should have no issue accommodating Ban's irregularly stacked workpieces if Ban's control steps are included. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Shapiro's pick and place system with Ban's additional control and processing methods in order to allow the system to reliably handle a variety of workpieces which may be irregularly stacked.
(Ban - [0008] It is an object of the present invention to provide an object picking system for picking up, one by one, a plurality of irregularly placed objects, which permits a picking operation on an object to be picked, on which another object is at least partially superimposed, to be performed without being deferred and in such a manner as to avoid the dropping of the object during the picking operation and the resulting damage of the object, and which thus can ensure a safe operation and prevent the lowering of the working efficiency)
Regarding the limitation of
… a camera device arranged on the base gantry…
Shapiro does not divulge extensive details regarding the gantry embodiment, and therefore does not explicitly state that the camera device is arranged on the gantry in the embodiments in which the robot is a gantry type robot. However, the idea of mounting a camera to a gantry is notoriously old and well-known in the art as evidenced by Xu. Xu utilizes a camera on a gantry to image objects in the same way Shapiro images objects.
(Xu - [0042] … The camera 102 here is located on a gantry 105, or tripod, above the table 101 and above the robot manipulator 104. Thus, the system correlates relationships among a plurality of various instruments (e.g. 201, 203, 204, etc) within the tray 200 during selection such that each instrument is individually identified and its position estimated based upon its occlusion within the clutter, or pile 210, of instruments.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to mount Shapiro's camera on the gantry, as demonstrated by Xu. The particular mounting point of the camera is irrelevant as long as the camera is able to sufficiently capture the workpieces to be identified.
Claim 2
The combination of Shapiro, Ban, and Xu teaches the limitations of claim 1 as outlined above. Shapiro further teaches
wherein the steps following the provision of the blank carrier are performed repeatedly for several different workpieces until no more workpieces are contained in the blank carrier.
(Shapiro - [0072] According to one embodiment, the method includes identifying that the spatial state of the stacking area 22a has reached a stop condition, wherein the stop condition is optionally that the working area 22 is empty or full of stackable objects 24, and outputting the first 30 and second image subsets 32.
An example of the stop condition is that the depth map obtained from the depth camera 28a for the entire area of the container bottom corresponds to the depth of the working area 22 bottom (within a specified tolerance which is lower than the height of the stackable objects 24).
This indicates that container 23 has been completely emptied by robot manipulator 26.)
Claim 3
The combination of Shapiro, Ban, and Xu teaches the limitations of claim 1 as outlined above. Shapiro further teaches
wherein when determining the position information of the workpieces, in addition to the position image, design data of the workpieces are also taken into account.
(Shapiro - [0054]According to one embodiment, the gripping proposal P1 is generated by scanning the point cloud representation of the work area 22 (initial depth map) using the dimension and/or shape attribute data of the stackable objects 24, and designating at least one region in the point cloud representation and/or the first depth map as the prospective gripping proposal P1, wherein the position of the prospective gripping proposal P1 correlates with the position of a stackable object 24 in the work area 22.)
EXAMINER NOTE: Point cloud representation of the work area includes shape attribute data of the stackable objects (design data of the workpieces).
Claim 4
The combination of Shapiro, Ban, and Xu teaches the limitations of claim 1 as outlined above. Shapiro further teaches
wherein when recording the position image of the blank carrier and the workpieces arranged in the blank carrier by means of the camera device, depth information is also recorded, wherein the depth information is preferably determined with the aid of a point cloud.
(Shapiro - [0054]According to one embodiment, the gripping proposal P1 is generated by scanning the point cloud representation of the work area 22 (initial depth map) using the dimension and/or shape attribute data of the stackable objects 24, and designating at least one region in the point cloud representation and/or the first depth map as the prospective gripping proposal P1, wherein the position of the prospective gripping proposal P1 correlates with the position of a stackable object 24 in the work area 22.)
Claim 9
The combination of Shapiro, Ban, and Xu teaches the limitations of claim 1 as outlined above. Shapiro further teaches
wherein when selecting a suitable workpiece to be gripped from the workpieces arranged in the blank carrier, gripping specifications are also taken into account.
(Shapiro - [0052] According to one embodiment, the method further comprises the calculation of at least one stack location hypothesis 82, 83, 84 using the initial depth map, wherein the at least one stack location hypothesis 82, 83, 84 defines a region of the stack area 22a that contains the stack.
During stack localization, the depth map (and/or the associated point cloud representation) is analyzed to identify objects above the floor of workspace 22.
One or more hypotheses are generated and the optimal hypothesis is selected.
The result of the stack localization affects the series of grab suggestions.
During depalletizing, a gripping suggestion should unload a pallet from the top of the pallet to the bottom of the pallet.
Within this requirement, there can be various grasping strategies (e.g.,...).
B. Stackable objects on each layer of the pallet can be picked from the back of the pallet to the front, or alternatively from the front of the pallet to the back, or alternatively in any pattern per layer.
Therefore, the precise localization of box stacks within the depth map determines the effectiveness of the grasping strategy.
EXAMINER NOTE: The requirement of unloading from top to bottom is a gripping specification in that it enforces the constraint that workpieces on the bottom cannot be removed until workpieces on top of them have been removed. See instant specification, page 10, lines 1-5.
Claim 12
The combination of Shapiro, Ban, and Xu teaches the method of claim 1 as outlined above. Modified Shapiro teaches this system such that the apparatus executes the method of claim 1. See rejection of claim 1.
Claim(s) 5-7 and 10-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shapiro, Ban, and Xu as applied to claim 1 above, and further in view of Wen (US 20220097889 A1).
Claim 5
The combination of Shapiro, Ban, and Xu teaches the limitations of claim 1 as outlined above. The cited combination may not explicitly teach the following limitations in combination. However, Wen teaches
wherein the gripped workpiece is moved by means of the robot device to a code detection device …,
and a first identification code applied to the gripped workpiece is detected by means of the code detection device.
(Wen - [0025] In an exemplary embodiment, the robotic labeling system 100 includes a package identification system 130 for identifying the package 102 at the labeling station 110. The package identification system 130 includes a scanning device 132 for identifying the package 102. The scanning device 132 is operably coupled to the controller 50. Signals or data from the scanning device 132 may be transmitted to the controller 50 to control other operations of the robotic labeling system 100, such as the palletizing robot 20 and/or the label application system 150. In various embodiments, the scanning device 132 may include one or more cameras 134. The scanning device 132 is used to scan, and may image, the identification tag 104 to identify the package 102. The identification tag 104 may be a barcode and the scanning device 132 may be a barcode reader. In various embodiments, the scanning device 132 may image the package 102. The package 102 may be identified with reference to the package database 12. The package 102 is scanned to identify the package to control other operations, such as printing appropriate labels, for proper label application, and the like.)
Regarding the limitations of
… code detection device arranged on the base gantry,
As discussed above, the combination of Shapiro and Xu anticipates the mounting of a camera on a gantry. Xu also discusses barcode reading functionality of the gantry-mounted camera.
(Xu - [0059] … A barcode reader or a high-resolution camera with barcode reading software may be used to locate and read the IDs of the visible barcodes. The integration of barcode reading software module with the camera allows simultaneous tracking and imaging of the instruments.)
Similarly, the mounting location of a code detection device is largely irrelevant as long as the device is able to sufficiently read the barcode. For this reason, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to arrange Wen's scanning device on Shapiro's gantry to achieve the predictable result of identifying workpiece identifiers. Further, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to apply Wen’s labeling scheme in Shapiro’s picking system in order to track the workpiece throughout the processing areas, and to apply additional useful information related to the piece being processed.
(Wen - [0022] In various embodiments, the identification tag 104 is a scannable tag, such as a barcode, a data matrix, a QR code, or another type of symbolic scan code. The identification tag 104 may be used to track the package 102 within a warehouse, such as movement of the package 102 between various processing stations.
[0024] In an exemplary embodiment, the package 102 receives one or more customer specified labels 108 at the labeling station 110. The customer specified label 108 or CSL 108 may contain information about the contents of the package 102 or other information. For example, the customer specified label 108 may contain information about the shipper of the package 102, the location of where the package 102 is being shipped from, return shipping information, warning labels regarding the package 102 or the content of the package 102, and the like. In various embodiments, the customer specified label 108 may include symbolic scan codes having data relating to the content of the package 102 or other information. The customer specified label 108 is applied to the package 102 by the label application system 150 at the labeling station 110.)
Claim 6
The combination of Shapiro, Ban, and Xu teaches the limitations of claim 1 as outlined above. The cited combination may not explicitly teach the following limitations in combination. However, Wen teaches
wherein a second identification code is applied to the gripped workpiece by means of a processing device assigned to the automation module, taking into account process information provided to the processing device.
(Wen - [0047] At step 524, if the system controller determines that the identification tag matches an identification tag in the package database, the method includes receiving 526 label data from the warehouse management system. The method includes receiving 528 positioning data for the label, which may be predetermined or specified by an AI algorithm in the system controller. For example, the AI algorithm may determine an appropriate label position based on the size and the shape of the package (for example, determined by the depalletizing scanning device). The method includes positioning 530 the package in front of the label application system, such as the label applicator. The package is positioned by the palletizing robot.
[0048] At step 532, the method includes printing a label and applying the label on the package. The label is printed by the labeling device and applied to the package by the label applicator. … The label may include a barcode and the label verification scanning device may be a barcode reader.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to apply Wen’s labeling scheme in Shapiro’s picking system in order to track the workpiece throughout the processing areas, and to apply additional useful information related to the piece being processed. See Wen – [0022] and [0024], cited above with respect to claim 5.
Claim 7
The combination of Shapiro, Ban, Xu, and Wen teaches the limitations of claim 6 as outlined above. In the cited combination, Wen further teaches
wherein the process information is stored in a workpiece database.
(Wen - [0025] … The package 102 may be identified with reference to the package database 12. The package 102 is scanned to identify the package to control other operations, such as printing appropriate labels, for proper label application, and the like.)
Claim 10
The combination of Shapiro, Ban, and Xu teaches the limitations of claim 1 as outlined above. The cited combination may not explicitly teach the following limitations in combination. However, Wen teaches
wherein at least one workpiece pallet is arranged in the storage area, onto which the gripped workpiece is placed,
(Wen - [0014] The target container can be a plastic container or cardboard box, in particular a shipping carton or other common load carrier, such as a box. For example, a wire mesh box or a pallet. …)
wherein, in particular after the workpiece pallet has been completely loaded with workpieces, the workpiece pallet is removed from the storage area by means of a transport system, in particular an unmanned transport system.
(Wen - [0033] FIG. 2 illustrates the robotic labeling system 100 in accordance with an exemplary embodiment showing additional pallets 90 of packages 102. The pallets 90 are transferred to and from the various stations using automated guided vehicles 92. For example, the automated guided vehicles 92 transfer the pallets 90 to the depalletizing station 60 and transfer the pallets 90 from the palletizing station 80, such as to a pallet wrapping station 94.
[0034] FIG. 3 illustrates the robotic labeling system 100 in accordance with an exemplary embodiment showing the additional pallets 90 of packages 102. The pallets 90 are transferred to and from the various stations using conveyors 96. For example, the conveyors 96 transfer the pallets 90 to the depalletizing station 60 and the conveyors 96 transfer the pallets 90 from the palletizing station 80 to the pallet wrapping station 94.)
EXAMINER NOTE: The pallets are carried out via AGVs or conveyors.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize automated means to move pallets through the facility in order to reduce labor costs. The automation of the pallet transfer process furthers Shapiro’s goal of handling workpieces without human intervention.
(Wen - [0050] … The method is automated to reduce labor cost in the depalletizing, labeling, and re-palletizing process. The method is flexible in that the palletizing robot is able to move and label different size and shape packages. The verification process reduces errors in the labeling process compared to user application systems.)
Claim 11
The combination of Shapiro, Ban, and Xu teaches the limitations of claim 1 as outlined above. The cited combination may not explicitly teach the following limitations in combination. However, Wen teaches
wherein when the blank carriers are supplied by means of a transport system, in particular a driverless transport system, and/or by means of a feeding system, they are positioned in the removal area.
(Wen - [0033] FIG. 2 illustrates the robotic labeling system 100 in accordance with an exemplary embodiment showing additional pallets 90 of packages 102. The pallets 90 are transferred to and from the various stations using automated guided vehicles 92. For example, the automated guided vehicles 92 transfer the pallets 90 to the depalletizing station 60 and transfer the pallets 90 from the palletizing station 80, such as to a pallet wrapping station 94.
[0034] FIG. 3 illustrates the robotic labeling system 100 in accordance with an exemplary embodiment showing the additional pallets 90 of packages 102. The pallets 90 are transferred to and from the various stations using conveyors 96. For example, the conveyors 96 transfer the pallets 90 to the depalletizing station 60 and the conveyors 96 transfer the pallets 90 from the palletizing station 80 to the pallet wrapping station 94.)
EXAMINER NOTE: The pallets are brought in via AGVs or conveyors.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize automated means to move pallets through the facility in order to reduce labor costs. The automation of the pallet transfer process furthers Shapiro’s goal of handling workpieces without human intervention. See Wen [0050], cited above with respect to claim 10.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shapiro, Ban, and Xu as applied to claim 1 above, and further in view of Truenbach (US 20190389062 A1)
Claim 8
EXAMINER NOTE: See comments regarding claim interpretation in the rejection of claim 8 under 35 U.S.C. 112(b).
The combination of Shapiro, Ban, and Xu teaches the limitations of claim 1 as outlined above. The cited combination may not explicitly teach the following limitations in the claimed combination. However, Truenbach teaches
wherein when selecting a suitable workpiece to be gripped from the workpieces arranged in the blank carrier,
at least for individual workpieces picked up in the blank carrier are checked at least until the check for a workpiece picked up in the blank loading carrier shows that the workpiece being checked would collide with another workpiece and/or with an outer side of the blank loading carrier until the check for a workpiece held in the blank carrier shows that there is no risk of a process malfunction for the checked workpiece,
whereupon this workpiece is selected as the workpiece to be gripped.
(Truebenbach - [0004] In one implementation, a method for identifying one or more candidate objects for selection by a robot. A path to the one or more candidate objects may be determined based upon, at least in part, a robotic environment and at least one robotic constraint. A feasibility of grasping a first candidate object of the one or more candidate objects may be validated. If the feasibility is validated, the robot may be controlled to physically select the first candidate object. If the feasibility is not validated, at least one of a different grasping point of the first candidate object, a second path, or a second candidate object may be selected.
[0113] Referring now to FIG. 18, a graphical user interface that allows for configuring workpiece collision shapes is provided. The user may configure the collision shapes for the workpiece. These shapes are used to detect and avoid collisions between the workpiece and the environment after the workpiece has been picked.
[0167] In some embodiments and as discussed above, determining 202 the path to the one or more candidate objects may include using information about one or more surfaces of at least one object adjacent to the candidate object and avoiding a collision with the at least one object adjacent the candidate object. In this manner, robotic bin picking process 10 may use information about surfaces of objects around the candidate workpiece when determining a path the candidate object to avoid a collision with the objects around the candidate workpiece. For example and in some embodiments, the information about the one or more surfaces of at least one object adjacent to the candidate object is gathered as part of identifying the candidate object. ...)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to further modify Shapiro by implementing Truenbach's suggestion to select workpieces based on collisions with the environment in order to avoid malfunctions due to external factors.
(Truenbach - [0175] … During the validation task, robotic bin picking process 10 may determine the robot motions that need to take place in order to pick and place a part, before the motion is actually executed. As a result, robotic bin picking process 10 can avoid a situation when robot stalls in the middle of the motion due to some environment or robot flexibility constraints.)
Claim(s) 13-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shapiro, Ban, and Xu as applied to claim 1 above, and further in view of Wagner (US-20230012957-A1).
Claim 13
The combination of Shapiro, Ban, and Xu teaches the limitations of claim 12 as outlined above.
As discussed above with respect to claims 1 and 12, Shapiro discloses an embodiment in which the robot is part of a gantry system. Shapiro does not disclose the specific structure of said gantry system. And does not explicitly teach the limitations of
wherein the base gantry has an outer first side part, an outer second side part spaced apart from the first side part, and a roof part resting on the first side part and on the second side part,
wherein the camera device and/or the robot device are arranged on the roof part in such a way that they hang into a working space bounded by the first side part, the second side part, and the roof part.
However, these structural features are well-known in the art, as evidenced by Wagner. With reference to the annotations made in Fig. 2 (see dashed lines and arrow labels), Wagner teaches
wherein the base gantry has an outer first side part, an outer second side part spaced apart from the first side part,
PNG
media_image3.png
642
787
media_image3.png
Greyscale
and a roof part resting on the first side part and on the second side part,
PNG
media_image4.png
642
787
media_image4.png
Greyscale
wherein the camera device and/or the robot device are arranged on the roof part in such a way that they hang into a working space bounded by the first side part, the second side part, and the roof part.
(Wagner - [0056] FIG. 2 , for example, shows a system 30 in accordance with an embodiment of the present invention that receives decanted vendor bins 32 on an in-feed conveyor 34 that pass by a plurality of processing stations 42. One (or more) programmable motion devices 40 such as an articulated arm having an end effector is provided suspended from a support fame 36. Adjacent a base of the articulated arm and also suspended from the support frame 36 is a perception unit 50 (as further discussed below with reference to FIG. 5 ). Additional perception units 56 may be provided (for example, near vendor bin in-feed conveyor 34) that capture perception information regarding a label that is applied to each bin 32 that associates the bin with the contents of the bin.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to incorporate Wagner's structural features of the gantry system into Shapiro's system as a specific means of achieving Shapiro's disclosed embodiment.
Claim 14
The combination of Shapiro, Ban, Xu, and Wagner teaches the limitations of claim 13 as outlined above. As part of the proposed modification, Wagner also teaches
wherein a third side part arranged between the first side part and the second side part, which divides the working space into a first working space area and a second working space area,
wherein at least one component from the group: removal area, storage area, robot device, camera device is assigned only to the first working space area,
wherein the automation module further has at least one further component from the group: removal area, storage area, robot device, camera device.
EXAMINER NOTE: See annotated Fig. 2 below. The ledge serves as a third part dividing the area into two working spaces, with the first working space containing the removal area and the second working space containing the storage area.
PNG
media_image5.png
642
761
media_image5.png
Greyscale
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAMES MILLER WATTS whose telephone number is (703)756-1249. The examiner can normally be reached 7:30-5:30 M-TH.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Mott can be reached at 571-270-5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAMES MILLER WATTS III/Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657
