DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Such claim limitation(s) is/are: representation unit, evaluation unit, planning unit in claim(s) 1 (first instance).
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
The representation unit, evaluation unit, and planning unit will be interpreted as functional software modules stored in computer memory and executed by a computer processor as described on page 15 of applicant’s specification as filed, or equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 15 rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because it is a “A computer program product including program code for carrying out the steps of the method as claimed in claim 14 when the program code is being executed on a computer”, which qualifies as software per se. The actual storage medium on which the computer program resides is not claimed. Products that do not have a physical or tangible form, such as information (often referred to as "data per se") or a computer program per se (often referred to as "software per se") are ineligible under 101 Step 1 when claimed as a product without any structural recitations. See MPEP 2106.03. It is recommended that Applicant amend to include a nontransitory computer readable medium storing the program to overcome this 101 rejection.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-8, 11-12, 14-17, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al (US 20210023711, hereinafter Lee) in view of Han et al (US 20200230817, hereinafter Han).
Regarding Claim 1, Lee teaches:
a device for loading or unloading (see at least " Robots 170A and 1708 are each a “robot arm” having multiple degrees of freedom to enable traversal of a corresponding grasping end effector 172A, 172B along any of a plurality of potential paths to position the grasping end effector in desired locations." in par. 0038 and “Robot 170A can access a robot workspace 101A that, in FIG. 1A, includes sunglasses 192A on a conveyor portion 103A of a conveyor system, and also includes a container 193A. The robot 170A can utilize object manipulation parameters, determined as described herein, in grasping the sunglasses 192A and placing them appropriately in the container 193A.” in par. 0039)
a sensor interface for receiving a sensor signal comprising information on workpieces present in a loading area (see at least " As described in detail herein, in various implementations additional vision component 194 can capture vision data that captures features of the spatula 192C. Further, the vision data can be utilized, by system 110 (described below), in determining object manipulation parameter(s) for enabling the robot 170A or robot 170B to manipulate (e.g., pick and place) the spatula 192C." in par. 0042)
a representation unit for creating a representation of the workpieces and of the loading area, based on the sensor signal; (see at least " For example, the system 110 can determine the object manipulation parameter(s) based at least in part on user interface input(s), from a remote client device 130, directed at a visual representation that is generated at least in part on the vision data captured by additional vision component 194 (e.g., based at least in part on object features, of the vision data, that capture features of the spatula 192C)." in par. 0042 and “The visual representation can also optionally include indication(s) of predicted object manipulation parameter(s) (if any), from prediction engine(s) 112. An example of a predicted object manipulation parameter from prediction engine(s) 112 is illustrated in FIG. 2E, described in more detail below. The visual representation can also optionally include an environmental representation of other environmental objects (e.g., a work surface, a container in which the at least one object is to be placed) and/or a robot representation of all or parts of the robot.” in par. 0049)
a user interface for providing said representation to a machine operator and for receiving a user input from the machine operator comprising information on a position of the workpieces in the representation; (see at least " The remote client device 130 includes a display engine 132, input engine(s) 134, and input device(s) 136." in par. 0045 and “For example, the object manipulation parameter(s) indicated by the data generated by the input engine(s) 134 of an instance of user interface input(s) can include: a grasp pose; a placement pose; a sequence of waypoint(s) to encounter in traversing to a grasp pose; a sequence of waypoints to encounter in traversing toward a placement pose (after grasping the object); a full path or trajectory (i.e., a path with velocity, acceleration, jerk, and/or other parameter(s)) in traversing to and/or from a manipulation pose (e.g., a grasp pose or other manipulation pose); and/or other object manipulation parameter(s). The user interface input(s) of an instance are provided by an operator of the remote client device 130, with reference to a visual representation rendered by display engine 132. For instance, an instance of user interface inputs can indicate a full trajectory that is utilized during assembly of a part utilizing a plurality of component parts.” In par. 0054)
an evaluation unit for determining a revised representation of the workpieces and the loading area, based on the sensor signal and on the user input, the user interface being designed for providing said revised representation to the machine operator and for receiving a further user input comprising information on a workpiece loading operation to be carried out; (see at least " The operator can, for example, actuate a first virtual button (e.g., virtual button 282A1) or hardware button to start defining of the path 289A1, and actuate a second virtual or hardware button to define an end of the path 289A1, which also constitutes the grasp pose. Although not illustrated, it is noted that the simulated robot 270A can “move” during defining of the trajectory 289A1 to provide the operator with visual feedback of the path 289A1 as it will be implemented by the robot 270A." in par. 0067 and “Also illustrated in FIG. 2A is a virtual button 282A2, which can be selected by the operator to use a predefined path that was “saved” by the operator after being defined for a previous instance of user interface inputs. Selecting the virtual button 282A2 can paste the predefined path into the virtual environment, along with an option for the user to modify the pre-defined path to adapt it for the particular object” in par. 0068)
a planning unit for determining control instructions enabling a loading robot to execute the loading operation to be carried out based on the sensor signal and on said further user input; and (see at least " At block 360, the system causes the robot to manipulate the object in accordance with the object manipulation parameter(s). The object manipulation parameters(s) can include those that are based on predicted object manipulation parameter(s) and/or those that are defined, based on the user interface input(s), independent of any predicted object manipulation parameter(s). In some implementations, the system provides, to the robot, the object manipulation parameter(s) and/or high-level commands that are based on the object manipulation parameter(s). In those implementations, a control system of the robot transforms the object manipulation parameter(s) and/or high-level commands to corresponding low-level actions, such as control command(s) issued to actuators of the robot." in par. 0089)
a control interface for activating and controlling the loading robot to execute the loading operation based on the control instructions. (see at least " For example, the robot can include a controller that translates high level commands into more specific control commands to provide to one or more actuators of the robot. The control commands can include one or more velocity control command(s) issued to actuator(s) of the robot at a corresponding instance, to control movement of the robot. For example, in controlling movement of the robot, velocity control commands can be issued to each of the actuators that control movement of an end effector of the robot." in par. 0089)
Lee does not appear to explicitly teach all of the following, but Han does teach:
a device for loading or unloading a sheet metal processing machine or a wood working machine (see at least " A third type is a device, which is a work cell item in which the robot interacts with a part described later through the end effector, and a reference point, an operation point, an approach point, etc. may be stored together as a parameter in the work cell DB of the master DB and/or the user DB. A CNC, a machine tool, a press, an air blower, etc. may be included in the device." in par. 0046 and “A fourth type is a part, which is a target workpiece of the robot, that is, a target that may be picked up or moved by the end effector, and the volume, height, etc. may be stored together as a parameter in the work cell DB of the master DB and/or the user DB. It also interacts with the feeder or the device and the operation of the end effector may be affected by the relevant parameter.” In par. 0047 and “Further, a display 800 and an input apparatus 900 may be further provided to provide an interface between the user and the robot programming apparatus. The display 800 may be an LCD monitor connected with a computer, a teaching pendant screen, or the screen of a mobile device such as a smartphone, and the input apparatus 900 may be a keyboard, a mouse, a touch screen, etc. Further, like the screen of the touch screen, the display 800 and the input apparatus 900 may be configured together in a single apparatus.” In par. 0056)
a sensor interface for receiving a sensor signal comprising information on workpieces present in a loading area of the sheet metal processing machine or wood working machine; (see at least "A fifth type is a sensor, which is work cell items that may monitor the operation of the robot, and affect an operation of the robot due to its monitoring result. A proximity sensor, a vision sensor, etc. may be included here." in par. 0048)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device taught by Lee to incorporate the teachings of Han wherein a work cell with a robot feeding parts to a CNC or press machine is configured with a display for the user to input parameters for the robot, parts to be supplied, or the CNC/press, in order to arrive at using the remote client assistance system taught by Lee with a robot feeding parts in for machining or pressing. The motivation to incorporate the teachings of Han would be to make it easier for a machine operator to create or adjust the robot programming (see par. 0062)
Regarding Claim 2, Lee as modified by Han (references to Lee) teaches:
the device as claimed in claim 1,
wherein the sensor interface is designed for receiving a sensor signal related to a color image and to a depth image of the workpieces present in the loading area. (see at least " The additional vision component 194 can be, for example, a monographic camera (e.g., generating 2D RGB images), a stereographic camera (e.g., generating 2.5D RGB images), a laser scanner (e.g., generating 2.5D “point clouds”), and can be operatively connected to one or more systems (e.g., the system 110) disclosed herein." in par. 0036)
Regarding Claim 3, Lee as modified by Han (references to Lee) teaches:
the device as claimed in claim 1,
wherein the representation unit is designed for creating an image representation. (see at least " FIG. 2D illustrates a visual representation that includes an object representation 292D, of the spatula 192C of FIG. 1A, that is a 2D image (e.g., RGB image) of the spatula. The visual representation can be rendered, for example, on a touchscreen of a remote client device. The operator of the client device is prompted, by indication 282D, to swipe on the touchscreen to define an antipodal grasp. In response, the operator touches the touchscreen at 289D1 and swipes over to 289D2, at which point the operator releases his/her touch. As a result, an antipodal grasp is defined with a first point at 289D1 and a second point at 289D2. The points 289D1 and 289D2 can be transformed from 2D to 3D points using, for example, a mapping between the 2D image and corresponding 2.5D or 3D vision data." in par. 0071)
Regarding Claim 4, Lee as modified by Han (references to Lee) teaches:
the device as claimed in claim 1,
wherein the user interface comprises a display. (see at least " The visual representation engine 114 receives vision data from vision components 194, 174A, and/or 174B, and generates a visual representation to transmit to the remote client device 130 for rendering by the display engine 132 of the remote client device 130." in par. 0048)
Regarding Claim 5, Lee as modified by Han (references to Lee) teaches:
the device as claimed in claim 1,
wherein the user interface is designed for receiving a further user input comprising at least one of
a position of a grasping point for the loading robot on the workpiece; a depositing position of the workpiece; and/or an orientation of the workpiece when it is being deposited. (see at least " The input engine(s) 134 of remote client device 130 process user interface input(s), provided via the input device(s) 136, to generate data that indicates (directly or indirectly) one or more object manipulation parameter(s) to be used in an object manipulation. For example, the object manipulation parameter(s) indicated by the data generated by the input engine(s) 134 of an instance of user interface input(s) can include: a grasp pose; a placement pose; a sequence of waypoint(s) to encounter in traversing to a grasp pose; a sequence of waypoints to encounter in traversing toward a placement pose (after grasping the object); a full path or trajectory (i.e., a path with velocity, acceleration, jerk, and/or other parameter(s)) in traversing to and/or from a manipulation pose (e.g., a grasp pose or other manipulation pose); and/or other object manipulation parameter(s). " in par. 0054 and “the training instance input would include the vision data and the placement pose (e.g., a representation of x, y, and z positions, as well as orientation)” in par. 0097)
Regarding Claim 6, Lee as modified by Han (references to Lee) teaches:
the device as claimed in claim 1, wherein the evaluation unit is designed for at least one of
determining a surface plane of a workpiece based on a segmentation and based on a user input and determining an orthographic view. (see at least "For example, an object can be represented by one or more bounding boxes and/or other bounding shapes that approximate surfaces of the objects. For instance, an object can be defined by a plurality of connected bounding boxes, each of which can be defined by a center point, a height dimension, and a width dimension—which comprises significantly less data than a representation that defines color, texture, and/or depth for each pixel or voxel that corresponds to a surface of the object." in par. 0022 and “For instance, the 3D representation of the spatula 192 can define positions (e.g., x, y, z position) for one or more points on a surface of the spatula, and can optionally include one or more color values for each of the positions. Examples of 3D representations of the spatula 192 are illustrated in FIGS. 2A, 2B, and 2C, described in more detail below.” In par. 0049)
Regarding Claim 7, Lee as modified by Han (references to Lee) teaches:
the device as claimed in claim 1,
wherein the planning unit is designed for determining control instructions comprising grasping coordinates and depositing coordinates of the workpiece. (see at least " The input engine(s) 134 of remote client device 130 process user interface input(s), provided via the input device(s) 136, to generate data that indicates (directly or indirectly) one or more object manipulation parameter(s) to be used in an object manipulation. For example, the object manipulation parameter(s) indicated by the data generated by the input engine(s) 134 of an instance of user interface input(s) can include: a grasp pose; a placement pose; a sequence of waypoint(s) to encounter in traversing to a grasp pose; a sequence of waypoints to encounter in traversing toward a placement pose (after grasping the object); a full path or trajectory (i.e., a path with velocity, acceleration, jerk, and/or other parameter(s)) in traversing to and/or from a manipulation pose (e.g., a grasp pose or other manipulation pose); and/or other object manipulation parameter(s). " in par. 0054 and “the training instance input would include the vision data and the placement pose (e.g., a representation of x, y, and z positions, as well as orientation)” in par. 0097)
Regarding Claim 8, Lee as modified by Han (references to Lee) teaches:
the device as claimed in claim 1, wherein the planning unit is designed for determining the control instructions based on at least one of
predefined path default data for the movement of the loading robot related to the loading area; and (see at least " Also illustrated in FIG. 2A is a virtual button 282A2, which can be selected by the operator to use a predefined path that was “saved” by the operator after being defined for a previous instance of user interface inputs. Selecting the virtual button 282A2 can paste the predefined path into the virtual environment, along with an option for the user to modify the pre-defined path to adapt it for the particular object." in par. 0068)
predefined sensor position data for the position of the sensor related to the loading area. (see at least " When the object representation is based on vision data from the additional vision component 194, the pose of the object relative to the robot simulation and/or robot workspace simulation can optionally be determined using a transformation between a pose of the additional vision component 194 and a pose of a corresponding one of the robotic vision components 174A, 174B. " in par. 0051)
Regarding Claim 11, Lee as modified by Han (references to Lee) teaches:
A system for loading or unloading a sheet metal processing machine or a wood working machine (see at least "system 110" in par. 0036) , including:
a device as claimed in claim 1 (see Claim 1 analysis);
a sensor for covering the loading area (see at least cameras in par. 0036 ) ; and
a loading robot for executing the loading operation based on the control instructions. (see at least robots in par. 0036)
Regarding Claim 12, Lee as modified by Han teaches:
A system for implementing the device of Claim 2 (see Claim 2 analysis for rejection of the device)
Regarding Claim 14, Lee as modified by Han also teaches:
A method for implementing the device of Claim 1 (see Claim 1 analysis for rejection of the device)
Regarding Claim 15, Lee as modified by Han (references to Lee) also teaches:
A computer program product including program code for carrying out the steps of the method as claimed in claim 14 when the program code is being executed on a computer. (see at least " These software modules are generally executed by processor 814 alone or in combination with other processors. Memory 825 used in the storage subsystem 824 can include a number of memories including a main random access memory (RAM) 830 for storage of instructions and data during program execution and a read only memory (ROM) 832 in which fixed instructions are stored. A file storage subsystem 826 can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. " in par. 0127 )
Regarding Claim 16, Lee as modified by Han (references to Lee) teaches:
the device as claimed in claim 8,
wherein at least one of the path default data and the sensor position data have been determined in the course of a calibration process. (see at least " In some implementations, poses(s) of the vision component(s) in the first area and pose(s) of the robot vision component(s) are known, enabling determination of a transformation between a reference frame of the vision component(s) in the first area and a robot frame of the robot vision component(s). Using this transformation enables the inputs at the remote client device to be defined directly in the robot frame, or to be defined initially in the first frame and then transformed to the robot frame." in par. 0019)
Regarding Claim 17, Lee as modified by Han (references to Lee) teaches:
the device as claimed in claim 4,
wherein the user interface comprises a touchscreen display. (see at least " The user interface input(s) can be provided, for example, via a mouse, a touchscreen, VR hand controllers, and/or VR gloves." in par. 0004)
Regarding Claim 19, Lee as modified by Han (references to Lee) teaches:
the system as claimed in claim 11,
wherein the loading robot is an industrial robot having a manipulator arm. (see at least " Robots 170A and 1708 are each a “robot arm” having multiple degrees of freedom to enable traversal of a corresponding grasping end effector 172A, 172B along any of a plurality of potential paths to position the grasping end effector in desired locations." in par. 0038 )
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al (US 20210023711, hereinafter Lee) in view of Han et al (US 20200230817, hereinafter Han) and Burtscher et al (US 20230108679, hereinafter Burtscher)
Regarding Claim 9, Lee as modified by Han (references to Lee) teaches:
the device as claimed in claim 1,
Lee as modified by Han does not appear to explicitly teach all of the following, but Burtscher does teach:
wherein the user interface is designed for receiving a user input comprising a thickness of the workpieces. (see at least "The operator can make adjustments by means of a screen, in particular a touch-sensitive screen. In a first input window, for example, the user is prompted to enter the data of the workpiece, in particular information about the material thickness, external shape, size, weight, surface finish and/or formation of the edges. A choice can be made, for example, between the materials aluminum, steel or stainless steel and between a small, medium or large strength of the burrs." in par. 0015)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device taught by Lee as modified by Han to incorporate the teachings of Burtscher wherein the user can enter the thickness of a workpiece on a display of the system. The motivation to incorporate the teachings of Burtscher would be to make it easier for a machine operator to create or adjust the machine programming (see par. 0012)
Claim(s) 10, 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al (US 20210023711, hereinafter Lee) in view of Han et al (US 20200230817, hereinafter Han) and Nusser et al (US 20170021502, hereinafter Nusser)
Regarding Claim 10, Lee as modified by Han (references to Lee) teaches:
the device as claimed in claim 1, wherein
Lee as modified by Han does not appear to explicitly teach all of the following, but Nusser does teach:
the user interface is designed for receiving a user input comprising information on a position of workpieces that are located in the representation in an uppermost one of a plurality of layers of workpieces; (see at least " Furthermore, the GUI 700 includes movable graphical elements 702, 704, and 706 overlaying the image of region 404, which take the form of arrows with respective numbers. As shown in the “Pick Order/Direction” portion of the GUI, the arrows correspond to (i) the order in which a particular box will be picked by ROBOT #1 and (ii) where on the particular box ROBOT #1 should grip the particular box. Accordingly, in the scenario illustrated in FIG. 7, the human user may “click and drag” each arrow to a particular box and adjust an orientation of each arrow based on which surface of the particular box the human user believes ROBOT #1 should grip the particular box. For instance, as shown, arrow 702 is placed so as to indicate to the control system 400 that the leftmost detected box should be picked up and removed off the stack first and gripped on its top surface, arrow 704 is placed so as to indicate to the control system 400 that the topmost leaning box should be picked up and removed off the stack second and gripped on its left surface, and arrow 706 is placed so as to indicate to the control system 400 that the larger leaning box should be picked up and removed off the stack third and gripped on its top surface." in par. 0147 and “Furthermore, by touching the touchscreen, or by other methods of user input, the human user can determine (i) which box ROBOT #2 should load next and (ii) where in the truck ROBOT #2 should place the box. For instance, the human user can touch along path 806 to move the visual indication 804 of the upcoming small box to a more desirable location where the box might better fit in the stack.” In par. 0150)
the planning unit is designed for recognizing an offset of a workpiece located in a second layer, subsequent to a loading operation of a workpiece placed in a first layer; (see at least " Each box may have three different side lengths (e.g., a height, width, and depth), and the “box dimension constraints” refer to lengths of two of these sides that are included in a given box hypothesis. " in par. 0084) and
the planning unit is designed for determining the control instructions on the basis of the offset. (see at least " A control system may then determine that box 222 is the next box to pick, possibly based on its shape and size, its position on top of the stack of boxes 220, and/or based on characteristics of a target container or location for the boxes. The robotic arm 102 may then be controlled to pick up the box 222 using gripper 104 and place the box 222 onto the conveyer belt 110 (e.g., to transport box 222 into a storage area)." in par. 0056 )
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device taught by Lee as modified by Han to incorporate the teachings of Nusser wherein the user provides feedback on gripping positions and placement positions on a GUI for robots stacking or unstacking boxes, in order to arrive at performing the same feedback for a robot manipulating stacked wood or sheet metal materials. The motivation to incorporate the teachings of Nusser would be to more efficiently receive and use human feedback for manipulating stacked objects (see par. 0120)
Regarding Claim 13, Lee as modified by Han (references to Lee) teaches:
the system as claimed in claim 11,
Lee as modified by Han does not appear to explicitly teach all of the following, but Nusser does teach:
wherein the loading robot is designed for executing a loading operation and the sensor is designed for covering a loading zone (see at least "Still further, the GUI includes a “Robot Status” portion, which identifies ROBOT #1 and other robotic devices (e.g., “ROBOT #2”) that may be operating in the same workplace or other workplace, identifies what state each robotic device is currently in (e.g., ROBOT #1 and ROBOT #2 are each scanning the environment, but other options may include “Picking” or “En route,” among other possibilities), and identifies how many total boxes each robotic device has correctly manipulated." in par. 0141 and “causing ROBOT #1 to pick up and move the boxes of region 406 after causing ROBOT #1 to pick up and move the boxes of region 404” in par. 0145) , including:
another loading robot for carrying out an unloading operation (see at least " As a more specific example, if the control system expects that it will need to cause the robotic manipulator to unload detected boxes on the top of a pallet before unloading detected boxes on the bottom of the pallet, the control system may assign a higher priority level to the task of detecting boxes on the top of the pallet and may assign a lower priority level to the task of detecting boxes on the bottom of the pallet." in par. 0114 and "Still further, the GUI includes a “Robot Status” portion, which identifies ROBOT #1 and other robotic devices (e.g., “ROBOT #2”) that may be operating in the same workplace or other workplace, identifies what state each robotic device is currently in (e.g., ROBOT #1 and ROBOT #2 are each scanning the environment, but other options may include “Picking” or “En route,” among other possibilities), and identifies how many total boxes each robotic device has correctly manipulated." in par. 0141) ; and
another sensor for covering an unloading zone and for providing a sensor signal comprising information on workpieces present in the unloading zone (see at least "In further examples, scans from one or more 2D or 3D sensors mounted on a mobile base, such as a front navigation sensor 116 and a rear navigation sensor 118, and one or more sensors mounted on a robotic arm, such as sensor 106 and sensor 108, may be integrated to build up a digital model of the environment, including the sides, floor, ceiling, and/or front wall of a truck or other container. Using this information, the control system 140 may cause the mobile base to navigate into a position for unloading or loading objects, for instance." in par. 0041) , wherein
the planning unit is designed for determining control instructions enabling the other loading robot to execute an unloading operation to be carried out based on the sensor signal, on said further sensor signal, and on said further user input (see at least " Furthermore, the GUI 700 includes movable graphical elements 702, 704, and 706 overlaying the image of region 404, which take the form of arrows with respective numbers. As shown in the “Pick Order/Direction” portion of the GUI, the arrows correspond to (i) the order in which a particular box will be picked by ROBOT #1 and (ii) where on the particular box ROBOT #1 should grip the particular box. Accordingly, in the scenario illustrated in FIG. 7, the human user may “click and drag” each arrow to a particular box and adjust an orientation of each arrow based on which surface of the particular box the human user believes ROBOT #1 should grip the particular box. For instance, as shown, arrow 702 is placed so as to indicate to the control system 400 that the leftmost detected box should be picked up and removed off the stack first and gripped on its top surface, arrow 704 is placed so as to indicate to the control system 400 that the topmost leaning box should be picked up and removed off the stack second and gripped on its left surface, and arrow 706 is placed so as to indicate to the control system 400 that the larger leaning box should be picked up and removed off the stack third and gripped on its top surface." in par. 0147 and “Furthermore, by touching the touchscreen, or by other methods of user input, the human user can determine (i) which box ROBOT #2 should load next and (ii) where in the truck ROBOT #2 should place the box. For instance, the human user can touch along path 806 to move the visual indication 804 of the upcoming small box to a more desirable location where the box might better fit in the stack.” In par. 0150); and
the control interface is designed for activating and controlling the other loading robot to execute the unloading operation based on the control instructions. (see at least " Furthermore, the GUI 700 includes movable graphical elements 702, 704, and 706 overlaying the image of region 404, which take the form of arrows with respective numbers. As shown in the “Pick Order/Direction” portion of the GUI, the arrows correspond to (i) the order in which a particular box will be picked by ROBOT #1 and (ii) where on the particular box ROBOT #1 should grip the particular box. Accordingly, in the scenario illustrated in FIG. 7, the human user may “click and drag” each arrow to a particular box and adjust an orientation of each arrow based on which surface of the particular box the human user believes ROBOT #1 should grip the particular box. For instance, as shown, arrow 702 is placed so as to indicate to the control system 400 that the leftmost detected box should be picked up and removed off the stack first and gripped on its top surface, arrow 704 is placed so as to indicate to the control system 400 that the topmost leaning box should be picked up and removed off the stack second and gripped on its left surface, and arrow 706 is placed so as to indicate to the control system 400 that the larger leaning box should be picked up and removed off the stack third and gripped on its top surface." in par. 0147 and “Furthermore, by touching the touchscreen, or by other methods of user input, the human user can determine (i) which box ROBOT #2 should load next and (ii) where in the truck ROBOT #2 should place the box. For instance, the human user can touch along path 806 to move the visual indication 804 of the upcoming small box to a more desirable location where the box might better fit in the stack.” In par. 0150)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device taught by Lee as modified by Han to incorporate the teachings of Nusser wherein the user provides feedback on gripping positions and placement positions on a GUI for multiple robots stacking and unstacking boxes, in order to arrive at performing the same feedback for a robot manipulating stacked wood or sheet metal materials. The motivation to incorporate the teachings of Nusser would be to more efficiently receive and use human feedback for manipulating stacked objects (see par. 0120).
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al (US 20210023711, hereinafter Lee) in view of Han et al (US 20200230817, hereinafter Han) and Terasawa et al (US 20230347509, hereinafter Terasawa)
Regarding Claim 18, Lee as modified by Han teaches:
the device as claimed in claim 6,
Lee as modified by Han does not appear to explicitly teach all of the following, but Terasawa does teach:
wherein the evaluation unit is designed for determining a surface plane of a workpiece based on a segmentation and based on a user input by employing a RANSAC algorithm. (see at least “For example, if there is a point cloud corresponding to an object on a support plane (table or the like) of the object to be gripped, the point cloud corresponding to the support plane (table) is removed.” In par. 0201 and “Point clouds are divided into clusters in units of objects, and thereafter, a point cloud constituted by clusters including the largest number of clusters in the rectangular region, which is the specified region of the object to be gripped set by the user, is extracted as the point cloud corresponding to the object to be gripped. The other point cloud clusters are point clouds of objects other than the object to be gripped, and thus, are deleted.” In par. 0203 and "Note that an existing technique, such as a RANSAC technique, can be applied to a process of detecting a support plane such as a table on which the object to be gripped is placed." in par. 0205)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device taught by Lee as modified by Han to incorporate the teachings of Terasawa wherein user input and a RANSAC algorithm is used to assist the robot in distinguishing the surfaces of the object to be grasped from a surface like a table that the object is placed on. The motivation to incorporate the teachings of Terasawa would be to more effectively identify the object to be grasped (see par. 0202)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN M KATZ whose telephone number is (571)272-2776. The examiner can normally be reached Mon-Thurs. 8:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on (571) 270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN M KATZ/Primary Examiner, Art Unit 3657
Prosecution Insights