DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
This office action is in response to amendments filed 03/11/2026. Claims 1-19 are pending.
Applicant’s arguments and amendments to the claims with respect to rejections of Claims 15, 19 under 35 USC 101 have been fully considered and are persuasive. The rejections of Claims 15, 19 under 35 USC 101 have been withdrawn.
Applicant’s arguments and amendments to the claims with respect to rejections of Claims 3 under 35 USC 112(b) have been fully considered and are persuasive. The rejections of Claims 3 under 35 USC 112(b) have been withdrawn.
Applicant’s arguments and amendments to the claims with respect to prior art rejections of Claims 1-7, 13-15, 17 under 35 USC 103 have been fully considered and are persuasive. The rejections of Claims 1-7, 13-15, 17 under 35 USC 103 have been withdrawn. However, upon further consideration, a new rejection is made in view of Simon et al (US 20210114826, hereinafter Simon)
Applicant’s arguments and amendments to the claims with respect to prior art rejections of Claims 8-12, 16, 18-19 under 35 USC 103 have been fully considered but are not persuasive.
With respect to applicant’s arguments that Watts fails to teach an apparatus for picking a layer of multiple items simultaneously, examiner agrees, but Eckman does teach this limitation. Eckman teaches a forklift with cameras attached the frame of the layer picker for recognizing the layers on a pallet and picking them (see par. 0149) that reads on the limitation.
With respect to applicant’s arguments that Watts fails to teach a analyzing, using a machine-learning model, the received image data and determining, using the machine-learning model, that an obstacle may cause a malfunction of the handling process, examiner respectfully disagrees.
Watts uses a computer model with algorithms that learn from human feedback to make a 3D reconstruction of stacked boxes on a pallet (see col. 12 and 22 cited below). Applicant has not limited the model to be any specific type of machine-learning model, so the model taught by Watts falls within the scope of the claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4-7, 13, 15, 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Watts et al (US 9802317, hereinafter Watts) in view of Simon et al (US 20210114826, hereinafter Simon).
Regarding Claim 1, Watts teaches:
a computer-implemented method of operating a palletizer (see at least "In further examples, a 3D model of a stack of boxes may be constructed and used as a model to help plan and track progress for loading/unloading boxes to/from a stack or pallet." in col. 11 lines 28-32 ) , comprising:
receiving image data, using at least one camera, of at least one item intended to be handled by the palletizer in a handling process (see at least “In some examples, the one or more images may include various facades of a stack of boxes (i.e., a near-planar group of boxes) from different viewpoints. For instance, when the physical objects are a stacked pallet of boxes in the physical environment, the images may include at least one side view facade of the stacked pallet of boxes, at least one top-down view facade of the stacked pallet of boxes, and/or other variations on these viewpoints (i.e., perspective views). In some implementations, one frontal view or one top view of stacked boxes may be used.” In col. 12 lines 23-32 ) ;
analyzing, using a machine-learning model, the received image data (see at least “After receiving the one or more images, the computing device or system may virtually reconstruct the physical environment based on the one or more images, a pose of the sensor(s) used to capture the images, and in some scenarios, additionally or alternatively based on other information obtained by sensors such as depth and visual cameras. A reconstruction of the environment may facilitate operations relating to object segmentation, including but not limited to operations described herein” in col. 12 lines 33-42 and "For instance, a predetermined confidence threshold may dynamically decrease as the control system has been trained to identify virtual boundary lines with higher confidence and greater precision and accuracy, based on repeated interaction between the control system and the remote assistor devices. " in col. 22 lines 41-46) ;
determining, using the machine-learning model, that an obstacle may cause a malfunction of the handling process (see at least “This may be advantageous in that remote assistor feedback can enable the control system to learn from its mistakes and/or receive acknowledgement for its correct perception of objects in a timely, efficient manner.” In col. 6 lines 29-33 and "In other examples, the 3D model may be used for collision avoidance. Within examples, planning a collision-free trajectory may involve determining the 3D location of objects and surfaces in the environment. A trajectory optimizer may make use of the 3D information provided by environment reconstruction to optimize paths in the presence of obstacles." in col. 11 lines 38-42 ) ; and
Watts does not appear to explicitly teach all of the following (Watts does not appear to explicitly teach that the robot is stopped while the user assistance is requested and received), but Simon does teach:
a laver picker apparatus, which is operable to lift a layer of multiple items simultaneously (see at least "The palletizer cell 10 has one or more robotic case manipulator(s) (also referred to herein as articulated robots, adaptive real time robots, robots, or product picking apparatus) that place (individually or manufactured pickfaces) mixed pallet load article units CU (also referred to herein as case units or cases or products 18) in stacks SL1-Sn and/or layers PL1-PL4 building a mixed case pallet load PAL with vision system assistance." in par. 0027)
generating a control signal for instructing the layer picker apparatus to stop the handling process (see at least “The visions system 310 is configured to detect the unexpected object, the sides/edges of the pallet load PAL, distances between the pallet load PAL and the robot exclusion zones, etc. and send data signals to the cell controller 10C so that the cell controller 10C commands the robot arm 12 to move the layer depalletizing tool 99 around the unexpected object, around the sides of the pallet load, between the pallet load PAL and the robot exclusion zone (or other obstacle adjacent the pallet load PAL), and/or in any other suitable manner for picking the pallet layer pL1, PL2, PL3, PL4, PL5; or in other aspects commands the robot arm 12 to stop moving." in par. 0062) .
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Watts to incorporate the teachings of Simon wherein the layer picking robot stops when an unexpected obstacle is detected. The motivation to incorporate the teachings of Simon would be to avoid products being damaged (see par. 0090)
Regarding Claim 2, Watts as modified by Simon teaches:
the method of claim 1,
Watts does not appear to explicitly teach all of the following (Watts does not appear to explicitly teach that the robot is stopped while the user assistance is requested and received), but Simon does teach:
wherein the control signal for instructing the layer picker apparatus to stop the handling process causes the layer picker apparatus to switch in a temporary stop mode that allows a seamless continuation of the handling process. (see at least “In one aspect, the cell controller 10C is configured to reject the pallet layer pick if the pallet support variance PSV1, PSV2 exceeds thresholds from a predetermined reference such as a plane define by the top pad 28 (and send stop bot signal until replaced). For example, if the missing case units CU of the pallet layer are greater than a predetermined area or if the spacing between case units CU in the pallet layer is greater than a predetermined distance, the pallet layer pick is rejected and the pallet layer will not be picked until the defects in the pallet layer are resolved (such as by manual intervention). If the pallet layer is within the predetermined thresholds, the cell controller 10C is configured to resolve a pallet layer planar variance (e.g., the position of pallet layer in the three-dimensional robot space X, Y, Z, RX, RY, RZ) and confirm or modify (compensates) planned robot pick/place path based on the above-noted variances for adaptive pose of the layer depalletizing tool 99 with higher resultant pallet layer pick probability." in par. 0072) .
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Watts as modified by Simon to incorporate the teachings of Simon wherein the robot is stopped temporarily until an issue with the layer to be picked can be resolved. The motivation to incorporate the teachings of Simon would be to increase the chances that the layer pick is successfully completed (see par. 0072)
Regarding Claim 4, Watts as modified by Simon teaches:
the method of one of claim 1,
Watts does not appear to explicitly teach all of the following, but Simon does teach:
wherein the step of determining that an obstacle may cause a malfunction of the handling process comprises detecting a slip sheet on top of the at least one item intended to be handled in a handling process, the slip sheet being the obstacle. (see at least "In accordance with aspects of the present disclosure, a layer depalletizing tool is provided with a component that retrieves a slip sheet 277 located over the pallet layer 816 of products 18 at the same time the pallet layer 816 is depalletized." in par. 0099 and “The tool includes top pad 28 including suction cups (not shown) to remove any slip sheet over the pallet layer 816 at the same time the pallet layer 816 is depalletized. When the pallet layer 816 is completely gripped by the layer depalletizing tool (FIG. 17, Block 17222), the suction cups are activated (FIG. 17, Block 17224). Another, e.g., slip sheet sensor 999 (e.g., such as a camera or scanner configured for resolution of case edges of intermediate cases as shown in FIGS. 9A and 9B and in particular FIG. 9B), which is facing upwards towards the gripped pallet layer 816 (See also FIG. 9B) that is gripped and lifted by the tool, or the same referred to hereinabove, is used to determine if there is a slip sheet 277 attached to, for example, the top pad 28. As the robot 14 lifts and transfers the pallet layer 816, the top pad 28 is slightly lifted (FIG. 17, Block 17226). If the slip sheet sensor 999 still detects the presence of an object (FIG. 17, Block 17228), this means that there is a slip sheet 277 under the pallet layer 816 being depalletized (FIG. 17, Block 17229) and the vacuum is kept on the suction cups (FIG. 17, Block 17230). With further reference to FIGS. 19A, 19B, and 19C, the robot 17 then places the pallet layer 816 (with the slip sheet 277 underneath the pallet layer) onto the mat top conveyor (such as outfeed conveyor 150) and slip sheet is removed and discarded in a bin (FIG. 17, Block 17236) after the pallet layer 816 is placed on the outfeed conveyor 150 (FIG. 17, Block 17232) or somewhere else. In the contrary, if the slip sheet sensor 999 does not detect anything, this means that there is no slip sheet under the layer being depalletized. When such is the case, the vacuum is removed from the suction cups and the robot 14 directly moves back to pick the next pallet layer 816 on the pallet once the previous pallet layer 816 is placed on the outfeed conveyor 150.” In par. 0100 )
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Watts to incorporate the teachings of Simon wherein a slip sheet sensor detects slip sheets on top of or underneath a layer being depalletized. The motivation to incorporate the teachings of Simon would be to reduce cost by removing the need for an independent slip sheet removal device (see par. 0099)
Regarding Claim 5, Watts as modified by Simon teaches:
the method of one of claim 1,
Watts further teaches: wherein the image data comprises images recorded by at least two cameras that are synchronized with respect to the time at which the image data is recorded. (see at least " In further examples, a virtual environment may be built up using a 3D volumetric or surface model to integrate information (e.g., from different sensors). This may allow the system to operate within a larger environment, such as in cases where one sensor may be insufficient to cover a large environment. Such techniques may also increase the level of detail captured, which may help the robotic device perform various tasks. In particular, integrating information can yield finer detail than from a single scan alone (e.g., by bringing down noise levels). This may enable better object detection, surface picking, or other applications." in col. 10 lines 16-26 and “The images of the objects may be captured by one or more sensors in sequence at various poses (i.e., positions and orientations) of the sensor(s) and/or of the objects themselves. Thus, respective images may correspond to respective different views of the objects. The images may be of various types, such as color/intensity images, gradient images, and depth map images (i.e., images representative of distances between respective surfaces of the objects and a reference plane associated with a perspective of the sensor(s) that capture the images), among other possibilities.” In col. 12 lines 8-17 )
Regarding Claim 6, Watts as modified by Simon (references to Watts) teaches:
the method of claim 5, wherein the step of determining that an obstacle may cause a malfunction of the handling process comprises:
generating sub-probabilities per camera that an obstacle may cause a malfunction of the handling process (see at least "As a further example aspect of volume reconstruction, the computing device may take into account a confidence in the depth reading. For instance, readings that are further away from the camera, or that strike a surface at a glancing angle, are discounted by the computing device. As such, more confident information may have more of an effect during integration, which may improve the quality of the environment reconstruction." in col. 13 lines 10-20 ) ; and
determining a combined probability that an obstacle may cause a malfunction of the handling process based on the sub-probabilities (see at least "This may result in a multi-modal gradient orientation image and a corresponding magnitude image. To compute an edge map, the computing device may implement different weighting of the modalities. After computation of the edge map, the computing device may compute all the line, corner, contour, and plane features mentioned above using the normal, color, depth, multi-modal orientation and magnitude images." in col. 15 lines 1-8 and “A confidence score may indicate a level of confidence in whether the virtual boundary line correctly distinguishes the objects in the model, and may take the form of a number value (e.g., one through ten, 0% through 100%, etc.) or another type of data representation, either visual or non-visual. If the control system determines that a confidence score associated to the virtual boundary line is lower than a predetermined confidence threshold, the control system may determine that it should request remote assistance with an identification of the virtual boundary line.” In col. 22 lines 11-21 ) .
Regarding Claim 7, Watts as modified by Simon (references to Watts) teaches
the method of claim 6,
wherein determining a combined probability comprises one or both of averaging and weighting of the generated sub-probabilities (see at least "As a further example aspect of volume reconstruction, the computing device may take into account a confidence in the depth reading. For instance, readings that are further away from the camera, or that strike a surface at a glancing angle, are discounted by the computing device. As such, more confident information may have more of an effect during integration, which may improve the quality of the environment reconstruction." in col. 13 lines 10-20 and "This may result in a multi-modal gradient orientation image and a corresponding magnitude image. To compute an edge map, the computing device may implement different weighting of the modalities. After computation of the edge map, the computing device may compute all the line, corner, contour, and plane features mentioned above using the normal, color, depth, multi-modal orientation and magnitude images." in col. 15 lines 1-8).
Regarding Claim 13, Watts as modified by Simon (references to Watts) teaches:
A data processing apparatus (see at least " Referring again to the example system of FIG. 4, the control system 400 may send, to a remote assistor device 412, a request for remote assistance with verifying one or more virtual boundary lines in order to facilitate distinguishing at least one box in the region 404 of the model 402 that the control system is interested in manipulating. " in col. 22 line 64 to col. 23 line 5) for carrying out the method of claim 1 (see Claim 1 analysis).
Regarding Claim 15, Watts as modified by Simon (references to Watts) teaches:
a computer program (see at least " In some implementations, memory 146 may contain instructions 144 (e.g., program logic) executable by the processor 142 to execute various operations of robotic device 100, including those described above in connection with FIGS. 1A-1B." in col. 9 lines 31-35 ) comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claim 1 (see Claim 1 analysis).
Regarding Claim 17, Watts as modified by Simon teaches:
a layer picker apparatus for carrying out the method of claim 1 (see Claim 1 analysis for rejection of the method).
Claim(s) 8-9, 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Watts et al (US 9802317, hereinafter Watts) in view of Eckman et al (US 20210133666, hereinafter Eckman).
Regarding Claim 8, Watts teaches
a computer-implemented method for generating training data for a machine-learning model for operating a palletizer, comprising:
receiving image data, using at least one camera, of at least one layer of items intended to be handled in a handling process; (see at least “After receiving the one or more images, the computing device or system may virtually reconstruct the physical environment based on the one or more images, a pose of the sensor(s) used to capture the images, and in some scenarios, additionally or alternatively based on other information obtained by sensors such as depth and visual cameras. A reconstruction of the environment may facilitate operations relating to object segmentation, including but not limited to operations described herein” in col. 12 lines 33-42 and "For instance, a predetermined confidence threshold may dynamically decrease as the control system has been trained to identify virtual boundary lines with higher confidence and greater precision and accuracy, based on repeated interaction between the control system and the remote assistor devices. " in col. 22 lines 41-46)
receiving user input from a user including whether an obstacle may cause a malfunction and generating a control signal to the palletizer; (see at least “In additional examples, planes or other mathematical surfaces in the environment may be extracted in 3D. These known “ideal” surface detections may be combined into a more accurate model of the environment. For instance, planes may be used to determine the full extents of walls (or mathematical description thereof) and other obstacles to avoid collisions and detect the locations of objects of interest.” in col. 10 line 35-45 and “The information indicative of how to perform the task may include information representative of any human user feedback determined by the remote assistor device based on human user input, such as the modification to the virtual boundary line. Within other examples, the human user feedback may include (i) a user-determined box hypothesis, (ii) a user-determined sequenced order in which to manipulate the at least one object of the region of the model, (iii) a user-determined location on each of the objects where the robotic manipulator should manipulate each object of the region, (iv) an indication of user acceptance of a control system-identified virtual boundary line, (v) an indication of user rejection of a control system-identified virtual boundary line, (vi) an indication of user acceptance of a control system-determined segmentation of the region, (vii) an indication of user rejection of a control system-determined segmentation of the region, and/or (viii) an instruction for the control system to rescan the objects in the region (or all the objects in the environment), among other types of labels/information associated to the objects that the control system is interested in.” in col. 24 lines 26-46 and "In practice, the control system may communicate back and forth with the remote assistor device at least two times in this manner before ultimately instructing the robotic manipulator to perform the task, in order to optimize the control system learning from the feedback received from the remote assistor device." in col. 25 line 65 to col. 26 line 4 ) and
Watts does not appear to explicitly teach all of the following, but Eckman does teach:
a layer picker apparatus, which is operable to lift a layer of multiple items simultaneously (see at least " In some configurations (not depicted), one or more cameras can be mounted around a frame of a layer picker on the forklift 1008. As the layer picker moves up and down, for example, the cameras can scan one or more pallets and take images of the pallets from various angles. The batch of images can be communicated to the server as previously described, to determine parameters associated with each of the pallets stacked on the layer picker." in par. 0149)
generating a training dataset comprising the user input and the image data (see at least " Once all the modules complete identifying the parameters from the visual information in the batch of images and/or the 3D point cloud structure of the pallet with high confidence values, for example, the identified parameters can be stored as processed data in the pallet information database (step 518). As previously discussed, the identified parameters can be stored in a profile that is associated with the scanned pallet. Once the identified parameters are stored in associated pallet profiles in the profile information database, for example, the processed data (e.g., identified parameters) can be outputted to the GUI of the user's computing device in step 520. The user can make any necessary adjustments to the identified parameters, such as when a parameter has a low confidence value, as previously discussed. User inputs can be stored by the computer server in the pallet profile and used to replace any parameter identifications made by the modules, whether or not those modules had low confidence values for their determinations. In some implementations, the server can store the user input along with the modules' identified parameters in order to enhance the training models used for each of the modules. The training models can be enhanced based upon a comparison of the user input and the modules' parameter determinations, for example. Improving the training models this way can ensure that future parameter determinations for different pallets are accurate and have high confidence values." in par. 0105).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Watts to incorporate the teachings of Eckman wherein the forklift picks entire layers and uses images from attached cameras to add to a training dataset that is built up consisting of pallet images and user inputted corrections. The motivation to incorporate the teachings of Eckman would be to increase the accuracy of the vision model recognizing items on the pallet (see par. 0105)
Regarding Claim 9, Watts as modified by Eckman teaches:
the method of claim 8,
Watts further teaches: wherein the control signal of the user input comprises at least one of the following:
an instruction to stop the handling process;
an instruction to wait for a predetermined time before continuing the handling process;
an instruction to reset the palletizer (see at least "providing the human user with a selectable option to cause the remote assistor device to send instructions to the control system to rescan the model if the virtual boundary line is incorrect." in col. 6 lines 13-15) ;
an instruction to switch the layer picker apparatus into a manual control mode allowing the user to manually control the layer picker apparatus ; and
opening a safety barrier surrounding the layer picker apparatus.
Watts does not appear to explicitly teach all of the following, but Eckman does teach:
A layer picker apparatus (see at least " In some configurations (not depicted), one or more cameras can be mounted around a frame of a layer picker on the forklift 1008. As the layer picker moves up and down, for example, the cameras can scan one or more pallets and take images of the pallets from various angles. The batch of images can be communicated to the server as previously described, to determine parameters associated with each of the pallets stacked on the layer picker." in par. 0149)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Watts to incorporate the teachings of Eckman wherein the forklift picks entire layers. The motivation to incorporate the teachings of Eckman would be to increase the efficiency of moving items throughout a warehouse (see par. 0059)
Regarding Claim 11, Watts as modified by Eckman also teaches:
a machine-learning training dataset comprising image data of at least one item suitable for being handled by a layer picker apparatus in a handling process and information including a control signal suitable for controlling a layer picker apparatus obtained by a method for generating training data for a machine-learning model according to claim 8 (see Claim 8 analysis for rejection of the method).
Claim(s) 12, 16, 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Watts et al (US 9802317, hereinafter Watts) in view of Eckman et al (US 20210133666, hereinafter Eckman) and Chavez et al (US 20200273138, hereinafter Chavez)
Regarding Claim 12, Watts as modified by Eckman (references to Eckman) also teaches:
a computer-implemented method of training a machine-learning model for operating a layer picker apparatus, comprising:
transmitting a plurality of machine-learning training datasets according to claim 11 (see Claim 11 analysis) to a cloud-based machine-learning environment, wherein the plurality of machine-learning training datasets are associated with a plurality of layer picker apparatuses (see at least “FIG. 7 depicts an example computer system. A computer server 700, as previously described, a pallet profile device 730, and a user computing device 704 can communicate through a network wirelessly (e.g., BLUETOOTH, WIFI) and/or through Ethernet-based communication.” In par. 0112 and “The computer server 700 in the present example includes a machine learning model 708A and a geometric-based learning model 708B.” in par. 0113 and "For example, the disclosed systems for scanning pallets can include at least one of a stereoscopic, 3D, or 2D camera in addition to a thermal imaging camera, the at least one camera being mounted to a frame to capture pallet information. The frame can be placed around a conveyor belt or other mechanism that moves pallets in and out of a scanning area in the warehouse. The cameras on the frame can be configured to take images of a pallet as the pallet moves through/under the frame. The images can be batched and sent or transmitted to a backend or remote server. The remote server can include one or more modules (e.g., applications) that may be trained based on machine learning and/or geometric-based training models, and configured to process the batch of images and identify particular parameters associated with the pallet." in par. 0008 and “For example, a camera module 732 can control the one or more cameras that take pictures of the pallet as it moves through the warehouse (e.g., on the conveyor belt, on a forklift, as part of an automated warehouse system including automated pallet transporters, such as robots to transport pallets).” In par. 0126) ; and
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Watts to incorporate the teachings of Eckman wherein a training dataset is built up consisting of pallet images and user inputted corrections. The motivation to incorporate the teachings of Eckman would be to increase the accuracy of the vision model recognizing items on the pallet (see par. 0105)
Watts and Eckman do not appear to explicitly teach all of the following, but Chavez does teach:
receiving a trained machine-learning model in a binary format (see at least " Techniques are disclosed to configure a robotic system to process and merge sensor data from multiple sensors, such as multiple 3D cameras, to perform a robotic operation. In various embodiments, an administrative user interface, configuration file, application programming interface (API), or other interface may be used to identify sensors and define one or more processing pipelines to process and use sensor output to perform robotic operations. In various embodiments, pipelines may be defined by identifying processing modules and how the respective inputs and outputs of such modules should be linked to form a processing pipeline. In some embodiments, the definition is used to generate binary code to receive, process, and use sensor inputs to perform robotic operations. In other embodiments, the definition is used by a single, generic binary code that dynamically loads plugins to perform the processing." in par. 0048).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Watts as modified by Eckman to incorporate the teachings of Chavez wherein a vision pipeline for identifying objects in the workspace of a robot is compiled into binary code. The motivation to incorporate the teachings of Chavez would be able to make the code dynamic to adapt to different sensor conditions and maintain the quality and reliability of the vision pipeline (see par. 0054-0055)
Regarding Claim 16, Watts as modified by Eckman and Chavez teaches:
a data processing apparatus for carrying out the method of claim 12 (see Claim 12 analysis).
Regarding Claim 18, Watts as modified by Eckman and Chavez teaches:
a layer picker apparatus for carrying out the method of claim 12 (see Claim 12 analysis).
Regarding Claim 19, Watts as modified by Eckman and Chavez teaches:
a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claim 12 (see Claim 12 analysis).
Claim(s) 3, 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Watts et al (US 9802317, hereinafter Watts) in view of Simon et al (US 20210114826, hereinafter Simon) and Li et al (US 20200086483, hereinafter Li)
Regarding Claim 3, Watts as modified by Simon teaches:
the method of claim 1,
Watts further teaches: wherein the image data comprises a sequence of still images (see at least " The images of the objects may be captured by one or more sensors in sequence at various poses (i.e., positions and orientations) of the sensor(s) and/or of the objects themselves. Thus, respective images may correspond to respective different views of the objects. The images may be of various types, such as color/intensity images, gradient images, and depth map images (i.e., images representative of distances between respective surfaces of the objects and a reference plane associated with a perspective of the sensor(s) that capture the images), among other possibilities." in col. 12 lines 8-17 ) ,
Watts and Simon do not appear to explicitly teach all of the following, but Li does teach:
wherein the step of analyzing comprises selecting a subset of images and wherein the selected subset of images comprises images recorded at predetermined times one or both of before and after a start of a handling process (see at least " At block 404, the system determines a grasp success label for a grasp attempt based on stored data for the grasp attempt. For example, as described with respect to bock 314 of process 300, additional data may be stored for the grasp attempt to enable determination of a grasp success label for the grasp attempt. The stored data may include data from one or more sensors, where the data is generated during and/or after the grasp attempt." in par. 0085) , and
wherein the images comprise one or both of a unique identifier and a time stamp (see at least " For example, the system may store a current image generated by a vision sensor associated with the robot and associate the image with the current instance (e.g., with a timestamp). " in par. 0075 and “At block 406, the system selects an instance for the grasp attempt. For example, the system may select data associated with the instance based on a timestamp and/or other demarcation associated with the data that differentiates it from other instances of the grasp attempt” in par. 0090) .
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Watts as modified by Simon to incorporate the teachings of Li wherein a subset of images collected during and after a grasp attempt are stored with grasp success labels and timestamps in a training set. The motivation to incorporate the teachings of Li would be to increase grasp success rate and make learning more efficient (see par. 0106)
Regarding Claim 14, Watts as modified by Simon (references to Watts) teaches:
a layer picker apparatus comprising a data processor for carrying out the method of claim 1 (see Claim 1 analysis), the layer picker apparatus further comprising
at least four cameras that are synchronized with respect to the time at which the image data is recorded (see at least “In further examples, scans from one or more 2D or 3D sensors mounted on a mobile base, such as a front navigation sensor 116 and a rear navigation sensor 118, and one or more sensors mounted on a robotic arm, such as sensor 106 and sensor 108, may be integrated to build up a digital model of the environment, including the sides, floor, ceiling, and/or front wall of a truck or other container. Using this information, the control system 140 may cause the mobile base to navigate into a position for unloading or loading objects, for instance.” In col. 8 lines 16-25 and "In further examples, a virtual environment may be built up using a 3D volumetric or surface model to integrate information (e.g., from different sensors). This may allow the system to operate within a larger environment, such as in cases where one sensor may be insufficient to cover a large environment. Such techniques may also increase the level of detail captured, which may help the robotic device perform various tasks. In particular, integrating information can yield finer detail than from a single scan alone (e.g., by bringing down noise levels). This may enable better object detection, surface picking, or other applications." in col. 10 lines 16-26 and “The images of the objects may be captured by one or more sensors in sequence at various poses (i.e., positions and orientations) of the sensor(s) and/or of the objects themselves. Thus, respective images may correspond to respective different views of the objects. The images may be of various types, such as color/intensity images, gradient images, and depth map images (i.e., images representative of distances between respective surfaces of the objects and a reference plane associated with a perspective of the sensor(s) that capture the images), among other possibilities.” In col. 12 lines 8-17), and
Watts and Simon do not appear to explicitly teach all of the following, but Li does teach:
wherein the step of analyzing comprises selecting a subset of images being recorded at predetermined times that occur at one or both of before and after a start of the handling process. (see at least " At block 404, the system determines a grasp success label for a grasp attempt based on stored data for the grasp attempt. For example, as described with respect to bock 314 of process 300, additional data may be stored for the grasp attempt to enable determination of a grasp success label for the grasp attempt. The stored data may include data from one or more sensors, where the data is generated during and/or after the grasp attempt." in par. 0085 and " For example, the system may store a current image generated by a vision sensor associated with the robot and associate the image with the current instance (e.g., with a timestamp). " in par. 0075 and “At block 406, the system selects an instance for the grasp attempt. For example, the system may select data associated with the instance based on a timestamp and/or other demarcation associated with the data that differentiates it from other instances of the grasp attempt” in par. 0090) .
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Watts as modified by Simon to incorporate the teachings of Li wherein a subset of images collected during and after a grasp attempt are stored with grasp success labels and timestamps in a training set. The motivation to incorporate the teachings of Li would be to increase grasp success rate and make learning more efficient (see par. 0106)
Allowable Subject Matter
Claims 10 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The closest prior art comes from Watts and Eckman but the prior art does not appear to teach “if the user input comprises one or both of manually controlling the layer picker apparatus and opening the safety barrier surrounding the layer picker apparatus, the respective user input and image data is excluded from the training dataset.” in combination with all of the other limitations in the independent claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN M KATZ whose telephone number is (571)272-2776. The examiner can normally be reached Mon-Thurs. 8:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on (571) 270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN M KATZ/Examiner, Art Unit 3657
Prosecution Insights