Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to RCE
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/09/2026 has been entered.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 6, 8, 9, 12, 15, 16, 20, 21, 27, 30 are rejected under 35 U.S.C. 103 as being unpatentable by Claussen (US20220347853) in view of Skyum (US20230150777) and Ban (US20070274812) and Yano (US20200376678).
Regarding claim 1, Claussen teaches a processor, comprising:
one or more circuits to use one or more neural networks to cause one or more autonomous devices to place a first object in a location and orientation based, at least in part, on a first image of a current location and orientation and a goal image ([0012]-[0015] disclosing a processor with memory, it is interpreted that a processor comprises circuits. [0020] disclosing based on the optical flow. [0027]-[0028] disclosing a robotic gripping device “autonomous device” that places an object in a desired location and pose based on the mismatch between the images of current pose “location and orientation” and desired pose “location and orientation” using neural network).
Claussen in a further embodiment teaches based at least in part on an optical flow ([0020] disclosing the configuration error between the poses in each image of the first and second image can be converted to optical flow).
The combination/substitution of pixel flow as taught by Claussen is obvious yielding predictable results in order to accurately place objects in target positions based on a difference between the pixels thus accurately matching the final goal configuration of the object improving the placement of objects, and for verification, adjustment and refinement of poses.
Claussen does not teach the optical flow of pixels; a plurality of objects; and to cause the one or more autonomous devices to move a second object of the plurality of objects based at least in part on the goal image and a second image of a current location and orientation captured after the first object is moved by the one or more autonomous devices.
Skyum teaches a plurality of objects; and to cause the one or more autonomous devices to move a second object of the plurality of objects based at least in part on the goal image and a second image of a current location and orientation captured ([0040]-[0067] disclosing steps of moving at least a second object using the same technique of moving the objects based on image current location and orientation and image of target desired location and orientation).
It would have been obvious to one of ordinary skill in the art to have combined the teaching of Skyum and Claussen yielding predictable results in order to move a plurality of objects based on images, its also obvious to apply the method of Claussen to the plurality of objects of Skyum in order to facilitate the picking and placing of objects improving efficiency.
While Claussen as modified by Skyum does not disclose after the first object is moved by the one or more autonomous devices.
Ban in the same field of endeavor teaches after the first object is moved by the one or more autonomous devices ([0029]-[0030] disclosing to capture a new image of the workpieces by the camera).
It would be obvious to combine the teaching of Ban of a new image captured after moving a first object to the method of comparing images Claussen as modified by Skyum yielding predictable results and improving the accuracy since one of the second objects may have moved due to the picking of a first object thus a more accurate location and orientation of the second object would be obtained.
Yano teaches the optical flow of pixels ([0038]-[0045] disclosing the control of the robot to place an object based on a difference in pixels between a current image and a target image in order to align the pixels together through iterative control, i.e., pixel flow).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban to incorporate the teaching of Yano of the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on one or more optical flow estimates. The combination/substitution of the method of Yano is obvious yielding predictable results in order to align the target image with a current image yielding predictable results, and incorporating the pixel flow as taught by Yano improves the placement by ensuring the brightness of each pixel is aligned thus improving object placement and can be used for verification and refinement of the method of Claussen improving accuracy and reducing errors.
Regarding claim 2, Claussen as modified by Skyum and Ban and Yano teaches the processor of claim 1. Claussen as modified by Skyum and Ban and Yano further teaches wherein the one or more images comprise an image of a goal location and orientation of the plurality of objects, and the one or more circuits are to cause the one or more autonomous devices to place the first object in the location and orientation based, at least in part, on identifying a correspondence between pixels in one or more first images and pixels in the goal image
Specifically, Yano teaches wherein the one or more images comprise an image of a goal location and orientation of the plurality of objects, and the one or more circuits are to cause the one or more autonomous devices to place the first object in the location and orientation based, at least in part, on identifying a correspondence between pixels in one or more first images and pixels in the goal image ([0025] disclosing a robot gripping an object to place the object in a target position and attitude for the object. [0038]-[0040] disclosing a difference between the corresponding pixels between a current image of current location and orientation and a target image of target location and orientation and controlling the robot to move closer to the target location and orientation).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban and Yano to incorporate the teaching of Yano of wherein the one or more images comprise an image of a goal location and orientation of the one or more objects, and the one or more circuits are to cause the one or more autonomous devices to place the one or more objects in the location and orientation based, at least in part, on identifying a correspondence between pixels in one or more first images of a current location and orientation of the one or more objects and pixels in a second image of the goal location and orientation of the one or more objects in order to determine a pose of an object to place an object in a target location as taught by Yano [0038]-[0040]. The combination/substitution is obvious yielding predictable results in order to accurately place objects in target positions, and for refinement, verification and adjustment improving accuracy and reducing errors.
Regarding claim 6, Claussen as modified by Skyum and Ban and Yano teaches the processor of claim 1. Claussen as modified by Skyum and Ban and yano further teaches wherein the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on one or more optical flow estimates.
Specifically, Yano teaches wherein the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on one or more optical flow estimates ([0038]-[0045] disclosing the control of the robot to place an object based on a difference in pixels between a current image and a target image in order to align the pixels together through iterative control, i.e., pixel flow).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban to incorporate the teaching of Yano of the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on one or more optical flow estimates. The combination/substitution of the method of Yano is obvious yielding predictable results in order to align the target image with a current image yielding predictable results , and for refinement, verification and adjustment improving accuracy and reducing errors.
Regarding claim 8, Claussen as modified by Skyum and Ban and yano teaches the processor of claim 1. Claussen as modified by Skyum and Ban and Yano further teaches wherein one or more circuits are to identify a correspondence between pixels in a first current image and pixels in a second image and cause the one or more autonomous devices to place the one or more objects based, at least in part, on the correspondence.
Specifically, Yano teaches wherein one or more circuits are to identify a correspondence between pixels in a first current image and pixels in a second image and cause the one or more autonomous devices to place the one or more objects based, at least in part, on the correspondence ([0025] disclosing a robot gripping an object to place the object in a target position and attitude for the object. [0038]-[0040] disclosing a difference between the corresponding pixels between the two images).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban to incorporate the teaching of Yano of wherein one or more circuits are to identify a correspondence between pixels in a first current image and pixels in a second image and cause the one or more autonomous devices to place the one or more objects based, at least in part, on the correspondence in order to determine a pose of an object to place an object in a target location as taught by Yano [0038]-[0040]. The combination/substitution is obvious yielding predictable results in order to accurately place objects in target positions, and for refinement, verification and adjustment improving accuracy and reducing errors.
Regarding claim 9, Claussen teaches a system, comprising:
one or more circuits to use one or more neural networks to cause one or more autonomous devices to place one or more objects in a location and orientation based, at least in part, on one or more images of the location and orientation ([0012]-[0015] disclosing a processor with memory, it is interpreted that a processor comprises circuits. [0028] disclosing a robotic gripping device “autonomous device” that places an object in a desired location and pose based on the mismatch between the images of current pose “location and orientation” and desired pose “location and orientation” using neural network).
One or more memories to store the one or more images ([0014]-[0015] disclosing a memory to read input images, i.e., memory stores images).
to cause one or more autonomous devices to place a first object in a location and orientation based, at least in part, on a first image of a current location and orientation and a goal image ([0012]-[0015] disclosing a processor with memory, it is interpreted that a processor comprises circuits. [0028] disclosing a robotic gripping device “autonomous device” that places an object in a desired location and pose based on the mismatch between the images of current pose “location and orientation” and desired pose “location and orientation” using neural network).
Claussen in a further embodiment teaches based at least in part on an optical flow of pixels ([0020] disclosing the configuration error between the poses in each image of the first and second image can be converted to optical flow).
The combination/substitution of pixel flow as taught by Claussen is obvious yielding predictable results in order to accurately place objects in target positions based on a difference between the pixels thus accurately matching the final goal configuration of the object improving the placement of objects, and for verification, adjustment and refinement of poses.
Claussen does not teach the optical flow of pixels; a plurality of objects; and to cause the one or more autonomous devices to move a second object of the plurality of objects based at least in part on the goal image and a second image of a current location and orientation captured after the first object is moved by the one or more autonomous devices.
Skyum teaches a plurality of objects; and to cause the one or more autonomous devices to move a second object of the plurality of objects based at least in part on the goal image and a second image of a current location and orientation captured ([0040]-[0067] disclosing steps of moving at least a second object using the same technique of moving the objects based on image current location and orientation and image of target desired location and orientation).
It would have been obvious to one of ordinary skill in the art to have combined the teaching of Skyum and Claussen yielding predictable results in order to move a plurality of objects based on images, its also obvious to apply the method of Claussen to the plurality of objects of Skyum in order to facilitate the picking and placing of objects improving efficiency.
While Claussen as modified by Skyum does not disclose after the first object is moved by the one or more autonomous devices.
Ban in the same field of endeavor teaches after the first object is moved by the one or more autonomous devices ([0029]-[0030] disclosing to capture a new image of the workpieces by the camera).
It would be obvious to combine the teaching of Ban of a new image captured after moving a first object to the method of comparing images Claussen as modified by Skyum yielding predictable results and improving the accuracy since one of the second objects may have moved due to the picking of a first object thus a more accurate location and orientation of the second object would be obtained.
Yano teaches the optical flow of pixels ([0038]-[0045] disclosing the control of the robot to place an object based on a difference in pixels between a current image and a target image in order to align the pixels together through iterative control, i.e., pixel flow).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban to incorporate the teaching of Yano of the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on one or more optical flow estimates. The combination/substitution of the method of Yano is obvious yielding predictable results in order to align the target image with a current image yielding predictable results, and incorporating the pixel flow as taught by Yano improves the placement by ensuring the brightness of each pixel is aligned thus improving object placement and can be used for verification and refinement of the method of Claussen improving accuracy and reducing errors.
Regarding claim 12, Claussen as modified by Skyum and Ban and Yano teaches the system of claim 9. Claussen as modified by Skyum and Ban and Yano further wherein the one or processors are to generate one or more optical flow estimates of pixels based, at least in part, on the first image, and the one or more processors are to cause the one or more autonomous devices to place the first object based, at least in part, on the one or more optical flow estimates.
Yano teaches wherein the one or processors are to generate one or more optical flow estimates of pixels based, at least in part, on the first image, and the one or more processors are to cause the one or more autonomous devices to place the first object based, at least in part, on the one or more optical flow estimates ([0038]-[0045] disclosing the control of the robot to place an object based on a difference in pixels between a current image and a target image in order to align the pixels together through iterative control, i.e., pixel flow estimates).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban to incorporate the teaching of Yano of the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on one or more optical flow estimates. The combination/substitution of the method of Yano is obvious yielding predictable results in order to align the target image with a current image yielding predictable results, and for verification, adjustment and refinement of poses.
Regarding claim 15, Claussen teaches a method comprising:
Using one or more neural networks to cause one or more autonomous devices to place one or more objects in a location and orientation based, at least in part, on one or more images of the location and orientation ([0012]-[0015] disclosing a processor with memory, it is interpreted that a processor comprises circuits. [0028] disclosing a robotic gripping device “autonomous device” that places an object in a desired location and pose based on the mismatch between the images of current pose “location and orientation” and desired pose “location and orientation” using neural network).
to cause one or more autonomous devices to place a first object in a location and orientation based, at least in part, on a first image of a current location and orientation and a goal image ([0012]-[0015] disclosing a processor with memory, it is interpreted that a processor comprises circuits. [0028] disclosing a robotic gripping device “autonomous device” that places an object in a desired location and pose based on the mismatch between the images of current pose “location and orientation” and desired pose “location and orientation” using neural network).
Claussen in a further embodiment teaches based at least in part on an optical flow of pixels ([0020] disclosing the configuration error between the poses in each image of the first and second image can be converted to optical flow).
The combination/substitution of pixel flow as taught by Claussen is obvious yielding predictable results in order to accurately place objects in target positions based on a difference between the pixels thus accurately matching the final goal configuration of the object improving the placement of objects, and for verification, adjustment and refinement of poses.
Claussen does not teach the optical flow of pixels; a plurality of objects; and to cause the one or more autonomous devices to move a second object of the plurality of objects based at least in part on the goal image and a second image of a current location and orientation captured after the first object is moved by the one or more autonomous devices.
Skyum teaches a plurality of objects; and to cause the one or more autonomous devices to move a second object of the plurality of objects based at least in part on the goal image and a second image of a current location and orientation captured ([0040]-[0067] disclosing steps of moving at least a second object using the same technique of moving the objects based on image current location and orientation and image of target desired location and orientation).
It would have been obvious to one of ordinary skill in the art to have combined the teaching of Skyum and Claussen yielding predictable results in order to move a plurality of objects based on images, its also obvious to apply the method of Claussen to the plurality of objects of Skyum in order to facilitate the picking and placing of objects improving efficiency.
While Claussen as modified by Skyum does not disclose after the first object is moved by the one or more autonomous devices.
Ban in the same field of endeavor teaches after the first object is moved by the one or more autonomous devices ([0029]-[0030] disclosing to capture a new image of the workpieces by the camera).
It would be obvious to combine the teaching of Ban of a new image captured after moving a first object to the method of comparing images Claussen as modified by Skyum yielding predictable results and improving the accuracy since one of the second objects may have moved due to the picking of a first object thus a more accurate location and orientation of the second object would be obtained.
Yano teaches the optical flow of pixels ([0038]-[0045] disclosing the control of the robot to place an object based on a difference in pixels between a current image and a target image in order to align the pixels together through iterative control, i.e., pixel flow).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban to incorporate the teaching of Yano of the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on one or more optical flow estimates. The combination/substitution of the method of Yano is obvious yielding predictable results in order to align the target image with a current image yielding predictable results, and incorporating the pixel flow as taught by Yano improves the placement by ensuring the brightness of each pixel is aligned thus improving object placement and can be used for verification and refinement of the method of Claussen improving accuracy and reducing errors.
Regarding claim 16, Claussen as modified by Skyum and Ban and Yano teaches the method of claim 15, wherein the goal image comprises an image of a goal location and orientation of objects (Claussen [0012]-[0015] disclosing a processor with memory, it is interpreted that a processor comprises circuits. [0028] disclosing a robotic gripping device “autonomous device” that places an object in a desired location and pose based on the mismatch between the images of current pose “location and orientation” and desired pose “location and orientation” using neural network).
Skyum further teaches an image of a goal and orientation of a plurality of objects ([0040]-[0067] disclosing selecting an object to place and placing any of the plurality of objects based on the goal image).
It would have been obvious to one of ordinary skill in the art to have combined the teaching of Skyum and Claussen yielding predictable results in order to move a plurality of objects based on images, its also obvious to apply the method of Claussen to the plurality of objects of Skyum in order to facilitate the picking and placing of objects improving efficiency.
Regarding claim 20, Claussen as modified by Skyum and Ban and Yano teaches the method of claim 15, wherein using the one or more neural networks to cause one or more autonomous devices to place one or more objects in a location and orientation is performed without using three-dimensional models of the one or more objects (Clausen [0028] disclosing the use of a configuration error between location and orientation of images of current pose and target pose in placing the target object in a target location and orientation, i.e., without the use of a three-dimensional model of the object).
Regarding claim 21, Claussen teaches a machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, is to cause the one or more processors to at least ([0012]-[0015] disclosing a processor with memory, it is interpreted that a processor comprises circuits):
to cause one or more autonomous devices to place a first object in a location and orientation based, at least in part, on a first image of a current location and orientation and a goal image ([0012]-[0015] disclosing a processor with memory, it is interpreted that a processor comprises circuits. [0028] disclosing a robotic gripping device “autonomous device” that places an object in a desired location and pose based on the mismatch between the images of current pose “location and orientation” and desired pose “location and orientation” using neural network).
Claussen in a further embodiment teaches based at least in part on an optical flow of pixels ([0020] disclosing the configuration error between the poses in each image of the first and second image can be converted to optical flow).
The combination/substitution of pixel flow as taught by Claussen is obvious yielding predictable results in order to accurately place objects in target positions based on a difference between the pixels thus accurately matching the final goal configuration of the object improving the placement of objects, and for verification, adjustment and refinement of poses.
Claussen does not teach the optical flow of pixels; a plurality of objects; and to cause the one or more autonomous devices to move a second object of the plurality of objects based at least in part on the goal image and a second image of a current location and orientation captured after the first object is moved by the one or more autonomous devices.
Skyum teaches a plurality of objects; and to cause the one or more autonomous devices to move a second object of the plurality of objects based at least in part on the goal image and a second image of a current location and orientation captured ([0040]-[0067] disclosing steps of moving at least a second object using the same technique of moving the objects based on image current location and orientation and image of target desired location and orientation).
It would have been obvious to one of ordinary skill in the art to have combined the teaching of Skyum and Claussen yielding predictable results in order to move a plurality of objects based on images, its also obvious to apply the method of Claussen to the plurality of objects of Skyum in order to facilitate the picking and placing of objects improving efficiency.
While Claussen as modified by Skyum does not disclose after the first object is moved by the one or more autonomous devices.
Ban in the same field of endeavor teaches after the first object is moved by the one or more autonomous devices ([0029]-[0030] disclosing to capture a new image of the workpieces by the camera).
It would be obvious to combine the teaching of Ban of a new image captured after moving a first object to the method of comparing images Claussen as modified by Skyum yielding predictable results and improving the accuracy since one of the second objects may have moved due to the picking of a first object thus a more accurate location and orientation of the second object would be obtained.
Yano teaches the optical flow of pixels ([0038]-[0045] disclosing the control of the robot to place an object based on a difference in pixels between a current image and a target image in order to align the pixels together through iterative control, i.e., pixel flow).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban to incorporate the teaching of Yano of the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on one or more optical flow estimates. The combination/substitution of the method of Yano is obvious yielding predictable results in order to align the target image with a current image yielding predictable results, and incorporating the pixel flow as taught by Yano improves the placement by ensuring the brightness of each pixel is aligned thus improving object placement and can be used for verification and refinement of the method of Claussen improving accuracy and reducing errors.
Regarding claim 27, Claussen teaches an autonomous device comprising:
A manipulator (Claussen [0027]-[0028] disclosing a robotic device with a gripper, i.e., robotic manipulator for gripping an object, see figure 2).
A processor that to include one or more autonomous devices to place a first object in a location and orientation based, at least in part, on a first image of a current location and orientation and a goal image ([0012]-[0015] disclosing a processor with memory, it is interpreted that a processor comprises circuits. [0028] disclosing a robotic gripping device “autonomous device” that places an object in a desired location and pose based on the mismatch between the images of current pose “location and orientation” and desired pose “location and orientation” using neural network).
Claussen in a further embodiment teaches based at least in part on an optical flow of pixels ([0020] disclosing the configuration error between the poses in each image of the first and second image can be converted to optical flow).
The combination/substitution of pixel flow as taught by Claussen is obvious yielding predictable results in order to accurately place objects in target positions based on a difference between the pixels thus accurately matching the final goal configuration of the object improving the placement of objects, and for verification, adjustment and refinement of poses.
Claussen does not teach the optical flow of pixels; a plurality of objects; and to cause the one or more autonomous devices to move a second object of the plurality of objects based at least in part on the goal image and a second image of a current location and orientation captured after the first object is moved by the one or more autonomous devices.
Skyum teaches a plurality of objects; and to cause the one or more autonomous devices to move a second object of the plurality of objects based at least in part on the goal image and a second image of a current location and orientation captured ([0040]-[0067] disclosing steps of moving at least a second object using the same technique of moving the objects based on image current location and orientation and image of target desired location and orientation).
It would have been obvious to one of ordinary skill in the art to have combined the teaching of Skyum and Claussen yielding predictable results in order to move a plurality of objects based on images, its also obvious to apply the method of Claussen to the plurality of objects of Skyum in order to facilitate the picking and placing of objects improving efficiency.
While Claussen as modified by Skyum does not disclose after the first object is moved by the one or more autonomous devices.
Ban in the same field of endeavor teaches after the first object is moved by the one or more autonomous devices ([0029]-[0030] disclosing to capture a new image of the workpieces by the camera).
It would be obvious to combine the teaching of Ban of a new image captured after moving a first object to the method of comparing images Claussen as modified by Skyum yielding predictable results and improving the accuracy since one of the second objects may have moved due to the picking of a first object thus a more accurate location and orientation of the second object would be obtained.
Yano teaches the optical flow of pixels ([0038]-[0045] disclosing the control of the robot to place an object based on a difference in pixels between a current image and a target image in order to align the pixels together through iterative control, i.e., pixel flow).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban to incorporate the teaching of Yano of the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on one or more optical flow estimates. The combination/substitution of the method of Yano is obvious yielding predictable results in order to align the target image with a current image yielding predictable results, and incorporating the pixel flow as taught by Yano improves the placement by ensuring the brightness of each pixel is aligned thus improving object placement and can be used for verification and refinement of the method of Claussen improving accuracy and reducing errors.
Regarding claim 30, Claussen as modified by Skyum and Ban and Yano teaches the autonomous device of claim 27, wherein the manipulator includes a robotic arm (Claussen [0027]-[0028] and figure 2 discloses a robotic arm).
Claims 3, 17, 29 are rejected under 35 U.S.C. 103 as being unpatentable by Claussen (US20220347853) in view of Skyum (US20230150777) and Ban (US20070274812) and Yano (US20200376678) and Xu (US20220134546).
Regarding claim 3, Claussen as modified by Skyum and Ban and Yano teaches the processor of claim 1. Claussen as modified by Skyum and Ban and Yano does not teach wherein the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on a transformation of a current image to at least one of the one or more images.
Xu teaches wherein the one or more circuits are to cause the one or more autonomous devices to place the first objects based, at least in part, on a transformation of the first image to the goal images ([0094] disclosing modifying the current image into an image defining the target location to place an object in the target location, i.e., the placing of the object is based on a transformation of a current image to one or more goal images).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban and Yano to incorporate the teaching of Xu of wherein the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on a transformation of a current image to at least one of the one or more images in order to push an object into a target location as taught by Xu [0094]. The combination is obvious yielding predictable results of placing objects in target positions.
Regarding claim 17, Claussen as modified by Skyum and Ban and Yano teaches the method of claim 15. Claussen as modified by Skyum and Ban and Yano further teaches wherein the method further includes segmenting the one or more second images of the location and orientation of the one or more objects, generating one or more transformations based, at least in part, on the segmented images, and causing the one or more autonomous devices to place the one or more objects based, at least in part, on the one or more transformations. However, Claussen teaches the second image of the location and orientation of the object [0028].
Yano further teaches image segmentation ([0038]-[0045] disclosing the control of the robot to place an object based on a difference in pixels between a current image and a target image in order to align the pixels together through iterative control, i.e., segmentation of image into pixels).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban to incorporate the teaching of Yano of the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on one or more optical flow estimates. The combination/substitution of the method of Yano is obvious yielding predictable results in order to align the target image with a current image yielding predictable results, and for verification, adjustment and refinement of poses.
Xu teaches generating one or more transformations based, at least in part, on the images, and causing the one or more autonomous devices to place the one or more objects based, at least in part, on the one or more transformations ([0094] disclosing modifying the current image into an image defining the target location to place an object in the target location, i.e., the placing of the object is based on a transformation of a current image to one or more images).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban and Yano to incorporate the teaching of Xu of wherein the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on a transformation of a current image to at least one of the one or more images in order to push an object into a target location as taught by Xu [0094]. Segmentation of images is known as cited by Lin, transforming of images is known as shown by Xu, it would have been obvious to try to transform a segmented image with reasonable expectation of success.
Claim 29 is rejected for similar reasons as claim 17, see above rejection.
Claims 4 are rejected under 35 U.S.C. 103 as being unpatentable by Claussen (US20220347853) in view of Skyum (US20230150777) and Ban (US20070274812) and Yano (US20200376678) and Cherian (US20220309672).
Regarding claim 4, Claussen as modified by Skyum and Ban and Yano teaches the processor of claim 1. Claussen as modified by Skyum and Ban and Yano does not teach wherein the one or more circuits are to cause the one or more autonomous devices to place the one or more objects in the target location and orientation further, based in part, on pixel depth information.
Cherian teaches wherein the one or more circuits are to cause the one or more autonomous devices to place the first objects in the target location and orientation further, based in part, on pixel depth information ([0023] disclosing pixel depth images with pixel depth information. [0128] disclosing a pose to inset an object in a pose “location and orientation”. See also [0024] disclosing placing an object using depth images).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban and Yano to incorporate the teaching of Cherian of wherein the one or more circuits are to cause the one or more autonomous devices to place the one or more objects in the target location and orientation further, based in part, on pixel depth information in order to place objects in a desired target position as taught by Cherian [0023]-[0024]. The combination/substitution is obvious yielding predictable results in order to accurately place objects in target positions.
Claims 5 and 31 are rejected under 35 U.S.C. 103 as being unpatentable by Claussen (US20220347853) in view of Skyum (US20230150777) and Ban (US20070274812) and Yano (US20200376678) and Watanabe (US20170106540).
Regarding claim 5, Claussen as modified by Skyum and Ban and Yano teaches the processor of claim 1. Claussen as modified by Skyum and Ban and Yano does not teach wherein the goal images is a color images that include pixel depth values.
Watanabe teaches wherein the goal images is a color images that include pixel depth values ([0063]-[0070] disclosing images with colors and depth info).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban and Yano to incorporate the teaching of Watanabe of wherein the goal images is a color images that include pixel depth values in order to easily view the degree of overlapping as disclosed by Watanabe [0064]. The combination/substitution is obvious yielding predictable results in order to accurately place objects in target positions.
Claim 31 is rejected for similar reasons as claim 5, see above rejection.
Claims 7 are rejected under 35 U.S.C. 103 as being unpatentable by Claussen (US20220347853) in view of Skyum (US20230150777) and Ban (US20070274812) and Yano (US20200376678) and Moreno (US20220297958).
Regarding claim 7, Claussen as modified by Skyum and Ban and Yano teaches the processor of claim 1. Claussen as modified by Skyum and Ban and Yano does not teach wherein the one or more circuits are to identify a set of objects of the plurality objects that can be moved to the location and orientation without colliding with another object of the plurality objects. However, Claussen teaches location and orientation of objects ([0028] disclosing a robotic gripping device “autonomous device” that places an object in a desired location and pose).
Moreno teaches wherein the one or more circuits are to identify a set of objects of the plurality objects that can be moved to the location and orientation without colliding with another object of the plurality objects ([0083] disclosing determining a path for the object that does not collide with other objects).
It would have been obvious to have modified the teaching of Claussen as modified by Skyum and Ban and Yano to incorporate the teaching of Moreno of identify objects that can be moved without colliding with other objects in order to avoid collision.
Claims 10, 11 and 23 are rejected under 35 U.S.C. 103 as being unpatentable by Claussen (US20220347853) in view of Skyum (US20230150777) and Ban (US20070274812) and Yano (US20200376678) and Fan (US20230124599).
Regarding claim 10, Claussen as modified by Skyum and Ban and Yano teaches the system of claim 9, wherein the goal images comprise location and orientation ([0011] disclosing the desired configuration is obtained via an image of the desired configuration of the object. [0012]-[0015] disclosing a processor with memory, it is interpreted that a processor comprises circuits. [0028] disclosing a robotic gripping device “autonomous device” that places an object in a desired location and pose based on the mismatch between the images of current pose “location and orientation” and desired pose “location and orientation” using neural network).
Fan teaches color image ([0018] disclosing color image and pixel depth data).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban and Yano to incorporate the teaching of Fan of color image in order to determine pose of objects for manipulation by a robotic arm as taught by Fan [0018]. It would have been obvious to try to use color images which is a predictable solution of imaging objects for manipulation with reasonable expectation of success.
Regarding claim 11, Claussen as modified by Skyum and Ban and Yano teaches the system of claim 9, wherein the goal images comprise ([0028] disclosing a robotic gripping device “autonomous device” that places an object in a desired location and pose based on the mismatch between the images of current pose “location and orientation” and desired pose “location and orientation”).
Claussen as modified by Skyum and Ban and Yano does not teach color image with pixel depth information.
Fan teaches color image with pixel depth information ([0018] disclosing color image and pixel depth data).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban and Yano to incorporate the teaching of Fan of color image with pixel depth information in order to determine pose of objects for manipulation by a robotic arm as taught by Fan [0018]. It would have been obvious to try to use color images which is a predictable solution of imaging objects for manipulation with reasonable expectation of success.
Regarding claim 23, Claussen as modified by Skyum and Ban and Yano teaches a machine-readable medium of claim 21. Claussen as modified by Skyum and Ban and Yano does not teach wherein the goal images comprise a color image with depth information for pixels of a goal location and orientation for the plurality objects. However, Claussen teaches images of goal location and orientation of at least one object [0028].
Fan teaches wherein the one or more images comprise a color image with depth information for pixels of a location and orientation for the one or more objects ([0018] disclosing color image and pixel depth data of the object).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban and Yano to incorporate the teaching of Fan of the one or more images comprise a color image with depth information for pixels of a location and orientation for the one or more objects in order to determine pose of objects for manipulation by a robotic arm as taught by Fan [0018]. It would have been obvious to try to use color images which is a predictable solution of imaging objects for manipulation with reasonable expectation of success.
Claims 13 are rejected under 35 U.S.C. 103 as being unpatentable by Claussen (US20220347853) in view of in view of Skyum (US20230150777) and Ban (US20070274812) and Yano (US20200376678) and Moreira (US20220281114).
Regarding claim 13, Claussen as modified by Skyum and Ban and Yano teaches the system of claim 9. Claussen as modified by Skyum and Ban and Yano does not teach wherein the one or more processors are to assign one or more movement values to one or more objects that can be moved to the location and orientation without colliding with another object, and the one or more processors are to cause the one or more autonomous devices to move an object based, at least in part, on the one or more movement values.
Moreira teaches wherein the one or more processors are to assign one or more movement values to one or more objects that can be moved to the location and orientation without colliding with another object, and the one or more processors are to cause the one or more autonomous devices to move an object based, at least in part, on the one or more movement values ([0016] disclosing defining a safety volume “value” for objects that are moved to a destination and controlling the robot based on the safety volume that avoids the collision with objects, see also [0020], [0036], [0095]).
It would have been obvious to one of ordinary skill in the art to have modified the teaching of Claussen as modified by Skyum and Ban and Yano to incorporate the teaching of Moreira of the one or more processors are to assign one or more movement values to one or more objects that can be moved to the location and orientation without colliding with another object, and the one or more processors are to cause the one or more autonomous devices to move an object based, at least in part, on the one or more movement values in order to avoid collision with other objects as taught by Moreira [0095].
Claims 14, 18, 22, 28 are rejected under 35 U.S.C. 103 as being unpatentable by Claussen (US20220347853) in view of Skyum (US20230150777) and Ban (US20070274812) and Yano (US20200376678) and Lin (US20220156946).
Regarding claim 14, Claussen as modified by Skyum and Ban and Yano teaches the system of claim 9. Claussen as modified by Skyum and Ban and Yano further teaches wherein the one or more processors are to estimate optical flow of pixels based, at least in part, on one or more of the one or more neural networks, two or more current images included in the one or more first images, and a goal image included in the one or more second images, and the one or more processors are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on the estimated optical flow.
Specifically, Claussen teaches and the one or more processors are to cause the one or more autonomous devices to place the first objects based on the first images and the goal images and one or more neural network ([0012]-[0015] disclosing a processor with memory, it is interpreted that a processor comprises circuits. [0027]-[0028] disclosing a robotic gripping device “autonomous device” that places an object in a desired location and pose based on the mismatch between the images of current pose “location and orientation” and desired pose “location and orientation” using neural network).
Specifically, Yano teaches controlling the robot to place an object in a target position and attitude based pixel flow ([0038]-[0045] disclosing the control of the robot to place an object based on a difference in pixels between a current image and a target image).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban to incorporate the teaching of Yano of controlling the robot to place an object in a target position and attitude based on pixel flow in order to control the robot to place an object based on optical flow difference as taught by Yano. The combination is obvious as stated above for verification and refinement of poses improving results.
while Claussen as modified by Yano and Lin is silent about the determining the optical flow estimate based on neural network.
Lin teaches wherein the one or more processors are to generate one or more optical flow estimates based, at least in part, on a one or more images and one or more neural networks (abstract and at least [0040]-[0041] disclosing an neural network estimates optical flow of pixels based on correspondence of pixels between a sequence of images. [0070] disclosing the optical flow based on more than two images).
It would have been obvious to one of ordinary skill in the art to have modified the teaching of Claussen as modified by Skyum and Ban to incorporate the teaching of Lin of optical flow of pixels between images in order to track an object as taught by Lin [0040]. Combining the teaching of neural network for estimating the optical flow improves the estimation based on learned features.
Claussen teaches the neural network placing objects based on images of the location and orientation of the object, Lin teaches the pixel flow based on neural network to track objects. Yano teaches placing objects based on pixel difference, i.e., pixel flow. It would have been obvious to try to use the optical flow of pixels to place objects which is an obvious solution for tracking objects for placement with reasonable expectation of success.
Regarding claim 18, Claussen as modified by Skyum and Ban and yano further teaches the method of claim 15, wherein the method further includes estimating the optical flow of pixels from the first image to a goal image at least in part on the one or more neural networks, and causing the one or more autonomous devices to place the first objects based, at least in part, on the estimated optical flow.
Yano teaches estimating the optical flow of pixels from the first image to a goal image, and causing the one or more autonomous devices to place the first objects based, at least in part, on the estimated optical flow. ([0038]-[0045] disclosing the control of the robot to place an object based on a difference in pixels between a current image and a target image, i.e., optical flow).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban and Lin to incorporate the teaching of Yano of controlling the robot to place an object in a target position and attitude based pixel flow in order to control the robot to place an object based on optical flow difference as taught by Yano and for verification, refinement of poses improving accuracy and reducing errors.
Lin teaches optical flow of pixels between images based on neural network (abstract and at least [0040]-[41] disclosing an neural network estimates optical flow of pixels based on correspondence of pixels between a sequence of images. [0070] disclosing the optical flow based on more than two images).
It would have been obvious to one of ordinary skill in the art to have modified the teaching of Claussen as modified by Skyum and Ban to incorporate the teaching of Lin of optical flow of pixels between images in order to track an object as taught by Lin [0040]. Combining the teaching of neural network for estimating the optical flow improves the estimation based on learned features.
Claussen teaches the neural network placing objects based on images of the location and orientation of the object, Lin teaches the pixel flow to track objects. Yano teaches placing objects based on pixel difference, i.e., optical flow. It would have been obvious to try to use the optical flow of pixels to place objects which is an obvious solution for tracking objects with reasonable expectation of success.
Regarding claim 28, Claussen as modified by Skyum and Ban and Yano further teaches the method of claim 15 wherein the one or more circuits are to estimate optical flow of pixels from the first image to the goal image based, at least in part on the one or more neural networks, and cause the manipulator to place the first objects based, at least in part, on the estimated optical flow of pixels.
Specifically, Claussen teaches placing objects based on current image and goal image ([0027]-[0028] disclosing placing object based on a current image and goal image).
Yano teaches estimating the optical flow of pixels from the first image to a goal image, controlling the robot to place an object in a target position and attitude based pixel flow ([0038]-[0045] disclosing the control of the robot to place an object based on a difference in pixels between a current image and a target image).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban and Lin to incorporate the teaching of Yano of controlling the robot to place an object in a target position and attitude based pixel flow in order to control the robot to place an object based on optical flow difference as taught by Yano.
Lin teaches optical flow of pixels between images based on neural network (abstract and at least [0040]-[41] disclosing an neural network estimates optical flow of pixels based on correspondence of pixels between a sequence of images. [0070] disclosing the optical flow based on more than two images).
It would have been obvious to one of ordinary skill in the art to have modified the teaching of Claussen as modified by Skyum and Ban to incorporate the teaching of Lin of optical flow of pixels between images in order to track an object as taught by Lin [0040]. Combining the teaching of neural network for estimating the optical flow improves the estimation based on learned features.
Claussen teaches the neural network placing objects based on images of the location and orientation of the object, Lin teaches the pixel flow to track objects. Yano teaches placing objects based on pixel difference. It would have been obvious to try to use the optical flow of pixels to place objects which is an obvious solution for tracking objects with reasonable expectation of success.
Claims 22 are rejected for similar reasons as claim 18, respectively, see above rejection.
Claims 19 are rejected under 35 U.S.C. 103 as being unpatentable by Claussen (US20220347853) in view of Skyum (US20230150777) and Ban (US20070274812) and Yano (US20200376678) and Kwak (US20240043227).
Regarding claim 19, Claussen as modified by Skyum and Ban and Yano teaches the method of claim 15. Claussen as modified by Skyum and Ban and Yano does not teach wherein the method further includes selecting an object from a set of noncolliding objects to move.
Kwak teaches wherein the method further includes selecting an object from a set of noncolliding objects to move ([0097] disclosing when it is determined that all the objects are separated from each other more than a reference distance, the reference object is picked).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban and Yano to incorporate the teaching of wherein the method further includes selecting an object from a set of noncolliding objects to move in order to pick up the one object without disturbing other objects.
Claims 24 are rejected under 35 U.S.C. 103 as being unpatentable by Claussen (US20220347853) in view of Skyum (US20230150777) and Ban (US20070274812) and Yano (US20200376678) and Oishi (US20220241982).
Regarding claim 24, Claussen as modified by Skyum and Ban and Yano teaches the machine-readable medium of claim 21. Claussen as modified by Skyum and Ban and Yano does not teach wherein the instructions, which if performed by the one or more processors, are to cause the one or more processors to at least iteratively select two or more objects of the one or more objects to be moved based, at least in part, on a series of two or more images showing a current location and orientation of the one or more objects.
Oishi teaches wherein the instructions, which if performed by the one or more processors, are to cause the one or more processors to at least iteratively select two or more objects of the one or more objects to be moved based, at least in part, on a series of two or more images showing a current location and orientation of the one or more objects ([0053]-[0054]] disclosing multiple images of multiple workpieces and determining based on the images of the workpieces, showing their areas and locations, i.e., orientation and location, an order to sequentially pick the pieces, i.e., two or more pieces).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban and yano to incorporate the teaching of Oishi of wherein the instructions, which if performed by the one or more processors, are to cause the one or more processors to at least iteratively select two or more objects of the one or more objects to be moved based, at least in part, on a series of two or more images showing a current location and orientation of the one or more objects in order to choose target workpieces based on image recognition and sequentially hold the workpieces as taught by Oishi [0054].
Claims 25 are rejected under 35 U.S.C. 103 as being unpatentable by Claussen (US20220347853) in view of Skyum (US20230150777) and Ban (US20070274812) and Yano (US20200376678) and Xu (US20220134546) and Shi (US20240029300).
Regarding claim 25, Claussen as modified by Skyum and Ban and Yano teaches the non-transitory machine-readable medium of claim 21, Claussen as modified by Skyum and Ban and Yano does not teach the instructions, which if performed by the one or more processors, are to cause the one or more processors to at least generate one or more transformations from the first image to the goal image and cause the one or more autonomous devices to place the first based, at least in part, on the one or more transformations.
Shi teaches the instructions, which if performed by the one or more processors, are to cause the one or more processors to at least generate one or more transformations from one or more current images included in one or more first images of the one or more objects to the goal image ([0022] disclosing the obtaining of images of keyframes. [0037] disclosing the transformation of current keyframes to final frame “goal image”).
It would have been obvious to one of ordinary skill in the art to have modified the teaching of Claussen as modified by Skyum and Ban and Yano to incorporate the teaching of Shi of the instructions, which if performed by the one or more processors, are to cause the one or more processors to at least generate one or more transformations from one or more current images included in one or more first images of the one or more objects to the goal image in order to assist in localization of the robot as taught by Shi.
Claussen as modified by Skyum and Ban and Yano ad Shi does not teach cause the one or more autonomous devices to place the one or more objects based, at least in part, on the one or more transformations
Xu teaches cause the one or more autonomous devices to place the one or more objects based, at least in part, on the one or more transformations ([0094] disclosing modifying the current image into an image defining the target location to place an object in the target location, i.e., the placing of the object is based on a transformation of a current image to one or more images).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Claussen as modified by Skyum and Ban and Yano and Shi and to incorporate the teaching of Xu of wherein the one or more circuits are to cause the one or more autonomous devices to place the one or more objects based, at least in part, on a transformation of a current image to at least one of the one or more images in order to push an object into a target location as taught by Xu [0094].
Response to Arguments
Applicant’s arguments have been fully considered but they are not persuasive. The amendment to the claim requires only one object to be moved based on the optical flow whereas the interpretation of the previously allowable subject matter in light of independent claims required moving both objects based on the optical flow. However, examiner believes that the current art on record can be used for rejecting the claim even if rewritten in the previously indicated allowable subject matter. The combination of Claussen and Skyum and ban and Yano is obvious to teach the use of the method on more than one object and thus improving the placement of the second object yielding predictable results improving accuracy and reducing errors in placement.
Conclusion
The prior art made of record and not relied upon is considered pertinent to
applicant's disclosure. The prior art cited in PTO-892 and not mentioned above disclose related devices and methods.
US20230071384 discloses determining a refined posture of the object based on images.
US10926952 discloses a neural network to move an article from a current position and orientation adjacent to a goal position and orientation.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMAD O EL SAYAH whose telephone number is (571)270-7734. The examiner can normally be reached on M-Th 6:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached on (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMAD O EL SAYAH/Examiner, Art Unit 3658B
Prosecution Insights