DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The IDS(s) has/have been considered and placed in the application file.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-4, 6-13 and 15 is/are rejected under 35 U.S.C. 102(a)(1) and (a)(2) as being anticipated by Kollar et al., US 2023/0401721 A1, (hereinafter “KOLLAR”).
Claims 1 and 9.
KOLLAR discloses an apparatus for detecting objects, comprising:
a first optical sensor having a first field of view configured to be directed toward a target object at a first angle relative to the apparatus; a second optical sensor having a second field of view configured to be directed toward the target object at a second angle relative to the apparatus (KOLLAR¶ 23: “captures images from two cameras and matches pixels in each image to a single point in 3D space.”); and
a processor (¶ 25) configured to: detect, by the first optical sensor, a first image of the target object at the first angle relative to the apparatus; detect, by the second optical sensor, a second image of the target object at the second angle relative to the apparatus (KOLLAR ¶ 57: “a left stereo RGB image 402 and a right stereo RGB image 404 are fed into a left feature extractor 410 and a right feature extractor 414.”);
infer, by a first machine learning model, depth information of the first image based on the first image and the second image (KOLLAR ¶ 58: the left and right feature extractors “are implemented using neural networks.” ¶ 59: “the SCVN 420 performs learned stereo matching to generate the low-resolution disparity image 430 … The left stereo RGB image 402 and the right stereo RGB image 404 are fed into neural networks.” ¶ 23: “a disparity … of the single point can be directly mapped to depth.”); and
infer, by a second machine learning model, data indicative of location of the target object based on the first image and the depth information (KOLLAR ¶ 63: features taken from
the left stereo RGB image 402 are “concatenated with the low-resolution disparity image
430 output of the SCVN 420, and fed into the feature extraction backbone 440”, whose
output “is fed into each of the output prediction heads.” This teaches a second network
receiving the first image and the inferred depth and outputting object location.).
Claims 2 and 10.
KOLLAR discloses the apparatus of claim 1, wherein: the first optical sensor comprises a first stereo camera; and the second optical sensor comprises a second stereo camera (KOLLAR ¶ 23.).
Claims 3, 11, and 12.
KOLLAR discloses training the first model on paired views at the two angles (KOLLAR ¶ 57, Fig. 4.).
Claims 4 and 13.
KOLLAR discloses wherein the second machine learning model comprises a neural network that is trained based on a training dataset comprising a plurality of training images and a plurality of corresponding depth information data associated with, respectively, the plurality of training images (KOLLAR ¶ 63.).
Claims 6, 7, and 15.
KOLLAR discloses wherein the data indicative of the location of the target object correspond to a bounding box around the target object on the first image and wherein the data indicative of the location of the target object comprise a set of coordinates corresponding to the target object on the first image (KOLLAR ¶ 15: “oriented bounding boxes.” Fig. 7.; ¶ 56: “predict segmentation masks, oriented bounding boxes (OBBs), and keypoints on unseen objects”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 5, 14, 16 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kollar in view of Rowell et al., US 2020/0342652 A1 (hereinafter “ROWELL”).
Claim 16.
KOLLAR and ROWELL disclose a method for training a machine learning model for detecting objects, comprising: generating a plurality of synthetic object images (KOLLAR¶ 57: “nonphotorealistic simulation graphics are used for generating the synthetic stereo pair of
images”);
(KOLLAR ¶ 57: “a left stereo RGB image 402 and a right stereo RGB image
404” – the first perspective);
generating a plurality of second training images associated with, respectively, the plurality of first training images, wherein each of the plurality of second training images comprises a second perspective of the respective randomized subset of the plurality of synthetic object images and the respective background image (KOLLAR ¶ 57 teaches the right image of the same pair as the second perspective.);
inferring, by another machine learning model, a plurality of depth information data associated with, respectively, the plurality of first training images based on the plurality of first training images and the plurality of second training images (KOLLAR ¶ 59, see claim 1.);
generating a training dataset by combining first data related to a first training image of the plurality of first training images with second data related to a corresponding depth information instance of the plurality of depth information data (KOLLAR¶ 63 see claim 1.); and
training the machine learning model based on the training dataset (KOLLAR ¶ 56: the
architecture is trained to “predict segmentation masks, oriented bounding boxes (OBBs),
and keypoints on unseen objects.”).
KOLLAR discloses all the subject matter as described above except for specifically teaching “selecting a plurality of randomized subsets of the plurality of synthetic object images; generating a plurality of first training images by adding each of the plurality of randomized subsets of the plurality of synthetic object images to a respective background image.” However, ROWELL in the same field of endeavor teaches selecting randomized subsets of the synthetic object images and using them for training (ROWELL ¶ 58: “After generating the background image and adding the background image to a background portion of the image scene, the image scene generator 124 populates the image scene by adding foreground objects to the foreground portion of the image scene” and “foreground objects are selected from a 3D model library including 36,000 or more 3D objects”; ¶ 60: “A 3D model arrangement routine executed by the image scene generator 124 to position the 3D models”; ¶19: “synthetic images are generated by assembling 2D backgrounds, objects, and textures into an image scene”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Kollar and Rowell before the effective filing date of the claimed invention. The motivation for this combination of references would have been, as Rowell states at ¶ 6, that “without additional image data channels or stereo image pairs, the range of problems networks can be trained to solve is limited.” This is combining prior art elements according to known methods to yield predictable results (MPEP 2143(A)). Kollar teaches the learned stereo network, the disparity output and the downstream detection (¶ 23-24, 59-63), and Rowell teaches
assembling synthetic scenes from backgrounds and object images and rendering the additional depth of objects in the scene (¶ 19, 57, 111). Substituting Rowell’s assembled scenes for Kollar’s domain randomized scenes yields the same training stereo pair of images with per pixel, so the resulting trained network behaves predictably.
Claims 5 and 14.
KOLLAR and ROWELL disclose wherein: the plurality of training images comprise a plurality of synthetically generated ground truth images, each comprising one or more object images included in a background image, wherein the one or more object images and the background image are proportionately sized (ROWELL ¶ 19, 57-58.).
Claim 20.
The method of claim 16, wherein combining the first data with the second data comprises concatenating RGB data related to the first training image with a depth value related to the corresponding depth information instance (KOLLAR ¶ 63; ROWELL ¶ 29.).
Claims 17 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kollar in view of Rowell, further in view of Solowjow et al., US 2024/0296662 A1 (hereinafter “SOLOWJOW”).
Claims 17 and 19.
KOLLAR and ROWELL disclose the training image generation of claim 16 as described above
except for specifically determining an arrangement based on a physics simulation and a
container image added to the respective background image.
However, SOLOWJOW in the same field of endeavor teaches both (SOLOWJOW ¶ 7: the method comprises "rendering a plurality of images by a physics-based simulator using the mesh representations of the objects", and each rendered image "captures a simulated environment containing objects belonging to multiple of said object classes of interest placed in a bin"; ¶ 27: "multiple objects 118 are placed in a bin 120" and may be "arranged in arbitrary poses within the bin 120".)
Therefore, it would have been obvious to one of ordinary skill in the art to combine Kollar, Rowell, and Solowjow before the effective filing date of the claimed invention. The motivation for this combination of references comes from Solowjow at ¶ 7 that states each rendered image “captures a simulated environment containing objects belonging to multiple of said object classes of interest placed in a bin.” This is combining prior art elements according to known methods to yield predictable results (MPEP 2143(A)). Kollar and Rowell teaches assembling synthetic object images onto a background, and Solowjow teaches rendering those objects with a physics-based simulator inside a bin (¶ 7, 27). Rendering the parts inside a container produces the predictable result of training images matching the deployed bin picking scene.
Claim 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kollar in view of Rowell, further in view of Tremblay et al., US 2019/0355150 A1 (hereinafter "TREMBLAY").
Claim 18.
KOLLAR and ROWELL disclose the training image generation as described above except for specifically teaching adding one or more distractor object images to the respective background image. However, TREMBLAY in the same field of endeavor teaches distractor object images added to the background image (TREMBLAY ¶ 67: “The process of generating domain randomized training data places the 3D objects within a virtual environment consisting
of various distractor objects in front of a 2D background image”)
Therefore, it would have been obvious to one of ordinary skill in the art to combine Kollar, Rowell, and Tremblay before the effective filing date of the claimed invention. The motivation for this combination of references comes from Tremblay’s ¶ 67 which states expanding pose distribution “avoid[s] overfitting to a particular set of 3D objects.” This is combining prior art elements according to known methods to yield predictable results (MPEP 2143(A)). The Kollar and Rowell combination teaches assembling synthetic object images onto a background image, and Tremblay teaches placing additional distractor objects in front of that background (¶ 67). The distractor objects of Tremblay force the detector to discriminate the objects of interest from irrelevant clutter in the combination exactly as they do in Tremblay. Adding rendered clutter to a synthetic training scene yields a training image of the same form and a detector that is more robust to clutter, a predictable result.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ross Varndell whose telephone number is (571)270-1922. The examiner can normally be reached M-F, 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached at (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ross Varndell/Primary Examiner, Art Unit 2674