Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The United States Patent & Trademark Office appreciates the application that is submitted by the inventor/assignee. The United States Patent & Trademark Office reviewed the following application and has made the following comments below.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/19/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The disclosure is objected to because of the following informalities:
In paragraph [0032], on line 4, “the machine learning model identify the car parts in the image” is suggested to read “the machine learning model identifies the car parts in the image”.
In paragraph [0051], on lines 3-4, “FIG. 4B shows an example depth that may be generated” is suggested to read “FIG. 4B shows an example depth map that may be generated”.
Appropriate correction is required.
Claim Objections
Claims 9 and 18 are objected to because of the following informalities:
In Claim 9, on lines 1-2, “a resting orientations” is suggested to read “resting orientations”.
In Claim 18, on line 2, “a resting orientations” is suggested to read “resting orientations”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 6, 8-11, 15, and 17-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Kollar et al. (U.S. Patent Pub. No. 20230401721 A1, hereafter referred to as Kollar) in view of Solowjow et al. (U.S. Patent Pub No. 20240296662 A1, hereafter referred to as Solowjow) in further view of Börold (NPL “Deep Learning-based Object Recognition for Counting Car Components to Support Handling and Packing Processes in Automotive Supply Chains”, 2020, hereafter referred to as Börold).
Regarding Claim 1, Kollar teaches the following:
A method (Abstract, Kollar teaches a method.) comprising: generating synthetic images of virtual car parts (Paragraphs [0073-74], Fig. 5A, Kollar teaches generating three synthetic datasets. Open GL shaders with PyRender are used instead of physically based rendering approaches to generate simulation images, for example, as shown in Figs. 5A-5C. Fig. 5A contains simulation images of cars (500). The Examiner interprets the cars in the synthetic data sets include “car parts”. For example, cars include taillights, which are “car parts”. Further, the Examiner interprets the cars (car parts) in the dataset have positions and orientations.);
PNG
media_image1.png
257
264
media_image1.png
Greyscale
[AltContent: arrow][AltContent: arrow]generating a left image and a right image associated with each synthetic image (Paragraphs [0054], [0057], Fig. 4, Kollar teaches generating non-photorealistic simulation graphics for which the synthetic stereo pair of images are generated. A left image and a right image are provided as a synthetic stereo pair of images.);
[AltContent: arrow]
PNG
media_image2.png
603
1202
media_image2.png
Greyscale
determining depth information for pixels of the synthetic images based on the left image and the right image associated with each synthetic image (Paragraph [0038], Kollar teaches passive stereo matching as an alternative to active depth sensing, which captures images from two cameras and matches pixels in each image to a single point in 3D space. A disparity (or horizontal difference in the pixel coordinates) of the single point can be directly mapped to depth.); and
training a machine learning model (Paragraphs [0030], [0064], Kollar teaches training of a neural network.) to identify a car part (Paragraphs [0075-76], Figs. 6 & 7, Kollar teaches the 3D object perception architecture (400), or “SimNet” configured to enable perception models trained on simulated data to transfer to real-world scenes. In an example, a left stereo RGB image (602) and a right stereo RGB image (604) are fed to the 3D object perception architecture (400), which produces oriented bounding boxes (OBBs) of detected vehicle objects. The scene depicted in an input image (710) includes an object (720) (e.g., a vehicle) and an object (730) (e.g., another vehicle). OBB prediction head has predicted an oriented bounding box (740a) for the object (720) and oriented bounding box (740b) for the object (730). Under BRI, the Examiner interprets the machine learning model (architecture – 400) identifies a “car part”. For example, the OBB for the car encompasses the entire car, which is made up of multiple “car parts” i.e., tires, taillights, bumper, rear window, which are each included within the OBB, and are therefore “identified”.)
PNG
media_image3.png
440
806
media_image3.png
Greyscale
using training data (Paragraphs [0013], [0073-74], Figs. 5A-5C, Kollar teaches three synthetic datasets generated to train the 3D object perception architecture (400). Simulation images for three datasets are generated: cars (500) of Fig. 5A, small objects (540) of Fig. 5B, and t-shirts of Fig. 5C.)
PNG
media_image4.png
431
975
media_image4.png
Greyscale
comprising the depth information and only one of the left image or the right image (Paragraphs [0064], [0058], Kollar teaches the 3D object perception architecture (400) uses the output of the feature extraction backbone (400) for the output prediction heads and the losses used for training the 3D object prediction architecture (400). The low-resolution disparity image (430) is fed in with features extracted from the left stereo RGB image (402) (e.g., by a feature extractor (406)) to a feature extraction backbone (440).) associated with each synthetic image (Paragraphs [0024], [0030], Kollar teaches the SimNet model may be trained entirely on simulated data to provide robust perception.).
Kollar does not explicitly disclose the following:
in virtual containers at selected positions and having selected orientations and a container in a received image
Solowjow is in the same field of art of generating synthetic datasets for training a machine learning model to perform object detection.
Further, Solowjow teaches generating synthetic images of virtual (Paragraph [0033], Fig. 3, Solowjow teaches rendering a multiplicity of images (204), where each rendered image captures a simulated environment containing objects belonging to multiple object classes of interest. The objects of interest map comprise irregular objects, such as industrial components.) in virtual containers (Paragraph [0033], Solowjow teaches rendering a multiplicity of images (204), where each rendered image (204) captures a simulated environment containing objects belonging to multiple object classes of interest placed on a receiving structure, such as a table or a bin.)
PNG
media_image5.png
519
686
media_image5.png
Greyscale
at (Paragraph [0038], Solowjow teaches the position of objects on the table or in the bin may be varied by the simulator (202) to render the images (204).) and having (Paragraph [0027], Solowjow teaches the objects (118) may be arranged in arbitrary poses within the bin (120).) and training a machine learning model (Paragraphs [0006-7], [0016], Solowjow teaches training a deep learning model for object detection and classification from input real-world images.) to identify a (Paragraphs [0052], [0033-34], Fig. 2, Solowjow teaches the deep learning model (212) is able to extract features from the rendered images (204) that transfer well to real-world images. The resulting trained deep learning model (214) is thereby configured to output object classes from an input image of a real-world physical environment. The objects of interest may comprise irregular objects, such as industrial components.) and a container in a received image (Paragraph [0028], Fig. 3, Solowjow teaches when the bin (120) contains an assortment of object types, the bin picking objective may require selectively picking objects (118) of a specified type (118) (for example, pick only “cups”). In this case, it is necessary to perform a semantic recognition of the objects (118) in the scene. Under BRI, the Examiner interprets a “cup” to be a container since it can contain liquids, items, etc.).
PNG
media_image6.png
437
724
media_image6.png
Greyscale
PNG
media_image7.png
402
429
media_image7.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar by generating/rendering randomized synthetic images of objects such as industrial components within bins to form a training dataset for a machine learning model that is taught by Solowjow, to make the invention that trains a deep learning model using the synthetic training dataset to output object classes from an input image of a real-world physical environment; thus, one of ordinary skilled in the art would be motivated to combine the references to enable specific robotic manipulation tasks such as bin picking (Solowjow, Paragraph [0026]) by training the neural network on synthetic data of various objects, after which they are able to generalize and compute grasp points on any object (Solowjow, Paragraph [0029]). Further, Kollar also states that the 3D object perception architecture can predict unseen objects for performing vision tasks such as robotic manipulation (Kollar, Paragraphs [0056], [0069]). Therefore, Kollar suggests using the model for robotic manipulation vision tasks.
Kollar in view of Solowjow does not explicitly disclose:
generating synthetic images of virtual car parts in virtual containers at selected positions and having selected orientations.
Börold is in the same field of art of deep-learning based object recognition. Further, Börold discloses:
generating (4.4 Generation of training data, Börold teaches a dataset of almost 4,000 image instances.) of car parts in selected positions and having selected orientations (4.4 Generation of training data, Börold teaches in each sequence of 30 frames, the position and orientation of the bin changes, the number and orientation of the components vary. In addition, pumps are included that lie upside down, sideways, or above another pump in the bin. The Examiner interprets the positions and orientations of the car parts (pumps) are “selected” since they are adjusted intentionally in order to add variation to the training dataset images.).
PNG
media_image8.png
281
604
media_image8.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow by replacing the objects/industrial components in the rendered images with car parts such as coolant pumps that is taught by Börold, to make the invention that utilizes a deep learning model trained using synthetic images of car parts in bins/containers to perform object recognition/detection of automotive components in standardized transport bins, specifically; thus, one of ordinary skilled in the art would be motivated to combine the references to automate the inspection in automotive operations such as packing, picking, warehouse entry and warehouse exit, which are part of a component sourcing process and involve handling of numerous identical or similar objects. Thus, it may improve order conformity of component sourcing processes in production networks in the automotive industry and decrease the related costs (Börold, 1. Introduction).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 6, Kollar in view of Solowjow in further view of Börold discloses the method of claim 1, further comprising selecting the virtual car (4.1 Experimental Setup, 4.4 Generation of Training Data, Börold teaches 130 sequences, each of 30 frames, are generated. The components (coolant pumps) are placed in the bin.) parts (Paragraph [0036], Solowjow teaches the mesh generator (210) may download, for each object class, multiple mesh representations from one or more mesh databases, and select one that provides the closest similarity metric with a real-world object of the same object class.) from among a database of potential virtual car (4.1 Experimental Setup, 4.4 Generation of Training Data, Börold teaches 130 sequences, each of 30 frames, are generated. The components (coolant pumps) are placed in the bin.) parts (Paragraph [0036], Solowjow teaches the mesh generator (210) may download a mesh representation of an object belonging to an object class of interest from a mesh database, which may be available online, such as Thingiverse, among many others.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by replacing the “objects”, such as industrial parts, in the mesh database with car parts such as coolant pumps that is taught by Börold, to make the invention that uses a deep learning-based model trained on training datasets containing synthetic car parts in synthetic bins to monitor supply chain processes that involve directly handling of individual parts and require distinguishing the type of variant of a part such as for sorting, picking, or packing; thus, one of ordinary skilled in the art would be motivated to combine the references to automate the inspection in operations such as packing, picking, warehouse entry and warehouse exit as well as improve the order conformity of component sourcing processes in production networks in the automotive industry and decrease related costs (1. Introduction, Börold).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 8, Kollar in view of Solowjow in further view of Börold discloses the method of claim 1, further comprising randomly selecting, the positions (Paragraph [0033], Solowjow teaches the images (204) are rendered by the simulator (202) by randomly placing 3D mesh representations (e.g., CAD models) of different classes of objects on a simulated tabletop or in a simulated bin.) and the orientations (Paragraph [0049], Fig. 3, Solowjow teaches the rendered simulation environment comprises a randomized arrangement of objects 302, 304, 306, 308, 310, 312, 314, and 316 belonging to five different object classes of interest, placed inside a simulated bin.) of the virtual car (4.1 Experimental Setup, 4.4 Generation of Training Data, Börold teaches 130 sequences, each of 30 frames, are generated. The components (coolant pumps) are placed in the bin.) parts in the synthetic images (Paragraph [0033], Fig. 3, Solowjow teaches rendering images (204) by randomly placing 3D mesh representations of different classes of objects on a simulated tabletop/bin. The objects of interest may comprise simple geometric objects or irregular objects, such as industrial components.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by replacing the “objects”, such as industrial parts, in the mesh database with car parts such as coolant pumps that is taught by Börold, to make the invention that uses a deep learning-based model trained on training datasets containing synthetic car parts in synthetic bins to monitor supply chain processes that involve directly handling of individual parts and require distinguishing the type of variant of a part such as for sorting, picking, or packing; thus, one of ordinary skilled in the art would be motivated to combine the references to automate the inspection in operations such as packing, picking, warehouse entry and warehouse exit as well as improve the order conformity of component sourcing processes in production networks in the automotive industry and decrease related costs (1. Introduction, Börold).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 9, Kollar in view of Solowjow in further view of Börold discloses the method of claim 1, further comprising determining resting positions (Paragraph [0033], Solowjow teaches randomly placing 3D mesh representations (e.g. CAD models) of different classes of objects on a simulated tabletop or in a simulated bin. Under BRI, the Examiner interprets “resting positions” to mean the position of the object resting on the tabletop/bin.) and a resting orientations for the virtual car (4.1 Experimental Setup, 4.4 Generation of Training Data, Börold teaches 130 sequences, each of 30 frames, are generated. The components (coolant pumps) are placed in the bin.) parts in the virtual containers (Paragraph [0033], Solowjow teaches randomly placing 3D mesh representations (e.g. CAD models) of different classes of objects on a simulated tabletop or in a simulated bin. Under BRI, the Examiner interprets “resting orientations” to mean the orientation of the object resting on the tabletop/bin.)
PNG
media_image9.png
441
426
media_image9.png
Greyscale
based on the selected positions, the selected orientations (Paragraph [0033], Solowjow teaches randomly placing 3D mesh representations (e.g., CAD models) of different classes of objects on a simulated tabletop or in a simulated bin. The Examiner interprets that the “selected positions” and “selected orientations” can be selected randomly by the simulator (“randomly placing”).), and a physics simulation (Paragraph [0033], Solowjow teaches a physics-based simulator (202) capable of rendering images in a simulation environment. The simulator (202) renders a multiplicity of images (204) where each rendered image (204) captures a simulated environment containing objects belonging to multiple object classes of interest. Examples of a physics-based simulator suitable for this application include physics engines such as Bullet™, Mujoco™, among others, that may have built-in renderers.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by replacing the “objects”, such as industrial parts, in the mesh database with car parts such as coolant pumps that is taught by Börold, to make the invention that uses a deep learning-based model trained on training datasets containing synthetic car parts in synthetic bins to monitor supply chain processes that involve directly handling of individual parts and require distinguishing the type of variant of a part such as for sorting, picking, or packing; thus, one of ordinary skilled in the art would be motivated to combine the references to automate the inspection in operations such as packing, picking, warehouse entry and warehouse exit as well as improve the order conformity of component sourcing processes in production networks in the automotive industry and decrease related costs (1. Introduction, Börold).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 10, Kollar in view of Solowjow in further view of Börold discloses the method of claim 1, further comprising: receiving the received image that includes one or more car parts (Paragraph [0075], Fig. 6, Kollar teaches a left stereo RGB image (602) and a right stereo image (604) are fed into the 3D object perception architecture (400).) in the container (Paragraphs [0033], [0052], Solowjow teaches the simulator (202) renders a multiplicity of images 204, where each rendered image (204) captures a simulated environment containing objects belonging to multiple object classes of interest placed on a receiving structure, such as a table or a bin. The rendered images (204) and the respective labels (206) can be used to train the deep learning model (212).);
PNG
media_image10.png
356
792
media_image10.png
Greyscale
inputting the received image into the trained model (Paragraph [0075], Fig. 6, Kollar teaches a left stereo RGB image (602) and a right stereo image (604) are fed into the 3D object perception architecture (400).); and identifying the one or more car parts (Paragraph [0075], Figs. 6 & 7, Kollar teaches the 3D object perception architecture (400) produces OBBs of detected vehicle objects.) and the container in the received image (Paragraph [0028], Fig. 3, Solowjow teaches when the bin (120) contains an assortment of object types, the bin picking objective may require selectively picking objects (118) of a specified type (118) (for example, pick only “cups”). In this case, it is necessary to perform a semantic recognition of the objects (118) in the scene. Under BRI, the Examiner interprets a “cup” to be a container since it can contain liquids, items, etc.) based on an output of the trained model (Paragraphs [0076], [0065], Fig. 6, Fig. 4, Kollar teaches the oriented bounding boxes (e.g., 740a, 740b) output to a 3D object detector may be cuboids (e.g., 3D bounding boxes). Also see Fig. 4, OBBs prediction head (460) outputs the predicted OBBs (462) of an image frame.).
In regards to Claim 11, Kollar teaches the following:
A computing device comprising one or more processors (Paragraph [0089], Kollar teaches a processor may be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.) configured to:
generate synthetic images of virtual car parts (Paragraphs [0073-74], Fig. 5A, Kollar teaches generating three synthetic datasets. Open GL shaders with PyRender are used instead of physically based rendering approaches to generate simulation images, for example, as shown in Figs. 5A-5C. Fig. 5A contains simulation images of cars (500). The Examiner interprets the cars in the synthetic data sets include “car parts”. For example, cars include taillights, which are “car parts”. Further, the Examiner interprets the cars (car parts) in the dataset have positions and orientations.);
generate a left image and a right image associated with each synthetic image (Paragraphs [0054], [0057], Fig. 4, Kollar teaches generating non-photorealistic simulation graphics for which the synthetic stereo pair of images are generated. A left image and a right image are provided as a synthetic stereo pair of images.);
determine depth information for pixels of the synthetic images based on the left image and the right image associated with each synthetic image (Paragraph [0038], Kollar teaches passive stereo matching as an alternative to active depth sensing, which captures images from two cameras and matches pixels in each image to a single point in 3D space. A disparity (or horizontal difference in the pixel coordinates) of the single point can be directly mapped to depth.); and
train a machine learning model (Paragraphs [0030], [0064], Kollar teaches training of a neural network.) to identify a car part (Paragraphs [0075-76], Figs. 6 & 7, Kollar teaches the 3D object perception architecture (400), or “SimNet” configured to enable perception models trained on simulated data to transfer to real-world scenes. In an example, a left stereo RGB image (602) and a right stereo RGB image (604) are fed to the 3D object perception architecture (400), which produces oriented bounding boxes (OBBs) of detected vehicle objects. The scene depicted in an input image (710) includes an object (720) (e.g., a vehicle) and an object (730) (e.g., another vehicle). OBB prediction head has predicted an oriented bounding box (740a) for the object (720) and oriented bounding box (740b) for the object (730). Under BRI, the Examiner interprets the machine learning model (architecture – 400) identifies a “car part”. For example, the OBB for the car encompasses the entire car, which is made up of multiple “car parts” i.e., tires, taillights, bumper, rear window, which are each included within the OBB, and are therefore “identified”.) using training data (Paragraphs [0013], [0073-74], Figs. 5A-5C, Kollar teaches three synthetic datasets generated to train the 3D object perception architecture (400). Simulation images for three datasets are generated: cars (500) of Fig. 5A, small objects (540) of Fig. 5B, and t-shirts of Fig. 5C.) comprising the depth information and only one of the left image or the right image (Paragraphs [0064], [0058], Kollar teaches the 3D object perception architecture (400) uses the output of the feature extraction backbone (400) for the output prediction heads and the losses used for training the 3D object prediction architecture (400). The low-resolution disparity image (430) is fed in with features extracted from the left stereo RGB image (402) (e.g., by a feature extractor (406)) to a feature extraction backbone (440).) associated with each synthetic image (Paragraphs [0024], [0030], Kollar teaches the SimNet model may be trained entirely on simulated data to provide robust perception.).
Kollar does not explicitly disclose the following:
in virtual containers at selected positions and having selected orientations and a container in a received image
Solowjow is in the same field of art of generating synthetic datasets for training a machine learning model to perform object detection.
Further, Solowjow teaches generate synthetic images of virtual (Paragraph [0033], Fig. 3, Solowjow teaches rendering a multiplicity of images (204), where each rendered image captures a simulated environment containing objects belonging to multiple object classes of interest. The objects of interest map comprise irregular objects, such as industrial components.) in virtual containers (Paragraph [0033], Solowjow teaches rendering a multiplicity of images (204), where each rendered image (204) captures a simulated environment containing objects belonging to multiple object classes of interest placed on a receiving structure, such as a table or a bin.) at (Paragraph [0038], Solowjow teaches the position of objects on the table or in the bin may be varied by the simulator (202) to render the images (204).) and having (Paragraph [0027], Solowjow teaches the objects (118) may be arranged in arbitrary poses within the bin (120).) and training a machine learning model (Paragraphs [0006-7], [0016], Solowjow teaches training a deep learning model for object detection and classification from input real-world images.) to identify a (Paragraphs [0052], [0033-34], Fig. 2, Solowjow teaches the deep learning model (212) is able to extract features from the rendered images (204) that transfer well to real-world images. The resulting trained deep learning model (214) is thereby configured to output object classes from an input image of a real-world physical environment. The objects of interest may comprise irregular objects, such as industrial components.) and a container in a received image (Paragraph [0028], Fig. 3, Solowjow teaches when the bin (120) contains an assortment of object types, the bin picking objective may require selectively picking objects (118) of a specified type (118) (for example, pick only “cups”). In this case, it is necessary to perform a semantic recognition of the objects (118) in the scene. Under BRI, the Examiner interprets a “cup” to be a container since it can contain liquids, items, etc.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar by generating/rendering randomized synthetic images of objects such as industrial components within bins to form a training dataset for a machine learning model that is taught by Solowjow, to make the invention that trains a deep learning model using the synthetic training dataset to output object classes from an input image of a real-world physical environment; thus, one of ordinary skilled in the art would be motivated to combine the references to enable specific robotic manipulation tasks such as bin picking (Solowjow, Paragraph [0026]) by training the neural network on synthetic data of various objects, after which they are able to generalize and compute grasp points on any object (Solowjow, Paragraph [0029]). Further, Kollar also states that the 3D object perception architecture can predict unseen objects for performing vision tasks such as robotic manipulation (Kollar, Paragraphs [0056], [0069]).
Kollar in view of Solowjow does not explicitly disclose:
generate synthetic images of virtual car parts in virtual containers at selected positions and having selected orientations.
Börold is in the same field of art of deep-learning based object recognition. Further, Börold discloses:
generate (4.4 Generation of training data, Börold teaches a dataset of almost 4,000 image instances.) of car parts in selected positions and having selected orientations (4.4 Generation of training data, Börold teaches in each sequence of 30 frames, the position and orientation of the bin changes, the number and orientation of the components vary. In addition, pumps are included that lie upside down, sideways, or above another pump in the bin. The Examiner interprets the positions and orientations of the car parts (pumps) are “selected” since they are adjusted intentionally in order to add variation to the training dataset images.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow by replacing the objects/industrial components in the rendered images with car parts such as coolant pumps that is taught by Börold, to make the invention that utilizes a deep learning model trained using synthetic images of car parts in bins/containers to perform object recognition/detection of automotive components in standardized transport bins, specifically; thus, one of ordinary skilled in the art would be motivated to combine the references to automate the inspection in automotive operations such as packing, picking, warehouse entry and warehouse exit, which are part of a component sourcing process and involve handling of numerous identical or similar objects. Thus, it may improve order conformity of component sourcing processes in production networks in the automotive industry and decrease the related costs (Börold, 1. Introduction).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 15, Kollar in view of Solowjow in further view of Börold discloses the computing device of claim 11, wherein the one or more processors are further configured to select the virtual car (4.1 Experimental Setup, 4.4 Generation of Training Data, Börold teaches 130 sequences, each of 30 frames, are generated. The components (coolant pumps) are placed in the bin.) parts (Paragraph [0036], Solowjow teaches the mesh generator (210) may download, for each object class, multiple mesh representations from one or more mesh databases, and select one that provides the closest similarity metric with a real-world object of the same object class.) from among a database of potential virtual car (4.1 Experimental Setup, 4.4 Generation of Training Data, Börold teaches 130 sequences, each of 30 frames, are generated. The components (coolant pumps) are placed in the bin.) parts (Paragraph [0036], Solowjow teaches the mesh generator (210) may download a mesh representation of an object belonging to an object class of interest from a mesh database, which may be available online, such as Thingiverse, among many others.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by replacing the “objects”, such as industrial parts, in the mesh database with car parts such as coolant pumps that is taught by Börold, to make the invention that uses a deep learning-based model trained on training datasets containing synthetic car parts in synthetic bins to monitor supply chain processes that involve directly handling of individual parts and require distinguishing the type of variant of a part such as for sorting, picking, or packing; thus, one of ordinary skilled in the art would be motivated to combine the references to automate the inspection in operations such as packing, picking, warehouse entry and warehouse exit as well as improve the order conformity of component sourcing processes in production networks in the automotive industry and decrease related costs (1. Introduction, Börold).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 17, Kollar in view of Solowjow in further view of Börold discloses the computing device of claim 11, wherein the one or more processors are further configured to randomly select, the positions (Paragraph [0033], Solowjow teaches the images (204) are rendered by the simulator (202) by randomly placing 3D mesh representations (e.g., CAD models) of different classes of objects on a simulated tabletop or in a simulated bin.) and the orientations (Paragraph [0049], Fig. 3, Solowjow teaches the rendered simulation environment comprises a randomized arrangement of objects 302, 304, 306, 308, 310, 312, 314, and 316 belonging to five different object classes of interest, placed inside a simulated bin.) of the virtual car (4.1 Experimental Setup, 4.4 Generation of Training Data, Börold teaches 130 sequences, each of 30 frames, are generated. The components (coolant pumps) are placed in the bin.) parts in the synthetic images (Paragraph [0033], Fig. 3, Solowjow teaches rendering images (204) by randomly placing 3D mesh representations of different classes of objects on a simulated tabletop/bin. The objects of interest may comprise simple geometric objects or irregular objects, such as industrial components.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by replacing the “objects”, such as industrial parts, in the mesh database with car parts such as coolant pumps that is taught by Börold, to make the invention that uses a deep learning-based model trained on training datasets containing synthetic car parts in synthetic bins to monitor supply chain processes that involve directly handling of individual parts and require distinguishing the type of variant of a part such as for sorting, picking, or packing; thus, one of ordinary skilled in the art would be motivated to combine the references to automate the inspection in operations such as packing, picking, warehouse entry and warehouse exit as well as improve the order conformity of component sourcing processes in production networks in the automotive industry and decrease related costs (1. Introduction, Börold).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 18, Kollar in view of Solowjow in further view of Börold discloses the computing device of claim 11, wherein the one or more processors are further configured to determine resting positions (Paragraph [0033], Solowjow teaches randomly placing 3D mesh representations (e.g. CAD models) of different classes of objects on a simulated tabletop or in a simulated bin. Under BRI, the Examiner interprets “resting positions” to mean the position of the object resting on the tabletop/bin.) and a resting orientations for the virtual car (4.1 Experimental Setup, 4.4 Generation of Training Data, Börold teaches 130 sequences, each of 30 frames, are generated. The components (coolant pumps) are placed in the bin.) parts in the virtual containers (Paragraph [0033], Solowjow teaches randomly placing 3D mesh representations (e.g. CAD models) of different classes of objects on a simulated tabletop or in a simulated bin. Under BRI, the Examiner interprets “resting orientations” to mean the orientation of the object resting on the tabletop/bin.) based on the selected positions, the selected orientations (Paragraph [0033], Solowjow teaches randomly placing 3D mesh representations (e.g., CAD models) of different classes of objects on a simulated tabletop or in a simulated bin. The Examiner interprets that the “selected positions” and “selected orientations” can be selected randomly by the simulator (“randomly placing”).), and a physics simulation (Paragraph [0033], Solowjow teaches a physics-based simulator (202) capable of rendering images in a simulation environment. The simulator (202) renders a multiplicity of images (204) where each rendered image (204) captures a simulated environment containing objects belonging to multiple object classes of interest. Examples of a physics-based simulator suitable for this application include physics engines such as Bullet™, Mujoco™, among others, that may have built-in renderers.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by replacing the “objects”, such as industrial parts, in the mesh database with car parts such as coolant pumps that is taught by Börold, to make the invention that uses a deep learning-based model trained on training datasets containing synthetic car parts in synthetic bins to monitor supply chain processes that involve directly handling of individual parts and require distinguishing the type of variant of a part such as for sorting, picking, or packing; thus, one of ordinary skilled in the art would be motivated to combine the references to automate the inspection in operations such as packing, picking, warehouse entry and warehouse exit as well as improve the order conformity of component sourcing processes in production networks in the automotive industry and decrease related costs (1. Introduction, Börold).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 19, Kollar in view of Solowjow in further view of Börold discloses the computing device of claim 11, wherein the one or more processors are further configured to: receive the received image that includes one or more car parts (Paragraph [0075], Fig. 6, Kollar teaches a left stereo RGB image (602) and a right stereo image (604) are fed into the 3D object perception architecture (400).) in the container (Paragraphs [0033], [0052], Solowjow teaches the simulator (202) renders a multiplicity of images 204, where each rendered image (204) captures a simulated environment containing objects belonging to multiple object classes of interest placed on a receiving structure, such as a table or a bin. The rendered images (204) and the respective labels (206) can be used to train the deep learning model (212).); inputting the received image into the trained model (Paragraph [0075], Fig. 6, Kollar teaches a left stereo RGB image (602) and a right stereo image (604) are fed into the 3D object perception architecture (400).); and identify the one or more car parts (Paragraph [0075], Figs. 6 & 7, Kollar teaches the 3D object perception architecture (400) produces OBBs of detected vehicle objects.) and the container in the received image (Paragraph [0028], Fig. 3, Solowjow teaches when the bin (120) contains an assortment of object types, the bin picking objective may require selectively picking objects (118) of a specified type (118) (for example, pick only “cups”). In this case, it is necessary to perform a semantic recognition of the objects (118) in the scene. Under BRI, the Examiner interprets a “cup” to be a container since it can contain liquids, items, etc.) based on an output of the trained model (Paragraphs [0076], [0065], Fig. 6, Fig. 4, Kollar teaches the oriented bounding boxes (e.g., 740a, 740b) output to a 3D object detector may be cuboids (e.g., 3D bounding boxes). Also see Fig. 4, OBBs prediction head (460) outputs the predicted OBBs (462) of an image frame.).
In regards to Claim 20, Kollar teaches the following:
A non-transitory computer readable storage medium comprising a memory storing a program that, when executed by a processor, cause the processor to:
generate synthetic images of virtual car parts (Paragraphs [0073-74], Fig. 5A, Kollar teaches generating three synthetic datasets. Open GL shaders with PyRender are used instead of physically based rendering approaches to generate simulation images, for example, as shown in Figs. 5A-5C. Fig. 5A contains simulation images of cars (500). The Examiner interprets the cars in the synthetic data sets include “car parts”. For example, cars include taillights, which are “car parts”. Further, the Examiner interprets the cars (car parts) in the dataset have positions and orientations.);
generate a left image and a right image associated with each synthetic image (Paragraphs [0054], [0057], Fig. 4, Kollar teaches generating non-photorealistic simulation graphics for which the synthetic stereo pair of images are generated. A left image and a right image are provided as a synthetic stereo pair of images.);
determine depth information for pixels of the synthetic images based on the left image and the right image associated with each synthetic image (Paragraph [0038], Kollar teaches passive stereo matching as an alternative to active depth sensing, which captures images from two cameras and matches pixels in each image to a single point in 3D space. A disparity (or horizontal difference in the pixel coordinates) of the single point can be directly mapped to depth.); and
train a machine learning model (Paragraphs [0030], [0064], Kollar teaches training of a neural network.) to identify a car part (Paragraphs [0075-76], Figs. 6 & 7, Kollar teaches the 3D object perception architecture (400), or “SimNet” configured to enable perception models trained on simulated data to transfer to real-world scenes. In an example, a left stereo RGB image (602) and a right stereo RGB image (604) are fed to the 3D object perception architecture (400), which produces oriented bounding boxes (OBBs) of detected vehicle objects. The scene depicted in an input image (710) includes an object (720) (e.g., a vehicle) and an object (730) (e.g., another vehicle). OBB prediction head has predicted an oriented bounding box (740a) for the object (720) and oriented bounding box (740b) for the object (730). Under BRI, the Examiner interprets the machine learning model (architecture – 400) identifies a “car part”. For example, the OBB for the car encompasses the entire car, which is made up of multiple “car parts” i.e., tires, taillights, bumper, rear window, which are each included within the OBB, and are therefore “identified”.) using training data (Paragraphs [0013], [0073-74], Figs. 5A-5C, Kollar teaches three synthetic datasets generated to train the 3D object perception architecture (400). Simulation images for three datasets are generated: cars (500) of Fig. 5A, small objects (540) of Fig. 5B, and t-shirts of Fig. 5C.) comprising the depth information and only one of the left image or the right image (Paragraphs [0064], [0058], Kollar teaches the 3D object perception architecture (400) uses the output of the feature extraction backbone (400) for the output prediction heads and the losses used for training the 3D object prediction architecture (400). The low-resolution disparity image (430) is fed in with features extracted from the left stereo RGB image (402) (e.g., by a feature extractor (406)) to a feature extraction backbone (440).) associated with each synthetic image (Paragraphs [0024], [0030], Kollar teaches the SimNet model may be trained entirely on simulated data to provide robust perception.).
Kollar does not explicitly disclose the following:
in virtual containers at selected positions and having selected orientations and a container in a received image.
Solowjow is in the same field of art of generating synthetic datasets for training a machine learning model to perform object detection.
Further, Solowjow teaches generate synthetic images of virtual (Paragraph [0033], Fig. 3, Solowjow teaches rendering a multiplicity of images (204), where each rendered image captures a simulated environment containing objects belonging to multiple object classes of interest. The objects of interest map comprise irregular objects, such as industrial components.) in virtual containers (Paragraph [0033], Solowjow teaches rendering a multiplicity of images (204), where each rendered image (204) captures a simulated environment containing objects belonging to multiple object classes of interest placed on a receiving structure, such as a table or a bin.) at (Paragraph [0038], Solowjow teaches the position of objects on the table or in the bin may be varied by the simulator (202) to render the images (204).) and having (Paragraph [0027], Solowjow teaches the objects (118) may be arranged in arbitrary poses within the bin (120).) and training a machine learning model (Paragraphs [0006-7], [0016], Solowjow teaches training a deep learning model for object detection and classification from input real-world images.) to identify a (Paragraphs [0052], [0033-34], Fig. 2, Solowjow teaches the deep learning model (212) is able to extract features from the rendered images (204) that transfer well to real-world images. The resulting trained deep learning model (214) is thereby configured to output object classes from an input image of a real-world physical environment. The objects of interest may comprise irregular objects, such as industrial components.) and a container in a received image (Paragraph [0028], Fig. 3, Solowjow teaches when the bin (120) contains an assortment of object types, the bin picking objective may require selectively picking objects (118) of a specified type (118) (for example, pick only “cups”). In this case, it is necessary to perform a semantic recognition of the objects (118) in the scene. Under BRI, the Examiner interprets a “cup” to be a container since it can contain liquids, items, etc.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar by generating/rendering randomized synthetic images of objects such as industrial components within bins to form a training dataset for a machine learning model that is taught by Solowjow, to make the invention that trains a deep learning model using the synthetic training dataset to output object classes from an input image of a real-world physical environment; thus, one of ordinary skilled in the art would be motivated to combine the references to enable specific robotic manipulation tasks such as bin picking (Solowjow, Paragraph [0026]) by training the neural network on synthetic data of various objects, after which they are able to generalize and compute grasp points on any object (Solowjow, Paragraph [0029]). Further, Kollar also states that the 3D object perception architecture can predict unseen objects for performing vision tasks such as robotic manipulation (Kollar, Paragraphs [0056], [0069]).
Kollar in view of Solowjow does not explicitly disclose:
generate synthetic images of virtual car parts in virtual containers at selected positions and having selected orientations.
Börold is in the same field of art of deep-learning based object recognition. Further, Börold discloses:
generate (4.4 Generation of training data, Börold teaches a dataset of almost 4,000 image instances.) of car parts in selected positions and having selected orientations (4.4 Generation of training data, Börold teaches in each sequence of 30 frames, the position and orientation of the bin changes, the number and orientation of the components vary. In addition, pumps are included that lie upside down, sideways, or above another pump in the bin. The Examiner interprets the positions and orientations of the car parts (pumps) are “selected” since they are adjusted intentionally in order to add variation to the training dataset images.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow by replacing the objects/industrial components in the rendered images with car parts such as coolant pumps that is taught by Börold, to make the invention that utilizes a deep learning model trained using synthetic images of car parts in bins/containers to perform object recognition/detection of automotive components in standardized transport bins, specifically; thus, one of ordinary skilled in the art would be motivated to combine the references to automate the inspection in automotive operations such as packing, picking, warehouse entry and warehouse exit, which are part of a component sourcing process and involve handling of numerous identical or similar objects. Thus, it may improve order conformity of component sourcing processes in production networks in the automotive industry and decrease the related costs (Börold, 1. Introduction).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claims 2-4 and 12-13 are rejected under 35 U.S.C. 103(a) as being unpatentable over Kollar et al. (U.S. Patent Pub. No. 20230401721 A1, hereafter referred to as Kollar) in view of Solowjow et al. (U.S. Patent Pub No. 20240296662 A1, hereafter referred to as Solowjow) in further view of Börold (NPL “Deep Learning-based Object Recognition for Counting Car Components to Support Handling and Packing Processes in Automotive Supply Chains”, 2020, hereafter referred to as Börold) in further view of Li et al. (NPL “Stereo R-CNN based 3D Object Detection for Autonomous Driving”, 2019, hereafter referred to Li).
Regarding Claim 2, Kollar in view of Solowjow in further view of Börold discloses the method of claim 1, further comprising: placing first bounding boxes around the virtual car parts in the synthetic images (Paragraphs [0075-76], Kollar teaches the scene depicted in an input image (710) includes an object (720) (e.g., a vehicle) and an object (730) (e.g., another vehicle). The OBBs prediction head (460) has predicted an oriented bounding box (740b) for the object (730).)
PNG
media_image11.png
495
784
media_image11.png
Greyscale
and the virtual car parts (Paragraph [0057], Kollar teaches generating the synthetic stereo pair of images (e.g., the left stereo RGB image (402) and the right stereo RGB image (404).); and training the machine learning model (Paragraph [0075], [0085], Kollar teaches the 3D object perception architecture (400), which may be referred to as simulation network (e.g., “SimNet”), and configured to enable perception models trained on simulated data to transfer to real-world scenes.) to identify car parts using the first bounding boxes (Paragraph [0085], Fig. 7, Kollar teaches SimNet is trained as a single multi-headed neural network using simulated stereo data and simulated 3D oriented bounding boxes (OBBs) as outputs.).
Kollar in view of Solowjow in further view of Börold does not explicitly disclose (placing first bounding boxes around) the virtual car parts in the left image and the right image.
Li is in the same field of art of object detection using a machine learning model. Further, Li teaches (placing first bounding boxes around) the (4. 3D Box Estimation, Li teaches bounding boxes around the car in the left and right images. Under BRI, the Examiner interprets the bounding boxes are “around” the car parts since the bounding box outlines the entire car which has car parts within it and visibly seen from the outside. For example, the car depicted in Fig. 5 (left and right) images shows taillights for example. Under BRI, the taillight on the car is shown to be within the 2D bounding box in both the left and right images. Therefore, the Examiner interprets bounding boxes are placed around the car parts in the left and right images.).
PNG
media_image12.png
684
1328
media_image12.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by placing bounding boxes around the cars (car parts) in both the left and right (synthetic) images that is taught by Li, to make the invention that recovers the accurate stereo detection to accurately localize the object depicted in the stereo image pair; thus, one of ordinary skilled in the art would be motivated to combine the references since monocular-based methods unavoidably suffer from the lack of accurate depth information (Li, Monocular-based 3D Object Detection.).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 3, Kollar in view of Solowjow in further view of Börold discloses the the method of claim 2, further comprising: placing second bounding boxes around the virtual containers in the synthetic images (Paragraph [0051], Solowjow teaches generating a label for each rendered image (204) comprises determining pixels that define a respective set of corner points, where each set of corner points corresponds to a bounding box around an object in the rendered image frame (204) belonging to one of the object classes of interest. A set of corner points (pixels) defining respective bounding boxes may be determined for each object of interest in the rendered image frame (204) and assigned a class label. The Examiner interprets since bounding boxes are assigned to each object in the rendered image, there will be 8 bounding boxes for example rendered image in Fig. 3, since it contains 8 objects. As shown in Fig. 3 below, the bin contains cups which under BRI, are being interpreted as “containers” since they can “contain”/hold liquid, objects, etc.)
PNG
media_image13.png
396
424
media_image13.png
Greyscale
and the virtual containers (Paragraphs [0033], [0060], Solowjow teaches each rendered image (204) and the respective label (206) constitute a data sample of a synthetic training dataset (208).).
Kollar in view of Solowjow in further view of Börold does not explicitly disclose (placing second bounding boxes around) the virtual containers in the left image and the right image.
Li is in the same field of art of object detection using a machine learning model. Further, Li teaches (placing second bounding boxes around) the (4. 3D Box Estimation, Li teaches bounding boxes around the car in the left and right images. Under BRI, the Examiner interprets a car to be a “container” since a car is an enclosed vehicle that carries people, seats, steering wheel, engine, cargo, etc. Container is defined as “one that contains: especially: something into which other things can be put (as for storage).” Therefore, under BRI, the Examiner interprets a car to be a container. As shown in the image below, the car contains a steering wheel. This is just one example.)
PNG
media_image14.png
92
196
media_image14.png
Greyscale
in the left image and the right image (Fig. 6, Li teaches bounding boxes placed around the cars (“containers”) in the left and right images (top image = left image), (middle image = right image).)
PNG
media_image15.png
535
532
media_image15.png
Greyscale
PNG
media_image16.png
53
1405
media_image16.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by placing bounding boxes around the cars (containers) in both the left and right (synthetic) images that is taught by Li, to make the invention that recovers the accurate stereo detection to accurately localize the object depicted in the stereo image pair; thus, one of ordinary skilled in the art would be motivated to combine the references since monocular-based methods unavoidably suffer from the lack of accurate depth information (Li, Monocular-based 3D Object Detection.).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 4, Kollar in view of Solowjow in view of Börold in further view of Li discloses the method of claim 3, further comprising training the machine learning model using supervised learning (Paragraph [0052], Solowjow teaches training a deep learning model (212) in a supervised learning regime.), wherein the first bounding boxes and the second bounding boxes (Paragraphs [0051-52], Fig. 3, Solowjow teaches the rendered images (204) and the respective labels (206) can be used to train the deep learning model. The generated label for each rendered image frame (204) includes two-dimensional representation of locations of each set of corner point pixels, and the class label assigned to each corner point of pixels. A set of corner points (pixels) defining respective bounding boxes may be determined for each object of interest in the rendered image frame (204) and assigned a class label. Fig. 3 contains 8 objects. Therefore, the Examiner interprets defining respective bounding boxes for each object in the rendered image results in 8 total bounding boxes for the example in Fig. 3.) comprise ground truth values (Paragraph [0033], Solowjow teaches since the simulator (202) has access to ground truth, it can generate respective labels (206) for the rendered images (204). The (206) label of each rendered image (204) includes a two-dimensional representation indicative of location and object classes of objects in the respective rendered image frame.).
PNG
media_image17.png
519
686
media_image17.png
Greyscale
In regards to Claim 12, Kollar in view of Solowjow in further view of Börold discloses the computing device of claim 11, wherein the one or more processors are further configured to: place first bounding boxes around the virtual car parts in the synthetic images (Paragraphs [0075-76], Kollar teaches the scene depicted in an input image (710) includes an object (720) (e.g., a vehicle) and an object (730) (e.g., another vehicle). The OBBs prediction head (460) has predicted an oriented bounding box (740b) for the object (730).) and the virtual car parts (Paragraph [0057], Kollar teaches generating the synthetic stereo pair of images (e.g., the left stereo RGB image (402) and the right stereo RGB image (404).); and train the machine learning model (Paragraph [0075], [0085], Kollar teaches the 3D object perception architecture (400), which may be referred to as simulation network (e.g., “SimNet”), and configured to enable perception models trained on simulated data to transfer to real-world scenes.) to identify car parts using the first bounding boxes (Paragraph [0085], Fig. 7, Kollar teaches SimNet is trained as a single multi-headed neural network using simulated stereo data and simulated 3D oriented bounding boxes (OBBs) as outputs.).
Kollar in view of Solowjow in further view of Börold does not explicitly disclose (placing first bounding boxes around) the virtual car parts in the left image and the right image.
Li is in the same field of art of object detection using a machine learning model. Further, Li teaches (place first bounding boxes around) the (4. 3D Box Estimation, Li teaches bounding boxes around the car in the left and right images. Under BRI, the Examiner interprets the bounding boxes are “around” the car parts since the bounding box outlines the entire car which has car parts within it and visibly seen from the outside. For example, the car depicted in Fig. 5 (left and right) images shows taillights for example. Under BRI, the taillight on the car is shown to be within the 2D bounding box in both the left and right images. Therefore, the Examiner interprets bounding boxes are placed around the car parts in the left and right images.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by placing bounding boxes around the cars (car parts) in both the left and right (synthetic) images that is taught by Li, to make the invention that recovers the accurate stereo detection to accurately localize the object depicted in the stereo image pair; thus, one of ordinary skilled in the art would be motivated to combine the references since monocular-based methods unavoidably suffer from the lack of accurate depth information (Li, Monocular-based 3D Object Detection.).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 13, Kollar in view of Solowjow in further view of Börold discloses the computing device of claim 12, wherein the one or more processors are further configured to: place second bounding boxes around the virtual containers in the synthetic images (Paragraph [0051], Solowjow teaches generating a label for each rendered image (204) comprises determining pixels that define a respective set of corner points, where each set of corner points corresponds to a bounding box around an object in the rendered image frame (204) belonging to one of the object classes of interest. A set of corner points (pixels) defining respective bounding boxes may be determined for each object of interest in the rendered image frame (204) and assigned a class label. The Examiner interprets since bounding boxes are assigned to each object in the rendered image, there will be 8 bounding boxes for example rendered image in Fig. 3, since it contains 8 objects. As shown in Fig. 3 below, the bin contains cups which under BRI, are being interpreted as “containers” since they can “contain”/hold liquid, objects, etc.) and the virtual containers (Paragraphs [0033], [0060], Solowjow teaches each rendered image (204) and the respective label (206) constitute a data sample of a synthetic training dataset (208).).
Kollar in view of Solowjow in further view of Börold does not explicitly disclose (place second bounding boxes around) the virtual containers in the left image and the right image.
Li is in the same field of art of object detection using a machine learning model. Further, Li teaches (placing second bounding boxes around) the (4. 3D Box Estimation, Li teaches bounding boxes around the car in the left and right images. Under BRI, the Examiner interprets a car to be a “container” since a car is an enclosed vehicle that carries people, seats, steering wheel, engine, cargo, etc. Container is defined as “one that contains: especially: something into which other things can be put (as for storage).” Therefore, under BRI, the Examiner interprets a car to be a container. As shown in the image below, the car contains a steering wheel. This is just one example.) in the left image and the right image (Fig. 6, Li teaches bounding boxes placed around the cars (“containers”) in the left and right images (top image = left image), (middle image = right image).)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by placing bounding boxes around the cars (containers) in both the left and right (synthetic) images that is taught by Li, to make the invention that recovers the accurate stereo detection to accurately localize the object depicted in the stereo image pair; thus, one of ordinary skilled in the art would be motivated to combine the references since monocular-based methods unavoidably suffer from the lack of accurate depth information (Li, Monocular-based 3D Object Detection.).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claims 5 and 14 are rejected under 35 U.S.C. 103(a) as being unpatentable over Kollar et al. (U.S. Patent Pub. No. 20230401721 A1, hereafter referred to as Kollar) in view of Solowjow et al. (U.S. Patent Pub No. 20240296662 A1, hereafter referred to as Solowjow) in further view of Börold (NPL “Deep Learning-based Object Recognition for Counting Car Components to Support Handling and Packing Processes in Automotive Supply Chains”, 2020, hereafter referred to as Börold) in further view of Hinterstoisser et al. (U.S. Patent No. 11,741,666 B2, hereafter referred to as Hinterstoisser) in further view of Rowell et al. (U.S. Patent Pub. No. 20200342652 A1, hereafter referred to as Rowell).
Regarding Claim 5, Kollar in view of Solowjow in further view of Börold discloses the method of claim 1, further comprising (Paragraphs [0073-74], Kollar teaches generating three synthetic datasets. Open GL shaders with PyRender are used instead of physically based rendering approaches to generate simulation images, for example, as shown in Figs. 5A-5C. Fig. 5A contains simulation images of cars (500).); and generating the virtual car parts in the synthetic images having sizes (Paragraphs [0073-74], Kollar teaches generating three synthetic datasets. Open GL shaders with PyRender are used instead of physically based rendering approaches to generate simulation images, for example, as shown in Figs. 5A-5C. Fig. 5A contains simulation images of cars (500).)
Kollar in view of Solowjow in further view of Börold does not explicitly disclose receiving an input range of sizes of virtual car parts to be included in the synthetic images; and generating the virtual car parts in the synthetic images having sizes within the input range of sizes.
Hinterstoisser is in the same field of art of generating synthetic images and training machine learning models based on synthetic images. Further, Hinterstoisser teaches (Col. 2, lines 31-67, Hinterstoisser teaches determining a range of scaling values including determining a lower bound scaling value of the scaling values and determining an upper bound scaling value of the scaling values.) to be included in the synthetic images (Col. 5, lines 26-29, Hinterstoisser teaches the synthetic images each include one or more corresponding first scale foreground objects that are each within a first range of sizes.); and generating the virtual (Col. 19, lines 66-67 through Col. 20, lines 1-13, Hinterstoisser teaches synthetic image (500A) includes, as indicated by background object descriptor (552A), a first group of background objects that are within a first size range. Synthetic image (500A) can also include one or more foreground objects, one of which is (foreground object 501). Foreground object (501) has, in synthetic image (500A) a first size that corresponds to the first size range of the background objects.).
PNG
media_image18.png
638
398
media_image18.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by generating synthetic images including objects that are in a specific range of sizes that is taught by Hinterstoisser, to make the invention that generates synthetic images of car parts of varying sizes within containers/bins for training a machine learning model to perform an object detection task; thus, one of ordinary skilled in the art would be motivated to combine the references since generating training data (rendered images) with a variety of simulated environments with randomized parameters enables the model to adapt to a real-world environment (Solowjow, Paragraph [0018]). Therefore, by adding sufficient variation to the training dataset of images, the machine learning model may be better equipped to handle real-world scenes in which a variety of objects of varying sizes may be present in the bin/container in a factory or industrial environment.
Kollar in view of Solowjow in further view of Börold in further view of Hinterstoisser does not explicitly disclose receiving an input.
Rowell is in the same field of art of generating synthetic image data for training machine learning models for computer vision (CV) applications. Further, Rowell teaches receiving an input (Paragraphs [0050-51], Rowell teaches a user interface (109) for communicating with multiple components of the synthetic image generation system (101) to control the generation of synthetic image data. The user interface (109) is a digital interface comprising digital buttons for selecting and inputting control commands to synthetic image generation system components. Current settings, the range of possible settings, and the setting type may be printed above or proximate to the set of physical buttons.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold in further view of Hinterstoisser by receiving an input from a user regarding the settings for generating the synthetic image data that is taught by Rowell, to make the invention that allows users to control particular components of the synthetic images; thus, one of ordinary skilled in the art would be motivated to combine the references since the user may want to select a specific range of sizes of car parts in the synthetic images in order to train the machine learning model to accurately identify varying sizes of objects within real-world images or scenarios. Therefore, providing synthetic images similar enough to real images to produce generalizable machine learning models (Rowell, Paragraph [0032]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 14, Kollar in view of Solowjow in further view of Börold discloses
the computing device of claim 11, wherein the one or more processors are further configured to: (Paragraphs [0073-74], Kollar teaches generating three synthetic datasets. Open GL shaders with PyRender are used instead of physically based rendering approaches to generate simulation images, for example, as shown in Figs. 5A-5C. Fig. 5A contains simulation images of cars (500).); and generate the virtual car parts in the synthetic images having sizes (Paragraphs [0073-74], Kollar teaches generating three synthetic datasets. Open GL shaders with PyRender are used instead of physically based rendering approaches to generate simulation images, for example, as shown in Figs. 5A-5C. Fig. 5A contains simulation images of cars (500).)
Kollar in view of Solowjow in further view of Börold does not explicitly disclose receive an input range of sizes of virtual car parts to be included in the synthetic images; and generate the virtual car parts in the synthetic images having sizes within the input range of sizes.
Hinterstoisser is in the same field of art of generating synthetic images and training machine learning models based on synthetic images. Further, Hinterstoisser teaches (Col. 2, lines 31-67, Hinterstoisser teaches determining a range of scaling values including determining a lower bound scaling value of the scaling values and determining an upper bound scaling value of the scaling values.) to be included in the synthetic images (Col. 5, lines 26-29, Hinterstoisser teaches the synthetic images each include one or more corresponding first scale foreground objects that are each within a first range of sizes.); and generate the virtual (Col. 19, lines 66-67 through Col. 20, lines 1-13, Hinterstoisser teaches synthetic image (500A) includes, as indicated by background object descriptor (552A), a first group of background objects that are within a first size range. Synthetic image (500A) can also include one or more foreground objects, one of which is (foreground object 501). Foreground object (501) has, in synthetic image (500A) a first size that corresponds to the first size range of the background objects.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by generating synthetic images including objects that are in a specific range of sizes that is taught by Hinterstoisser, to make the invention that generates synthetic images of car parts of varying sizes within containers/bins for training a machine learning model to perform an object detection task; thus, one of ordinary skilled in the art would be motivated to combine the references since generating training data (rendered images) with a variety of simulated environments with randomized parameters enables the model to adapt to a real-world environment (Solowjow, Paragraph [0018]). Therefore, by adding sufficient variation to the training dataset of images, the machine learning model may be better equipped to handle real-world scenes in which a variety of objects of varying sizes may be present in the bin/container in a factory or industrial environment.
Kollar in view of Solowjow in further view of Börold in further view of Hinterstoisser does not explicitly disclose receiving an input.
Rowell is in the same field of art of generating synthetic image data for training machine learning models for computer vision (CV) applications. Further, Rowell teaches receiving an input (Paragraphs [0050-51], Rowell teaches a user interface (109) for communicating with multiple components of the synthetic image generation system (101) to control the generation of synthetic image data. The user interface (109) is a digital interface comprising digital buttons for selecting and inputting control commands to synthetic image generation system components. Current settings, the range of possible settings, and the setting type may be printed above or proximate to the set of physical buttons.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold in further view of Hinterstoisser by receiving an input from a user regarding the settings for generating the synthetic image data that is taught by Rowell, to make the invention that allows users to control particular components of the synthetic images; thus, one of ordinary skilled in the art would be motivated to combine the references since the user may want to select a specific range of sizes of car parts in the synthetic images in order to train the machine learning model to accurately identify varying sizes of objects within real-world images or scenarios. Therefore, providing synthetic images similar enough to real images to produce generalizable machine learning models (Rowell, Paragraph [0032]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over Kollar et al. (U.S. Patent Pub. No. 20230401721 A1, hereafter referred to as Kollar) in view of Solowjow et al. (U.S. Patent Pub No. 20240296662 A1, hereafter referred to as Solowjow) in further view of Börold (NPL “Deep Learning-based Object Recognition for Counting Car Components to Support Handling and Packing Processes in Automotive Supply Chains”, 2020, hereafter referred to as Börold) in further view of Rowell et al. (U.S. Patent Pub. No. 20200342652 A1, hereafter referred to as Rowell).
In regards to Claim 7, Kollar in view of Solowjow in further view of Börold discloses the method of claim 6.
Kollar in view of Solowjow in further view of Börold does not explicitly disclose randomly selecting the virtual car parts from among the database of potential virtual car parts.
Rowell is in the same field of art of generating synthetic image data for training machine learning models for computer vision (CV) applications. Further, Rowell teaches randomly selecting (Paragraph [0062], Rowell teaches the image scene generator (124) randomly selects objects from a library of 3D models.) the virtual car parts (Paragraphs [0058], [0060], Rowell teaches foreground objects are selected from a 3D library including 36,000 or more 3D objects including cars.) from among the database (Paragraphs [0100], [0104-105], Fig. 8, reference character “802”, Rowell teaches an object database.) of potential virtual car parts (Paragraph [0058], Rowell teaches 3D objects include cars.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by randomly selecting objects from an object database that is taught by Rowell, to make the invention that sufficiently varies/creates variation in the synthetic images for the training dataset by including different types of car parts in the synthetic training images; thus, one of ordinary skilled in the art would be motivated to combine the references to generate highly varied synthetic images to avoid manual overfitting problems caused by truncated or highly specific image datasets (Rowell, Paragraph [0030]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 16, Kollar in view of Solowjow in further view of Börold discloses the computing device of claim 15.
Kollar in view of Solowjow in further view of Börold does not explicitly disclose wherein the one or more processors are further configured to randomly select the virtual car parts from among the database of potential virtual car parts.
Rowell is in the same field of art of generating synthetic image data for training machine learning models for computer vision (CV) applications. Further, Rowell teaches wherein the one or more processors (Paragraph [0038], Rowell teaches The image computing device 100 may further include, a processor (113).) are further configured to randomly select (Paragraph [0062], Rowell teaches the image scene generator (124) randomly selects objects from a library of 3D models.) the virtual car parts (Paragraphs [0058], [0060], Rowell teaches foreground objects are selected from a 3D library including 36,000 or more 3D objects including cars.) from among the database (Paragraphs [0100], [0104-105], Fig. 8, reference character “802”, Rowell teaches an object database.) of potential virtual car parts (Paragraph [0058], Rowell teaches 3D objects include cars.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Kollar in view of Solowjow in further view of Börold by randomly selecting objects from an object database that is taught by Rowell, to make the invention that sufficiently varies/creates variation in the synthetic images for the training dataset by including different types of car parts in the synthetic training images; thus, one of ordinary skilled in the art would be motivated to combine the references to generate highly varied synthetic images to avoid manual overfitting problems caused by truncated or highly specific image datasets (Rowell, Paragraph [0030]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Shahapurkar et al. (U.S. Patent Pub. No. 20250303558 A1) teaches methods, systems, and apparatuses that generate synthetic datasets for training grasping neural networks.
Dudzik et al. (U.S. Patent Pub. No. 20210150278 A1) teaches techniques for training a machine learned (ML) model to determine depth data based on image data are discussed herein. Training can use stereo image data and depth data (e.g., lidar data). A first (e.g., left) image can be input to a ML model, which can output predicted disparity and/or depth data. The predicted disparity data can be used with second image data (e.g., a right image) to reconstruct the first image.
Holz et al. (NPL “Real-time object detection, localization and verification for fast robotic depalletizing”, 2015) teaches a system for depalletizing and a complete pipeline for detecting and localizing objects as well as verifying that the found object does not deviate from the known object model, e.g., if it is not the object to pick.
Torres et al. (NPL “Bin-Picking Solution for Randomly Placed Automotive Connectors Based on Machine Learning Techniques”, 2022) teaches a bin-picking solution based on low-cost vision systems for the manipulation of automotive electrical connectors using machine learning techniques.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SYDNEY L BLACKSTEN whose telephone number is (571)272-7651. The examiner can normally be reached 8:30am-4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached at 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SYDNEY L BLACKSTEN/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights