DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The Applicant’s Remarks filed 26 February 2026 have been received and considered.
The IDS filed 15 May 2026 has been received and considered.
Claims 1 – 19 are pending.
Claims 1, 2, 4 – 5, 10 – 11, 13, 15, 17, and 19 have been amended.
Claim 3 has been canceled.
Claims 1, 2, and 4 – 19, all of the remaining claims pending in this application, have been rejected.
The objection to the Specification (Abstract) in the Non-Final Office Action mailed 04 December 2025 has been withdrawn.
Response to Applicant’s Remarks
In view of the Applicant’s remarks filed 26 February 2026, regarding amendments to independent claims 1 and 10, the previously applied prior art rejections are withdrawn. Applicant's remarks are rendered moot in view of the new grounds of rejection set forth below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 – 2, 4 – 7, and 10 – 15 are rejected under 35 U.S.C. 103 as being unpatentable over Non-Patent Literature "Safe Robotic Manipulation to Extract Objects from Piles: From 3D Perception to Object Selection" to Mojtahedzadeh et al. (hereinafter Mojtahedzadeh) in view of Non-Patent Literature "GeneSIS-Rt: Generating Synthetic Images for Training Secondary Real-World Tasks" to Stein et al. (hereinafter Stein) in further view of “3D OBJECT REPRESENTATION FOR PHYSICS SIMULATION ENGINES AND ITS EFFECT ON VIRTUAL ASSEMBLY TASKS” to Gonzalez et al. (hereinafter Gonzalez).
Claim 1
Regarding Claim 1, Mojtahedzadeh teaches a method for generating a learning model for use in machine learning to automatically examine [[the]] a number of target objects accommodated in a container, the method comprising: by a device that generates the learning model, inputting model data which represents a shape of the container and a shape of a target object in an image (Figures 3.2 and 3.3; "The process starts by sensing the environment (i.e., the scene of objects) using perception sensors. The sensor fusion and pre-processing are then performed to reduce the noise and prepare the data for high level processing. Scene segmentation further prepares the input data for object detection and pose estimation algorithms.", Section 1.3 - Challenges; "The evaluation process starts with capturing a 3D scan of the target scene by a set of 3D range sensors. The captured data is then fed to the performance indicators where the object detection and pose estimation algorithms attempt to find the best match of the given objects templates to the captured data and estimate the poses.", Section 3.2.2 - Evaluation Methodology; "For collecting data, a set of different arrangements of two selected objects (i.e., carton boxes and tires) inside a mock-up container was used to generate several data sets (see Figure 3.3).", Section 3.2.3 Data Collection);
PNG
media_image1.png
271
448
media_image1.png
Greyscale
PNG
media_image2.png
320
455
media_image2.png
Greyscale
creating, by using the model data, a plurality of unit formative assemblies each having a plurality of target objects arranged in a specific array and arranging the unit formative assemblies in a container area corresponding to the container in a specific formation to create shape image data of the container accommodating the target objects at a specific density (Figures 5.8 and 5.9; "For the simulated configurations, a scene generator based on physics simulation was developed. The simulator generates random configurations of polyhedron shaped objects inside a simulated container…Test configurations generated in simulation contained three types of objects, namely, carton boxes (CBX), cylinders (CYL) and barrels (BRL). A circle in a shape is approximated by a convex polygon with 36 equal length edges. A cuboid represents the shape of a carton box. A cylinder with the approximated circles represents a cylinder object. Two semi-cones with the approximated circles construct the shape of a barrel (see Figure 5.8)…The total number of configurations generated in simulation is 1600, which is the result of multiplying 4 categories of objects, 40 configurations per category, and 10 different sets of values uniformly drawn from the dimensions of the shapes of the objects per configuration.", Section 5.5.1 - Simulated Configurations).
PNG
media_image3.png
261
453
media_image3.png
Greyscale
PNG
media_image4.png
385
450
media_image4.png
Greyscale
Mojtahedzadeh does not teach creating training image data for use in establishing the learning model by applying processing of giving a real texture of each of the container and the target object to the shape image data.
However, Stein teaches creating training image data for use in establishing the learning model by applying processing of giving a real texture of each of the container and the target object to the shape image data (Figure 1), where the simulated images are mapped to unlabeled real images to generate a realistic synthetic training data as seen in step 3.
PNG
media_image5.png
355
443
media_image5.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Mojtahedzadeh to incorporate mapping real images to simulated images to generate realistic synthetic data that can be used for training, as disclosed by Stein. The suggestion/motivation for doing so would have been to demonstrate that learning algorithms trained using data generated by GeneSIS-RT make high-accuracy predictions and outperform systems trained on raw simulated data alone, and as well or better than those trained on real data, as disclosed by Stein in the Abstract.
Neither Mojtahedzadeh, or Stein, or the combination teach wherein a specific number of the target objects are induced to freefall into the container area in accordance with a physical simulation to come in the container area in the specific array.
Examiner notes that while Mojtahedzadeh wherein a specific number of the target objects are simulated into the container area in a specific array, the induced freefall of the objects is not explicitly taught.
However, Gonzalez teaches simulated objects being induced to a freefall using physical based modeling (PMB) and physics simulation engines (PSE) (“In order to assess its influence on integrator performance, a virtual free fall experiment was performed (Fig.14) using three objects with different shape and complexity, each model being represented in the PSE using different algorithms as follows: the first object is a box represented by primitive, GIMPACT and ConvexFT, the second object is a pin, represented by Primitive, GIMPACT and ConvexFT, and finally the third model is a gear, represented by GIMPACT, ConvexFT, HACD and ACD. Each virtual object is dropped from a height of 500 units and the falling time is measured. Table 2 shows the results of the free fall virtual tests.”, Section 6 – Performance Evaluation: Integrator Performance).
PNG
media_image6.png
175
195
media_image6.png
Greyscale
PNG
media_image7.png
223
357
media_image7.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the teachings of Mojtahedzadeh, in view of Stein, to incorporate testing of the freefall of the various simulated objects of Mojtahedzadeh, as disclosed by Gonzalez, to acquire the freefall data of each respective object. The suggestion/motivation for doing so would have been by including the freefall data of the objects with various sizes, the simulation would then be able to provide a more accurate array of how the objects would have landed in the container. Also, one of ordinary skill in the art, would further be able to identify how the objects would fall with respect to gravitational force, including how the objects act on other objects after landing, as disclosed in Mojtahedzadeh (Section 5.2 – Extracting Support Relations). Therefore, in an induced freefall, both gravity and an object's shape would determine how the objects land with gravity determining how fast the objects fall, the direction they accelerate, and how the objects act on other objects after landing, while the distinct shape of the objects (along with mass distribution) determines each objects’s aerodynamics, orientation, and stability as it plummets, further increasing the accuracy of the simulation.
Claim 2
Regarding Claim 2, dependent on claim 1, Mojtahedzadeh, in view of Stein and Gonzalez, teaches the invention as claimed in claim 1.
Mojtahedzadeh, in view of Stein and Gonzalez, further teaches wherein each of the unit formative assemblies is created by presetting a confined area which is smaller than the container area, and arranging a specific number of the target objects in the confined area by the freefall (Rejected as applied to claim 1).
Claim 4
Regarding Claim 4, dependent on claim 2, Mojtahedzadeh, in view of Stein and Gonzalez, teaches the invention as claimed in claim 2.
Mojtahedzadeh further teaches wherein the step of creating of the shape image data includes: setting a mixture area which is equal to or smaller than the container area and larger than the confined area, and arranging the unit formative assemblies in the mixture area in a specific formation (Rejected as applied to claim 1); and
arranging the mixture area including the unit formative assemblies in the container area at a specific position in a specific direction (Figures 5.2 and 5.3; "The contact points are computed based on the available geometrical information (shape and pose) of the objects. The geometrical consistency of configurations, as discussed in Chapter 4 suggests that the shapes of two adjacent objects cannot penetrate into each other. Among six possibilities, four types of geometrically possible contacts between two adjacent objects are considered and computed", Section 5.2.1 Contact Point-Set Network).
PNG
media_image8.png
230
527
media_image8.png
Greyscale
PNG
media_image9.png
340
517
media_image9.png
Greyscale
Claim 5
Regarding Claim 5, dependent on claim 1, Mojtahedzadeh, in view of Stein and Gonzalez, teaches the invention as claimed in claim 1.
Mojtahedzadeh, in view of Stein and Gonzalez, further teaches wherein the unit formative assemblies are induced to freefall from a fall start position where the target objects stay in a specific array into the container area in accordance with [[a]]the physical simulation to come in the container area in the specific formation (Rejected as applied to claims 1 and 4).
Claim 6
Regarding Claim 6, dependent on claim 1, Mojtahedzadeh, in view of Stein and Gonzalez, teaches the invention as claimed in claim 1.
Mojtahedzadeh further teaches defining information indicating a position of each of the target objects in the shape image data as true data indicating a position of the target object in the training image data (Figure 4.4; "This section presents results showing the performance of the object pose refinement approach on both simulated and real-world data. Using scenarios generated in simulation enables us to create a large data set of different configurations of objects with their ground truth poses to capture the statistical properties of the approach. The real-world configurations are used to verify the approach on real data.", Section 4.3 - Results);
PNG
media_image10.png
268
514
media_image10.png
Greyscale
and causing a storage included in the device that generates the learning model to store the training image data and the true data in association with each other (Rejected as applied directly above).
Claim 7
Regarding Claim 7, dependent on claim 1, Mojtahedzadeh, in view of Stein and Gonzalez, teaches the invention as claimed in claim 1.
Mojtahedzadeh, in view of Stein, further teaches wherein the processing of giving the texture is executed by physically based rendering including: a setting of a photographic optical system for each of the target objects and the container, and a setting of a variation range of the photographic optical system (Rejected as applied to claim 1), wherein the combination of Mojtahedzadeh, Stein, and Gonzalez makes obvious for one ordinarily skilled in the art to take the simulated images of objects in a container and map them to unlabeled real image data to generate the realistic synthetic data to allow for a more accurate representation of the objects; and
a setting of a material of each of the target object and the container, and a setting of a variation range of the material (Rejected as applied directly above).
Claim 10, an independent non-transitory computer readable medium claim, is rejected for the same reasons as applied to claim 1.
Claims 11 – 15 are rejected for the same reasons as applied to the above claims.
Claims 8 – 9 and 16 – 19 are rejected under 35 U.S.C. 103 as being unpatentable over Non-Patent Literature "Safe Robotic Manipulation to Extract Objects from Piles: From 3D Perception to Object Selection" to Mojtahedzadeh et al. (hereinafter Mojtahedzadeh) in view of Non-Patent Literature "GeneSIS-Rt: Generating Synthetic Images for Training Secondary Real-World Tasks" to Stein et al. (hereinafter Stein) in further view of “3D OBJECT REPRESENTATION FOR PHYSICS SIMULATION ENGINES AND ITS EFFECT ON VIRTUAL ASSEMBLY TASKS” to Gonzalez et al. (hereinafter Gonzalez) in further view of Non-Patent Literature "On Pre-Trained Image Features and Synthetic Images for Deep Learning" to Hinterstoisser et al. (hereinafter Hinterstoisser).
Claim 8
Regarding Claim 8, dependent on claim 1, Mojtahedzadeh, in view of Stein and Gonzalez, teaches the invention as claimed in claim 1.
Neither Mojtahedzadeh or Gonzalez teach updating the learning model by creating another training image data reflecting a feature of the actual image when a similarity between the training image data and the actual image is lower than a predetermined threshold.
However, Stein further teaches updating the learning model by creating another training image data reflecting a feature of the actual image when a similarity between the training image data and the actual image is lower than a predetermined threshold (Section II: LEARNING THE MAPPING FUNCTION, Part A - The CycleGAN Procedure).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the teachings of Mojtahedzadeh, in view of Stein and Gonzalez, to incorporate creating another training image data based on a comparison between the real images and the simulated images, as disclosed by Stein. The suggestion/motivation for doing so would have been to perform image analysis with a higher accuracy or allow for a better performance from robotic functions.
Neither Mojtahedzadeh, or Stein, or Gonzalez, or the combination teach comparing the training image data with an actual image of the container accommodating the target objects, the actual image being actually acquired in the automatic examination of the number of target objects.
However, Hinterstoisser teaches comparing the training image data with an actual image of the container accommodating the target objects, the actual image being actually acquired in the automatic examination of the number of target objects (Figure 6; "We then compared the distributions of the Euclidean distances between image features generated for the real images and the corresponding synthetic images. As we can see Fig. 6(c), the distribution is much more clustered around 0 when the features are computed using a frozen feature extractor pre-trained on real images (red) compared to the distribution obtained when the pre-trained feature extractor is finetuned on synthetic images (blue).", Section 4.3: Freezing the Feature Extractor).
PNG
media_image11.png
276
514
media_image11.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the teachings of Mojtahedzadeh, in view of Stein and Gonzalez, to incorporate comparing a real image and a synthetic version of that image, as disclosed by Hinterstoisser. The suggestion/motivation for doing so would have been to calculate the Euclidean distance as a means of determining their similarity and determining form if they are similar enough too dissimilar to be used for training.
Claim 9
Regarding Claim 9, dependent on claim 1, Mojtahedzadeh, in view of Stein and Gonzalez, teaches the invention as claimed in claim 1.
Neither Mojtahedzadeh, or Stein, or Gonzalez¸or the combination teach wherein the creating of the shape image data includes arranging in the container area an unacceptable object other than the target object.
However, Hinterstoisser teaches wherein the creating of the shape image data includes arranging in the container area an unacceptable object other than the target object (Figure 3; "The object is rendered at a random location in a randomly selected background image using a uniform distribution. The selected background image is part of a large collection of highly cluttered real background images taken with the camera of choice where the objects of interest are not included. To increase the variability of the background image set, we randomly swap the three background image channels and randomly flip and rotate the images (0 ◦ , 90◦ , 180◦ and 270◦ ). We also tried to work without using real background images and experimented with backgrounds only exhibiting one randomly chosen color, however, that did not lead to good results.", Section 3.1 Synthetic Data Generation Pipeline).
PNG
media_image12.png
276
486
media_image12.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the teachings of Mojtahedzadeh, in view of Stein and Gonzalez, to incorporate creating images containing undesired noise/objects while including the target object, as disclosed by Hinterstoisser. The suggestion/motivation for doing so would have been to determine just how accurately a model could distinguish between targeted objects all while reducing the amount of noise detected.
Claims 16 and 17 are rejected for the same reasons as applied to claim 8.
Claims 18 and 19 are rejected for the same reasons as applied to claim 9.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ronde Miller whose telephone number is (703) 756-5686 The examiner can normally be reached Monday-Friday 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Gregory Morse can be reached on (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RONDE LEE MILLER/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights