DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged.
Specification
The disclosure is objected to because of the following informalities:
See Specification paragraph [0038], where a typographical error appears to exists, and “Fig. 21” is assumed to be intended to recite “Fig. 2A”.
Appropriate correction is required.
Claim Objections
Claim 6 is objected to because of the following informalities:
Claim 6 appears to recite a typographical error in the body of the claim, and “wherein the one or more composite images are generated based at least on determining” is assumed to be intended.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over ReMine et al. (US 2020/0192389, effectively filed 14 Dec. 2018), herein ReMine, in view of Tsirikoglou et al. (“Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications”), herein Tsirikoglou
Regarding claim 1, ReMine discloses a system comprising:
one or more processing units (see ReMine [0054]-[0055], where processors are disclosed to implement the disclosed teachings) to execute operations including:
determining search criteria based at least on one or more conditions identified in one or more real-world images (see ReMine [0044]-[0045], where images of real world scene are accessed a semantically labeled mask is produced for the real world scene);
searching one or more simulated images that depict instances of one or more objects for a subset of the instances (see ReMine [0046], where one or more images of a simulated object corresponding to a real world object is generated);
inserting the subset of the instances into the one or more real-world images to generate one or more composite images (see ReMine [0047]-[0048], where real world scene images are procedurally generated which injects real and simulated objects into the real-world scene); and
applying the one or more composite images to one or more machine learning models (MLMs) (see ReMine [0044][0049], where the realistic images of the real-world scene can be used as a training set of images of the real world scene for an AI system).
ReMine does not explicitly disclose that the one or more objects are depicted in one or more virtual environments under the one or more conditions that satisfy the search criteria.
Tsirikoglou teaches in a related and pertinent method for a procedural world modeling approach coupled with physically accurate image synthesis (see Tsirikoglou Abstract), where the images are procedurally generated with unique virtual worlds, where a set of parameters is used parameterizes all aspects of the 3D world generation and the image synthesis, such as road width, Sun, cloud cover amount, and that a mixture of procedurally generated geometry and model libraries are used (see Tsirikoglou sect. 3. Method and sect. 3.2.World generation using procedural modeling).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Tsirikoglou to the teachings of ReMine, such that the synthesized objects are also depicted in synthesized virtual environments corresponding to the real world scene.
This modification is rationalized as an application of a known technique to a known system ready for improvement to yield predictable results.
In this instance, ReMine disclose a base system for generating simulated images of the real world scene, where one or more images of a simulated object corresponding to a real world object are generated.
Tsirikoglou teaches a known technique of procedurally generating synthesized images, where the images are procedurally generated with unique virtual worlds, where a set of parameters is used parameterizes all aspects of the 3D world generation and the image synthesis, such as road width, Sun, cloud cover amount, and that a mixture of procedurally generated geometry and model libraries are used.
One of ordinary skill in the art would have recognized that by applying Tsirikoglou's technique would allow for the synthesis of simulated objects and virtual scenes corresponding to real world object and scenes, predictably leading to an improved system for generating simulated images of the real world scene.
Regarding claim 2, please see the above rejection of claim 1. ReMine and Tsirikoglou disclose the system of claim 1, wherein the one or more simulated images include an image in which multiple instances of an object are depicted under different values for a condition of the one or more conditions, and the searching selects, for the subset of the instances, a subset of the multiple instances of the object that satisfy the search criteria based at least on the values for the condition (see ReMine [0046], where one or more images of a simulated object corresponding to a real world object is generated, where the simulated object may include different specified poses of the simulated object ).
Regarding claim 3, please see the above rejection of claim 1. ReMine and Tsirikoglou disclose the system of claim 1, wherein the one or more simulated images include multiple images in which multiple instances of an object are depicted under different conditions in different images, and the searching selects, for the subset of the instances, a subset of the multiple instances of the object that satisfy the one or more conditions (see ReMine [0046], where one or more images of a simulated object corresponding to a real world object is generated, where the simulated object may include different specified poses of the simulated object).
Regarding claim 4, please see the above rejection of claim 1. ReMine and Tsirikoglou disclose the system of claim 1, further comprising:
generating the instances of the one or more objects in the one or more virtual environments using set increments for values of a condition of the one or more conditions (see ReMine [0046], where one or more images of a simulated object corresponding to a real world object is generated, where the simulated object may include different specified poses of the simulated object; see Tsirikoglou sect. 3. Method and sect. 3.2.World generation using procedural modeling, where the set of parameters are used to parameterizes all aspects of the 3D world generation and the image synthesis, such as road width, Sun, cloud cover amount); and
rendering the instances of the one or more objects in the one or more virtual environments to generate the one or more simulated images (see ReMine [0047]-[0048], where real world scene images are procedurally generated which injects real and simulated objects into the real-world scene).
Regarding claim 5, please see the above rejection of claim 1. ReMine and Tsirikoglou disclose the system of claim 1, wherein the one or more conditions include one or more of:
a weather condition; a lighting condition; a location condition; a time of day condition; a visibility distance condition; or a virtual camera distance condition (see Tsirikoglou sect. 3. Method and sect. 3.2.World generation using procedural modeling, where the set of parameters are used to parameterizes all aspects of the 3D world generation and the image synthesis, such as road width, Sun, and cloud cover amount).
Regarding claim 6, please see the above rejection of claim 1. ReMine and Tsirikoglou disclose the system of claim 1, wherein the one or more composite images are generated based at east on determining that an instance of the subset of the instances is inserted into a real-world image of the one or more real-world images within a threshold vertical distance from a driving surface depicted in the real-world image (see ReMine [0047]-[0048], where real world scene images are procedurally generated which injects real and simulated objects into the real-world scene; see Tsirikoglou Fig. 2, where the objects are rendered above the road semantic segments).
Regarding claim 7, please see the above rejection of claim 1. ReMine and Tsirikoglou disclose the system of claim 1, wherein the one or more composite images are generated based at least on determining that first pixels in a bounding shape of an instance of the subset of the instances inserted into a real-world image of the one or more real-world images are not overlapping second pixels corresponding to a real-world object identified in the real-world image (see ReMine [0031], where semantic segmentation of the images of the real world scene is performed to produce a mask of the real-world scene including segments of pixels assigned to object classes for respective objects in the images of the real-world scene for inserting the simulated object into the real world scene properly).
Regarding claim 8, please see the above rejection of claim 1. ReMine and Tsirikoglou disclose the system of claim 1, wherein the determining of the search criteria further includes:
randomly sampling values of a condition of the at least one condition identified in the one or more real-world images (see ReMine [0045], where the segmentation model can identify areas or objects of interest in the images; and see Tsirikoglou sect. 3. Method and sect. 3.2.World generation using procedural modeling, where the set of parameters are used to parameterizes all aspects of the 3D world generation and the image synthesis, where each image instantiates an entirely unique virtual world); and
based at least on the randomly sampling, including one or more criterion corresponding to the values in the search criteria (see Tsirikoglou sect. 3. Method and sect. 3.2.World generation using procedural modeling, where the set of parameters are used to parameterizes all aspects of the 3D world generation and the image synthesis, where each image instantiates an entirely unique virtual world).
Regarding claim 9, please see the above rejection of claim 1. ReMine and Tsirikoglou disclose the system of claim 1, further comprising generating a segmentation mask corresponding to an instance of the subset of the instances in a simulated image of the one or more simulated images and the inserting uses the segmentation mask to identify pixels corresponding to the instance in the simulated image for inclusion in at least one real-world image of the one or more real-world images (see ReMine [0031], where semantic segmentation of the images of the real world scene is performed to produce a mask of the real-world scene including segments of pixels assigned to object classes for respective objects in the images of the real-world scene).
Regarding claim 10, please see the above rejection of claim 1. ReMine and Tsirikoglou disclose the system of claim 1, wherein the system is comprised in at least one of:
a system for performing simulation operations;
a system for performing deep learning operations (see ReMine [0044] and [0049], where simulated images of the real-world scene are provided as training data for an AI system);
a system implemented using an edge device;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
Regarding claim 11, ReMine and Tsirikoglou disclose a method comprising:
identifying one or more conditions corresponding to one or more real-world images (see ReMine [0044]-[0045], where images of real world scene are accessed and a semantically labeled mask is produced for the real world scene);
based at least on the identifying, determining one or more criteria for the one or more conditions (see ReMine [0044]-[0045], where a semantically labeled mask is produced for the real world scene);
searching one or more simulated images for instances of one or more objects that are depicted in one or more virtual environments and match the one or more criteria for the one or more conditions (see ReMine [0046], where one or more images of a simulated object corresponding to a real world object is generated; see Tsirikoglou sect. 3. Method and sect. 3.2.World generation using procedural modeling, where the set of parameters are used to parameterizes all aspects of the 3D world generation and the image synthesis, where each image instantiates an entirely unique virtual world; where the combined teachings suggests the synthesized objects are also depicted in synthesized virtual environments corresponding to the real world scene);
inserting the subset of the instances into the one or more real-world images to generate one or more composite images (see ReMine [0047]-[0048], where real world scene images are procedurally generated which injects real and simulated objects into the real-world scene); and
updating one or more parameters of one or more machine learning models (MLMs) using the one or more composite images (see ReMine [0044][0049], where the realistic images of the real-world scene can be used as a training set of images of the real world scene for an AI system).
Please see the above rejection for claim 1, as the rationale to combine the teachings of ReMine and Tsirikoglou are similar, mutatis mutandis.
Regarding claim 12, see above rejection for claim 11. It is a method claim reciting similar subject matter as claim 2. Please see above claim 2 for detailed claim analysis as the limitations of claim 12 are similarly rejected.
Regarding claim 13, see above rejection for claim 11. It is a method claim reciting similar subject matter as claim 3. Please see above claim 3 for detailed claim analysis as the limitations of claim 13 are similarly rejected.
Regarding claim 14, see above rejection for claim 11. It is a method claim reciting similar subject matter as claim 4. Please see above claim 4 for detailed claim analysis as the limitations of claim 14 are similarly rejected.
Regarding claim 15, ReMine and Tsirikoglou disclose a processor comprising: one or more circuits to implement one or more machine learning models (MLMs) trained using one or more training images (see ReMine [0044][0049], where the realistic images of the real-world scene can be used as a training set of images of the real world scene for an AI system; and see ReMine [0054]-[0055], where processors are disclosed to implement the disclosed teachings), the one or more training images generated based at least on:
determining search criteria based at least on at least one condition identified in one or more real-world images (see ReMine [0044]-[0045], where images of real world scene are accessed a semantically labeled mask is produced for the real world scene);
searching one or more simulated images that depict instances of one or more objects for a subset of the instances in which the one or more objects are depicted in one or more virtual environments under one or more conditions that satisfy the search criteria (see ReMine [0046], where one or more images of a simulated object corresponding to a real world object is generated; see Tsirikoglou sect. 3. Method and sect. 3.2.World generation using procedural modeling, where the set of parameters are used to parameterizes all aspects of the 3D world generation and the image synthesis, where each image instantiates an entirely unique virtual world; where the combined teachings suggests the synthesized objects are also depicted in synthesized virtual environments corresponding to the real world scene); and
inserting the subset of the instances into the one or more real-world images to generate one or more training images (see ReMine [0047]-[0048], where real world scene images are procedurally generated which injects real and simulated objects into the real-world scene).
Please see the above rejection for claim 1, as the rationale to combine the teachings of ReMine and Tsirikoglou are similar, mutatis mutandis.
Regarding claim 16, see above rejection for claim 15. It is a processor claim reciting similar subject matter as claim 2. Please see above claim 2 for detailed claim analysis as the limitations of claim 16 are similarly rejected.
Regarding claim 17, see above rejection for claim 15. It is a processor claim reciting similar subject matter as claim 3. Please see above claim 3 for detailed claim analysis as the limitations of claim 17 are similarly rejected.
Regarding claim 18, see above rejection for claim 15. It is a processor claim reciting similar subject matter as claim 4. Please see above claim 4 for detailed claim analysis as the limitations of claim 18 are similarly rejected.
Regarding claim 19, see above rejection for claim 15. It is a processor claim reciting similar subject matter as claim 5. Please see above claim 5 for detailed claim analysis as the limitations of claim 19 are similarly rejected.
Regarding claim 20, see above rejection for claim 15. It is a processor claim reciting similar subject matter as claim 10. Please see above claim 10 for detailed claim analysis as the limitations of claim 20 are similarly rejected.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIMOTHY WING HO CHOI whose telephone number is (571)270-3814. The examiner can normally be reached 9:00 AM to 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VINCENT RUDOLPH can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIMOTHY CHOI/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights