Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.All claims have priority date of 03/05/2020.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 5/202/2026 has been entered.
Claims 1, 9-10 are amended, claims 11-24 previously canceled; claims 31-33 newly added; claims 1-10 and 25-33 remain pending in this application.
Response to Arguments
Applicant’s arguments with respect to all claims have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Regarding applicants remarks that Collins does not teach training an artificial intelligence (AI) model to output an ideal image when a deteriorated image is input, examiner notes that the training of the AI model nor use of AI model to output an image is positively recited, thus applicant argues features which are not claimed.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 6-10, 25-27, and 29-33 are rejected under 35 U.S.C. 103 as obvious over US 20200005083 A1 Collins; Robert. In view of US 20190294923 A1 Riley; Ian et al.
Consider Claims 1, 9 and 10
Collins teaches An image generation method (Collins Fig. 1, Fig. 4, [0007] "a computer implemented method for generating a dataset having a multiplicity of corresponding images ... ), comprising:
acquiring a computer graphics (CG) model (¶15 “a 3D rendering engine”), including a representation of a three-dimensional (3D) object in an environment (Collins Fig. 1, Fig. 4, ¶15 " ... loading a further seed model into a 3D rendering engine"; [0044] "The seed model 110 may be any convenient data representation of the object to be recognized, although CAD data is preferred ... “);
and performing, by a processor, processing on the CG model (¶7 “a 3D rendering engine”) using a first imaging model ([0007] “selecting a first lighting arrangement”) to generate a first image (Collins Fig. 1, Fig. 4 [0007] " ... rendering the first view of the seed model with the first lighting arrangement... storing a 2D image of the first view of the seed model… “) wherein the first image is a deteriorated image (¶65 “the properties of the virtual camera 160 may comprise one or more camera artefacts. This may improve recognition accuracy and/or reliability as real-world problems may be anticipated. These artifacts may include, for example, blur, lighting flare, defocus, translation, rotation, scaling, obscuration, noise, motion, pixelization, edge clarity, aliasing, image compression, color banding, grain, quantization, macro blocking, mosaicking (or mosaicking), ringing, posterizing, and any combination thereof. It may be an artefact related to, for example, the dimensions and position of the camera, the optical properties, the sensor properties, the data compression, or any combination thereof.”);
generating, by the processor, a second image from the CG model using a second imaging model ([0007]-0009] “..repeating the rendering and storing using a plurality of further views and a plurality of further lighting arrangements to generate a multiplicity of 2D images of the seed model; and generating a dataset comprising the corresponding multiplicity of 2D images of the seed model..”), wherein the second imaging model simulates at least one physical characteristic of a sensor in the environment (Collins [0009] “..selecting one or more camera artefacts; rendering the seed model with the one or more camera artefacts; storing a 2D image of the seed model with the one or more camera artefacts..”), wherein the second image is an ideal image relative to the first image (Collins ¶56-57 “The virtual light source 150 is positioned at a second position to define a second lighting arrangement 150. In this example, the view 160 is kept the same as in FIG. 3c.” and Figs 3B and 3C where images from the same view with different lighting arrangements, where one image teaches an “ideal image” relative to the other “deterioraed image”; Examiner note: Compare instant specification criteria of “examples of ideal images and normal images (deteriorated images)” given in ¶¶115—156 to criteria for generating images with different lighting arrangements and artefacts given in Collins ¶¶50, 65, 67--79);
generating metadata of the first image and the second image for training an artificial intelligence (AI) model (Collins [0011] "associating a 2D image of the seed model with one or more corresponding labels"; the labels are considered to be metadata; [0010]-[0012] “..One or more labels, also called annotations or tags, may be used to inform the machine vision learning system what the object to be recognized should be. This may speed up the training of the vision system…”; [0095]-[0096] “..These labels may simply indicate the classification, such as dog, car, boat, or the labels may also include data about, for example, the views 160, lighting arrangement 150, background 170, camera artefact or any combination thereof. Any conventional metadata format may be used. Preferably a format is used which preserves the context of the label—for example, teal may be a color or a species of duck, flare may be an optical artefact or a signal flash…may be based on the Exchangeable Image File format (EXIF), as this already includes standard tags, or labels,..”; ¶17 “providing the dataset to a machine learning algorithm for use as a training set, a test set and/or a validation set to recognize the object”) to output an ideal image when a deteriorated image is input (Collins, ¶¶10-12, 17, disclosed structure of using the images for training machine vision AI models sufficient to meet intended function of training AI model to output ideal image from deteriorated image).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art, to modify the invention of Collins to include the intended use of an AI model to output an ideal image when a deteriorated image is input in order to “provid[e] the dataset to a machine learning algorithm for use as a training set [or] a test set.” Collins ¶17.
Collins does not teach generating metadata embedded in image data of the first image and the second image.
Riley teaches generating metadata embedded in image data of the first image and the second image for training an artificial intelligence (AI) model to output an ideal image when a deteriorated image is input (See Riley Fig. 3, ¶7 “.. . . a user may assign pixel level labels to defect images (multiple classes of defect images typically). The pixel level labels and defect images are used to train a defect detection deep learning model. . . Output from the detection deep learning model (blobs) and blob level labels may be used to train the classification deep learning model. . . .”).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art, to modify the invention of Collins to include the noted teachings of Riley, in order to train a machine learning model using one or more synthetic image. Riley ¶1.
The combination also teaches, regarding claims 9 and 10, and in the same manner as the method, An image generation device, comprising a processor and A non-temporary computer readable medium storing a program that executes an image generation method when a processor is executed (Collins Fig. 1, Fig. 4, [0027] “..the method may be implemented on any type of standalone system or client-server compatible system containing any type of client, network, server, and database elements..”).
Consider Claims 2 and 25.
The combination teaches The image generation method according to claim 1, further comprising:
selecting at least one parameter for processing the CG model or the artificial image (Collins [0007] “..selecting a first lighting arrangement; selecting a first view of the seed model..”);
and applying to the CG model or the artificial image based on the selected parameter at a timing at which the CG model or the artificial image is generated (Collins [0007] “..rendering the first view of the seed model with the first lighting arrangement; storing a 2D image of the first view of the seed model with the first lighting arrangement…”).
Consider Claims 3 and 26.
The combination teaches The image generation method according to claim 2, wherein the at least one parameter is a parameter related to the sensor (Collins [0053] “..The seed model 110 may be rendered using a plurality of virtual views 160—for each view 160, a virtual camera 160 is used to determine the view 160..” additionally view and lighting arrangement relate to a sensor i.e. camera as broadly claimed by the term “related”; SEE ALSO [0065]-[0066] “..the properties of the virtual camera 160 may comprise one or more camera artefacts..”).
Consider Claims 4 and 27.
The combination teaches The image generation method according to claim 3, wherein the sensor includes at least a camera (Collins [0065]-[0066] camera).
Consider Claim 6 and 29.
The combination teaches The image generation method according to claim 1, further comprising recording metadata of the second image or the artificial image in a storage medium (Collins [0095]-[0098] disclosing various formats of storage of images with labels, tags, metadata).
Consider Claim 7 and 30.
The combination teaches The image generation method according to claim 6,
wherein the metadata of the second image or the first image is associated with the artificial image and recorded in a storage medium (Collins [0095]-[0098] disclosing various formats of storage of images with labels, tags, metadata).
Consider Claim 8.
The combination teaches A recording medium generation method, comprising storing an image generated by the image generation method according to claim 1 in a recording medium (Collins [0007], [0082] storing images and storing the seed model and [0095]-[0098] disclosing various formats of storage of images with labels, tags, metadata).
Consider Claims 31-33
The combination teaches wherein generating the metadata includes adding an additional channel to respective pixels of each of the first image and the second image, such that the metadata is generated as multidimensional data for the respective pixels. (See Riley Fig. 3, ¶7 “.. . . a user may assign pixel level labels to defect images (multiple classes of defect images typically). The pixel level labels and defect images are used to train a defect detection deep learning model. . . Output from the detection deep learning model (blobs) and blob level labels may be used to train the classification deep learning model. . . .”).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art, to modify the combination to include the noted teachings of Riley, in order to train a machine learning model using one or more synthetic image. Riley ¶1.
Claim(s) 5 and 28 are rejected under 35 U.S.C. 103 as being unpatentable over US 20200005083 A1 Collins; Robert and US 20190294923 A1 Riley; Ian et al., further in view of JP 2020030569 A HIASA NORITO.
Consider Claims 5 and 28.
The combination teaches The image generation method according to claim 4 (See claim 4 above).
Collins does not explicitly disclose wherein the Al used for the image acquired by the sensor is used to correct a change in the image caused by the sensor or the camera.
HIASA NORITO teaches wherein the Al used for the image acquired by the sensor is used to correct a change in the image caused by the sensor or the camera (Pg. 8 Paragraph 4 “..thereby learning a CNN that enables robust correction of the magnitude of the blur. This makes it possible to simultaneously correct small defocus blur of an image having a large depth of field…”; SEE ALSO Pg. 1 Paragraph 1 ABSTRACT “An image processing method includes the steps of: acquiring a first image obtained by imaging a subject space through a first pupil of an optical system and a second image obtained by imaging the subject space through a second pupil different from the first pupil of the optical system S101; and generating a sharpened image obtained by correcting blur depending on the pupils of the optical system on the basis of the first image and the second image by using a multilayer neural network..”)
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art, to modify the invention of Collins to include the noted teachings of HIASA NORITO in order to obtain sharper images (HIASA NORITO Pg. 1 Paragraph 1 ABSTRACT)
Pertinent Prior Art(s)
The prior art made of record though not relied upon in the current rejection is considered pertinent to applicant's disclosure:
ROS GERMAN ET AL: "The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes", 2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 27 June 2016 (2016-06-27), pages 3234-3243, XP033021505
RONG JIANGPENG ET AL: "Radial Lens Distortion Correction Using Convolutional Neural Networks Trained with Synthesized Images", LECTURE NOTES IN COMPUTER SCIENCE; PAGE(S) 35 - 49, XP04 7 407322,
WO 2018/066351 Al (ADVANCED DATA CONTROLS CORP.) 12 April 2018 (2018-04-12), paragraphs [0056], [0074], [0075], [0080] & US 2019/0220029 Al, paragraphs [0114], [0133], [0134], [0139]
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to UMAIR AHSAN whose telephone number is (571)272-1323. The examiner can normally be reached Monday - Friday 10-5 PM EST or by emailing UMAIR.AHSAN@USPTO.GOV.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alison Slater can be reached on (571) 270-0375. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/UMAIR AHSAN/Primary Examiner, Art Unit 2647
