DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 6 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 6 recites “the probabilistic data” in line 3. There is insufficient antecedent basis for this limitation in both claim 4, from which it depends, and earlier limitations of claim 6.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-7, 9, 14-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Chadha et al. (US 20210150282) and Kim et al. (KR20230151863, utilizing a machine translation).
Regarding claim 1, Chadha et al. discloses a processor-implemented method, the method comprising:
generating a plurality of output images (“The system supports manipulation of the input image to produce at least one modified input image 210. As illustrated, the system manipulates the input image 205 to generate a first scaled image 210-a and a second scaled image 210-b. The first scaled image 210-a is a 50×50 pixel version of the input image 205, and the second scaled image 210-b is a 200×200 pixel version of the input image 205. Other image manipulations may be performed by the system 200 in addition to or instead of scaling. For example, the system may flip images, stretch or squeeze images, add noise to image data” at paragraph 0029, line 1; therefore, multiple modified versions of the input image are generated using different noise quantities) using a model that is provided a raw image of an image sensor (‘The object detection processing diagram illustrates how a system receives an input image 205, which may be referred to as a test image” at paragraph 0028, line 4);
generating, using a machine vision model that is provided the plurality of output images, plural output data respectively corresponding to the plurality of output images (“The input image 205 and the modified input images 210 may be processed by an object detection machine learning model. In FIG. 2, the machine learning model is an object detection neural network 220” at paragraph 0030, line 1); and
generating result data of the machine vision model by performing an ensemble on the plural output data (“The object detection neural network 220 may identify sets of objects (e.g., an object identified by a bounding box 225) for each processed image and confidence scores associated with the identified objects” at paragraph 0031, line 1).
Chadha et al. does not explicitly disclose that the modifications are produced via a generative model.
Kim et al. teaches a method in the same field of endeavor of generating noise images, comprising:
generating a plurality of output images using a generative model that is provided a raw image of an image sensor (“The synthetic noise image generation module (120) can extract noise information for a reference noise image based on a neural network that has been pre-trained by the learning module (110), and generate a synthetic noise image for a clean image based on the noise information extracted based on the neural network that has been pre-trained by the learning module (110)” at paragraph 0019, line 3).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the noise image synthesization as taught by Kim et al. to produce the modified images of Chadha et al. for purposes of closely representing characteristics of actual noise.
Regarding claim 15, Chadha et al. discloses an electronic device comprising:
an image sensor configured to generate a raw image (“In one example, a cloud client 105 may utilize the image detection service for document intake. A contact 110 may access the service using a personal device such as a laptop computer. The contact 110 may input a document (e.g., a contract) into the service, the service (or an associated service) may take an image of the document, and the image may be input into the image detection system” at paragraph 0027, line 1; implied that the taking of the image is accomplished via an image sensor); and
a processor (“The functions described herein may be implemented in hardware, software executed by a processor” at paragraph 0163, line 1) configured to:
generate a plurality of output images (“The system supports manipulation of the input image to produce at least one modified input image 210. As illustrated, the system manipulates the input image 205 to generate a first scaled image 210-a and a second scaled image 210-b. The first scaled image 210-a is a 50×50 pixel version of the input image 205, and the second scaled image 210-b is a 200×200 pixel version of the input image 205. Other image manipulations may be performed by the system 200 in addition to or instead of scaling. For example, the system may flip images, stretch or squeeze images, add noise to image data” at paragraph 0029, line 1; therefore, multiple modified versions of the input image are generated using different noise quantities) using a model provided the raw image (‘The object detection processing diagram illustrates how a system receives an input image 205, which may be referred to as a test image” at paragraph 0028, line 4);
generate, using a machine vision model that is provided the plurality of output images, plural output data respectively corresponding to the plurality of output images (“The input image 205 and the modified input images 210 may be processed by an object detection machine learning model. In FIG. 2, the machine learning model is an object detection neural network 220” at paragraph 0030, line 1); and
generate result data of the machine vision model through performance of an ensemble on the plural output data (“The object detection neural network 220 may identify sets of objects (e.g., an object identified by a bounding box 225) for each processed image and confidence scores associated with the identified objects” at paragraph 0031, line 1).
Chadha et al. does not explicitly disclose that the modifications are produced via a generative model.
Kim et al. teaches a device in the same field of endeavor of generating noise images, comprising:
an image sensor configured to generate a raw image (“The device (2000) may include an integrated circuit (2100) and components connected to the integrated circuit (2100), such as a sensor (2200),” at paragraph 0141, line 1; “For example, the GPU (2130) can perform image processing on image data received through the sensor interface (2150)” at paragraph 0146, line 2); and
a processor (“In an embodiment, the GPU (2130) or computing device (2160) can perform operations of the neural network during the learning phase or data recognition phase of the neural network” at paragraph 0147, line 4) configured to:
generate a plurality of output images using a generative model provided the raw image (“The synthetic noise image generation module (120) can extract noise information for a reference noise image based on a neural network that has been pre-trained by the learning module (110), and generate a synthetic noise image for a clean image based on the noise information extracted based on the neural network that has been pre-trained by the learning module (110)” at paragraph 0019, line 3).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the noise image synthesization as taught by Kim et al. to produce the modified images of Chadha et al. for purposes of closely representing characteristics of actual noise.
Regarding claims 2 and 16, Kim et al. discloses a method and device wherein the processor is further configured to generate a plurality of noise images through random sampling, and generates each of the plurality of output images by the generative model based on a corresponding one of the generated plurality of noise images input to the generative model (“In an embodiment, a Gaussian random noise distribution (e.g., Z~N(0,1)) may be concatenated with a noise embedding to generate spatially random noise. That is, randomness can be added by introducing Gaussian random noise z for each forward propagation in the generative neural network (220)” at paragraph 0032, line 6).
Regarding claim 3, Kim et al. discloses a method wherein the generating of the plurality of noise images comprises generating the plurality of noise images based on respective performances of at least one of Gaussian sampling In an embodiment, a Gaussian random noise distribution (e.g., Z~N(0,1)) may be concatenated with a noise embedding to generate spatially random noise. That is, randomness can be added by introducing Gaussian random noise z for each forward propagation in the generative neural network (220)” at paragraph 0032, line 6) or Poisson sampling.
Regarding claim 4, Kim et al. discloses a method wherein each of the plurality of output images is differently generated by inputting the raw image and a respective different noise image into the generative model (“The generative neural network (220) represents a neural network capable of synthesizing a new noise image using noise information derived from the first classification neural network (210). The generative neural network (220) can generate a synthetic noise image for a clean image using the noise embeddings generated by the first classification neural network (210)” at paragraph 0027).
Regarding claims 5 and 17, Chadha et al. discloses a method and device wherein, for the ensembling, the processor is configured to perform a combining of respective probabilistic data of the plural output data, or a selecting of respective probabilistic data from among the plural output data, to generate the result data (“That is, the object detection neural network 220 may identify a range (e.g., distribution) of confidence scores for each processed image, where each confidence score corresponds to an identified object from one of the processed images. Certain post processing and filtering algorithms 230 may perform certain procedures to output detected objects. For example, the post processing and filtering algorithms 230 may resize bounding boxes to original scale and filter confidence scores and associated objects” at paragraph 0031, line 4).
Regarding claims 6 and 18, Chadha et al. discloses a method and device wherein, for the ensembling, the processor is configured to perform the selecting, including a generation of the determined result data using a determined maximum confidence value for each corresponding element of the probabilistic data of the plural output data, where each of the plural output data include plural probabilistic data elements (“According to NMS thresholding per modification 250, the system may identify objects corresponding to overlapping bounding boxes per modified input image 210 and select the highest score from the overlapping boxes for consideration. This technique may be performed prior to resizing images/bounding boxes to the original scale (or changing other parameters of the modified input images 210 to correspond to the input image 205, such as removing transformations, removing brightness modifications, etc.)” at paragraph 0036, line 1).
Regarding claim 7, Kim et al. discloses a method wherein the generative model comprises at least one of a diffusion model or a generative adversarial network (GAN) model (“That is, the generative neural network (220) can be trained based on adversarial learning based on a Generative Adversarial Network (GAN)” at paragraph 0029, line 4).
Regarding claim 9, Chadha et al. discloses a method wherein the machine vision model comprises at least one of an object detection model (“The input image 205 and the modified input images 210 may be processed by an object detection machine learning model. In FIG. 2, the machine learning model is an object detection neural network 220” at paragraph 0030, line 1), an image segmentation model, or defect detection model.
Regarding claims 14 and 20, Kim et al. discloses a method and device wherein the processor comprises at least a generative image signal processor (ISP) that is configured to implement the generative model using one or more pipelines of processing elements of the generative ISP (“The generative neural network (220) represents a neural network capable of synthesizing a new noise image using noise information derived from the first classification neural network (210). The generative neural network (220) can generate a synthetic noise image for a clean image using the noise embeddings generated by the first classification neural network (210)” at paragraph 0027).
Claim(s) 8, 13 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Chadha et al. and Kim et al. as applied to claims 1 and 15 above, and further in view of Voroninski et al. (US 12,205,197).
Regarding claims 8 and 19, the Chadha et al. and Kim et al. combination discloses the elements of claims 1 and 15 above.
The Chadha et al. and Kim et al. combination does not explicitly disclose that the plurality of output images are red, green, and blue (RGB) images respectively generated by the generative model corresponding to the raw image that is input to the generative model.
Voroninski et al. teaches a method and device in the same field of endeavor of generative image models wherein the plurality of output images are red, green, and blue (RGB) images respectively generated by the generative model corresponding to the raw image that is input to the generative model (“In this example, platform 100 includes generative image synthesis engine 106. Generative image synthesis engine 106 is configured to generate synthesized images. As one example, the images to be synthesized are RGB (Red, Green, Blue) images, where the synthesized image includes some number of pixels, and each pixel has RGB channel information” at col. 4, line 14).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to generate RGB images as taught by Voroninski et al. for the output images of the Chadha et al. and Kim et al. combination as it common to utilize RGB images for purposes of object detection.
Regarding claim 13, the Chadha et al. and Kim et al. combination discloses the elements of claim 1 above.
The Chadha et al. and Kim et al. combination does not explicitly disclose that the plurality of output images comprises different semantic information.
Voroninski et al. teaches a method in the same field of endeavor of generative image models wherein the plurality of output images comprises different semantic information (“As one example if incorporating the label prompt as an input, the label prompt is combined with the random noise input (which for example, can be considered an image with RGB pixels, but of randomly sampled RGB noise values). For example, combiner 202 is configured to concatenate, append, or add (e.g., vector/component-wise addition) the label prompt to the random noise input. The combined random (image RGB space) noise/label prompt is then fed to the pixel diffusion model 204. As output, the pixel diffusion model 204 generates a synthetic image in the RGB space (image 108). The synthetic image is then combined with the label prompt and scene prompt, as described above, to generate a training sample (110)” at col. 12, line 26).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include semantic information as taught by Voroninski et al. in the output images of the Chadha et al. and Kim et al. combination for purposes of generating a training sample for subsequent object detector of the Chadha et al. and Kim et al. combination.
Claim(s) 10-12 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Chadha et al. and Kim et al. as applied to claims 1 and 15 above, and further in view of Mosleh et al. (US 20220301123).
Regarding claim 10, the Chadha et al. and Kim et al. combination discloses the elements of claim 1 above.
The Chadha et al. and Kim et al. combination does not explicitly disclose that the generative model is trained to optimize a performance of the machine vision model.
Mosleh et al. teaches a method in the same field of endeavor of generative image models, wherein the generative model is trained to optimize a performance of the machine vision model (“generate adapted images in the visual domain of the images used to train the perception module, the end-to-end operation of the machine vision system can be optimized with regard to the visual domain of the raw images generated by a new sensor” at paragraph 0014, line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to train the modules end-to-end as taught by Mosleh et al. in the system of the Chadha et al. and Kim et al. combination to “improve end-to-end performance though a low-cost process which bypasses conventional fine-tuning and data labelling” (Mosleh et al. at paragraph 0049, last sentence).
Regarding claim 11, Mosleh et al. discloses a method wherein the generative model and the machine vision model are trained together end-to-end (“generate adapted images in the visual domain of the images used to train the perception module, the end-to-end operation of the machine vision system can be optimized with regard to the visual domain of the raw images generated by a new sensor” at paragraph 0014, line 2).
Regarding claims 12 and 21, the Chadha et al. and Kim et al. combination discloses the elements of claims 1 and 15above.
The Chadha et al. and Kim et al. combination does not explicitly disclose that the generative model and the machine vision model are configured as having been trained together end-to-end.
Mosleh et al. teaches a method and device in the same field of endeavor of generative image models, wherein the generative model and the machine vision model are configured as having been trained together end-to-end (“generate adapted images in the visual domain of the images used to train the perception module, the end-to-end operation of the machine vision system can be optimized with regard to the visual domain of the raw images generated by a new sensor” at paragraph 0014, line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to train the modules end-to-end as taught by Mosleh et al. in the system of the Chadha et al. and Kim et al. combination to “improve end-to-end performance though a low-cost process which bypasses conventional fine-tuning and data labelling” (Mosleh et al. at paragraph 0049, last sentence).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/ Primary Examiner, Art Unit 2672
Prosecution Insights