Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 12 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Line 2 of claim 12 states “the synthetic spoof image example” it is unclear if that is referring to the same example as the “negative image example” on the last two lines of claim 11. Therefore claim 12 is rejected for being indefinite.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-6, 9-10, 13-19 are rejected under 35 U.S.C. 102 (a)(1) as being Unpatentable by “Text-to-Image Diffusion Models are Zero-Shot Classifiers” (hereinafter referred to as Clark).
Regarding claim 1, Clark teaches a method to perform material classification of imagery, the method comprising: obtaining, by a computing system, an input image [see Figure 1 of Clark where an image X0 is obtained]
PNG
media_image1.png
505
1057
media_image1.png
Greyscale
adding, by the computing system, a set of added noise to the input image to generate a noised image [see Fig 1 above where in xtn noise is introduced]; processing, by the computing system, the noised image with a denoising diffusion model conditioned with a first set of conditioning text to generate a first denoising output [See Figure 1 above where the photos are denoised] , wherein the first set of conditioning text describes a positive condition for a material [see Figure 1 above where the conditioning text says "photo of a horse" which is a positive condition based on the input image]; processing, by the computing system, the noised image with the denoising diffusion model conditioned with a second set of conditioning text to generate a second denoising output, wherein the second set of conditioning text describes a negative condition for the material [see Fig. 1 above where a false conditioning text is applied such as "photo of a frog"]; and determining, by the computing system, a material score based at least in part on the first denoising output and the second denoising output [See Figure 1 where a scores vector is created for the first and second outputs, additionally an object can be reasonably interpreted as a material].
Regarding claim 2, Clark teaches the first denoising output comprises a first set of predicted noise predicted by the denoising diffusion model to be removed from the noisy image to generate a first denoised image [see page 2 paragraph one where a diffusion model is used to repeatedly noise and denoise the image according while being conditioned to a specific class (where the class could be the positive condition);
PNG
media_image2.png
283
920
media_image2.png
Greyscale
The second denoising output comprises a second set of predicted noise predicted by the denoising diffusion model to be removed from the noisy image to generate a second denoised image [see page 2 paragraph one above where a diffusion model is used to repeatedly noise and denoise the image according while being conditioned to a specific class (where the class could be the negative condition);
and the material score is based on (1) a first denoising error generated by application of a loss function to the set of added noise and the first set of predicted noise and (2) a second denoising error generated by application of the loss function to the set of added noise and the second set of predicted noise. [See section 2 page 3 where the loss function is estimated to find the denoising error. These impact the scores calculated in Fig. 1 above and the scores are compared to determine the class which results in the best denoising ability]
PNG
media_image3.png
490
566
media_image3.png
Greyscale
Regarding claim 3, Clark teaches the computer-implemented method of claim 2, wherein the loss function comprises a mean square error function [See section 2 page 3 above where the loss function can be written as the underlined section, which is a mean squared function].
Regarding claim 4, Clark teaches that the first denoising output comprises a first denoised image [See Fig. 3 where the first output from the denoising system (positive condition) is a denoised arctic fox];
PNG
media_image4.png
369
913
media_image4.png
Greyscale
the second denoising output comprises a second denoised image [See Fig. 3 where the second output (negative condition) is a denoised pillow]; and the material score is based on (1) a first reconstruction error generated by application of a loss function to the input image and the first denoised image [See page 4 paragraph 4 where the error generated when denoising the image is a score, see also section 2 above where the error is used to calculate a prior and reconstruction loss image]
PNG
media_image5.png
216
942
media_image5.png
Greyscale
and (2) a second reconstruction error generated by application of the loss function to the input image and the second denoised image [See Fig. 3 above where the denoising occurs for the second image as well (negative condition)].
Regarding claim 5, Clark teaches determining, by the computing system, the material score comprises determining, by the computing system, a step- wise denoising error computed over a single denoising step [See page 4 paragraph 4 above where error is calculated for each step].
Regarding claim 6, Clark teaches that the denoising diffusion model operates over a plurality of time steps, and wherein determining the material score comprises aggregately computing the material score over multiple of the plurality of time steps. [See the caption under Fig. 1 above where the scores matrix for calculating the error is calculated across multiple time steps].
Regarding claim 9, Clark teaches that the denoising diffusion model has been training on a training dataset that includes images of the material [See page 8 paragraph 1 where the diffusion model is trained on various images, which would include images of the material].
PNG
media_image6.png
243
950
media_image6.png
Greyscale
Regarding claim 10, Clark teaches providing, by the computing system for display to a user, one or both of the first denoising output and the second denoising output to facilitate an interpretation of the material score [See Fig. 3 above which shows an output of the first and second denoised image where a user could interpret which appears more accurate, and therefore have a higher material score].
Claim 13 is similarly analyzed to Claim 1.
Claim 14 is similarly analyzed to Claim 2.
Claim 15 and 17 are similarly analyzed to Claim 3.
Claim 16 is similarly analyzed to Claim 4.
Claim 18 is similarly analyzed to Claim 5.
Claim 19 is similarly analyzed to Claim 6.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Clark.
Regarding claim 8, Clark teaches the first set of conditioning text comprises the word "material" and the second set of conditioning text comprises the words "not material" [See Fig. 1 where the conditioning input can be "a photo of a cat" (negative input) or "a photo of a Horse" (positive input), or any variation of a caption that matches an image, and another that does not as seen in Fig. 3].
It is obvious to one with ordinary skill in the art to make the conditioning text say "material" and "not material" instead of an object and an alternative object. This would be considered an obvious design choice that aims to obtain the same outcome, either the denoised image is "correct" based off the original image, or not.
Claim 7 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Clark in view of “Your Diffusion Model is Secretly a Zero-Shot Classifier” (hereinafter referred to as Li).
Regarding claim 7, Clark teaches the ty method of claim 1, and that the denoising diffusion model operates over a plurality of time steps, and wherein determining the material score comprises aggregately computing the material score over a middle subset of the plurality of time steps [See caption under Fig. 1 above where the efficiency is improved by computing only a subset of the full scores matrix].
Clark does not teach that the material score is determined by using the middle time steps.
Li does teach that the middle timesteps could be used instead to calculate the prediction error and therefore the material score. [See section 4.1 of Li which teaches that middle steps can be sampled at higher rates]
PNG
media_image7.png
365
348
media_image7.png
Greyscale
It would have been obvious to one with ordinary skill in the art before the effective filing date to combine the step wise error calculation using aggregated steps of Clark, with the selection of only middle timesteps. The motivation to combine would be in order to reduce the computational power required in the calculations, therefore increasing efficiency.
Claim 20 is similarly analyzed to claim 7.
Claim 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Clark in view of Semi-Supervised Learning with Pseudo-Negative Labels for Image Classification” (hereinafter referred to as Xu).
Regarding claim 11, Clark teaches A computing system for generating synthetic negative image examples for a material, the computer system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store: a set of positive images that depict a material [See Stable diffusion details under Section A: Model Details which state that the method uses a diffusion model, which would require a processer and a memory (non-transitory computer readable media). See also Fig. 1, where the positive image would be the input image into the model];
PNG
media_image8.png
129
894
media_image8.png
Greyscale
And instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising, for each positive image of the set of positive images:
adding, by the computing system, a set of added noise to the positive image to generate a noised image [see Fig. 1 above where noise is added to the image to generate a noised image]; processing, by the computing system, the noised image with the denoising diffusion model conditioned with a set of conditioning text to generate a denoised image [See Fig. 1 above], wherein the set of conditioning text textually describes a negative condition for the material [See Fig 1 and Fig 3];
Clark does not explicitly teach storing the denoised image as a negative image example for the material.
Xu does teach storing a negative image example for the material [see section 3.3.1 where the negative labels are stored and used to teach that the image does not belong in that category (i.e. not the material)].
PNG
media_image9.png
682
469
media_image9.png
Greyscale
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date to combine the method of generating denoised images of Clark with storing the negative images of Xu. The motivation to combine would be to utilize as much data as possible from the model in training, and to prevent model degeneration and error accumulation as stated above in section 3.3 of Xu.
Regarding claim 12, Clark and Xu teach the computing system of claim 11, Xu further teaches that the operations further comprise training a discriminative model using the synthetic spoof image example [See section 3.3 above where the model is trained in a semi-supervised framework using the negatively labeled images. See also Fig 5 of Xu where the model more accurately represents the discrimination between labels, indicating that the model is a discriminative model].
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANUSHA KASHYAPA whose telephone number is (571)272-8766. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANUSHA KASHYAPA/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
