DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The claims will be read under the broadest reasonable interpretation standard outlined in
MPEP § 2111.01.
Claim Objections
Claims 5 and 16 are objected to for the following informalities:
“adjust at least one weight plurality of weights” reads as a possible grammatical error for “adjust at least one weight of the plurality of weights”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1-21 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1, 11 and 12 recite the limitation "the synthesized visual content samples”. There is insufficient antecedent basis for this limitation in the claim. With the open-ended transition phrase “comprising”, it is possible that other synthesized visual content samples exist, including within the pool of “first visual content samples”. Therefore, it cannot be presumed with certainty that “the synthesized visual content samples” refers back to the “second visual content samples”, and accordingly antecedent basis is lacking.
The examiner suggests changing the relevant language from:
“creating a training set including the synthesized visual content samples” to
“creating a training set including the synthesized second visual content samples”
In the interest of compact prosecution, the examiner will interpret the language in line with the suggestion.
Claims 2-10 and 13-21, depending on the independent claims of 1 and 12 respectively, are accordingly also rejected under 35 U.S.C. § 112(b) by virtue of their dependency.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-4, 9-15, and 20-21 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Hirota et. al (US 20250148657 A1) (Hereinafter, “Hirota”).
With respect to claim 1, Hirota discloses:
A method for augmented machine learning ([Abstract]), comprising:
synthesizing a plurality of second visual content samples ([0007] “Using masked person images and text prompts, aspects of the present invention can generate counterfactual images by inpainting only the masked regions, addressing ethical concerns of altering nonconsensual persons and ensuring equal representation of protected groups across attributes”)
wherein synthesizing the plurality of second visual content samples further comprises removing at least a portion of a plurality of first visual content samples with respect to an object in order to create a plurality of removed portion visual content items ([0007] “Using masked person images and text prompts, aspects of the present invention can generate counterfactual images by inpainting only the masked regions, addressing ethical concerns of altering nonconsensual persons and ensuring equal representation of protected groups across attributes”; [0012] “Embodiments of the present invention provide a computer-implemented method for generating a synthetic training dataset with group-independent image attribute distributions, comprising using a text-guided inpainting model to inpaint a person mask in an original image of an original dataset with a synthetic person from a protected group to generate a synthetic image… maintaining consistent context of the original image…”; wherein the “removed portion visual content items” corresponds to the resulting masked images; [0046] “ω ∈ [0, 1]d is a person mask…inpaints ω in x with a synthetic person from protected group g described in t(g)”; [0054]) and providing the plurality of removed portion visual content items to a generative machine learning model ([0012] “using a text-guided inpainting model to inpaint a person mask in an original image”; [0046]), wherein the generative machine learning model is trained to generate at least a portion of visual content with respect to the plurality of removed portion visual content items ([0005] “While equal group distributions in real-world datasets are challenging to achieve, generative text-to-image models now enable targeted image modification”; [0012] “text-guided inpainting model”; [0046])
creating a training set including the synthesized visual content samples ([0007]; [0012] “generating a synthetic training dataset with group-independent image attribute distributions”; [0046]-[0047])
training a machine learning model using the training set, wherein the machine learning model is trained to classify visual content with respect to the object (Figs. 4-7; [0008] “Unlike conventional approaches, training on the counterfactual data decorrelates both labeled and unlabeled attributes from protected groups without impacting model performance. Comprehensive evaluations show approaches according to aspects of the present invention significantly reduces prediction rules based on spurious correlations in multi-label classification and image captioning across various architectures (e.g., ResNet-50, Swin Transformer), datasets (COCO, Open-Images), and protected groups (gender, skin tone).”; [0053]; [0055])
With respect to claim 2, Hirota discloses:
The method of claim 1, wherein removing at least a portion of the plurality of first visual content samples further comprises:
segmenting the plurality of first visual content samples with respect to the object, wherein the at least a portion of the plurality of first visual content samples is removed based on the segmenting of the plurality of the first visual content samples with respect to the object ([0007]; [0012] mask generation; [0067])
With respect to claim 3, Hirota discloses:
The method of claim 2, wherein segmenting each of the plurality of first visual content samples results in at least one set of pixels corresponding to the object for each first visual content sample, wherein the at least one set of pixels corresponding to the object are removed from the at least one of the plurality of first visual content samples (Figs. 2A-2D; [0007]; [0012] mask generation; [0046] “where x ∈
PNG
media_image1.png
29
29
media_image1.png
Greyscale
d is an image, ω ∈ [0, 1]d is a person mask”)
With respect to claim 4, Hirota discloses:
The method of claim 1, further comprising:
querying the generative machine learning model with respect to the plurality of first visual content samples, wherein the query indicates a variation of the object for which the at least a portion of visual content is to be generated by the generative machine learning model ([0005] “While equal group distributions in real-world datasets are challenging to achieve, generative text-to-image models now enable targeted image modifications”; [0012] “text-guided inpainting model”; [0044] “Training datasets were created with group-independent image attribute distributions by using masked person images and text prompts with a diffusion model, as outlined in FIGS. 2A through 2E”; [0046])
With respect to claim 9, Hirota discloses:
The method of claim 1, wherein the generative machine learning model is a diffusion model ([0044] “Training datasets were created with group-independent image attribute distributions by using masked person images and text prompts with a diffusion model, as outlined in FIGS. 2A through 2E.”)
With respect to claim 10, Hirota discloses:
applying the trained machine learning model to a plurality of third visual content samples ([0055] “This results in 28,487/13,487 train/test samples”; wherein the 13,487 test samples correspond to the “third visual content samples”)
With respect to claim 11, Hirota discloses:
A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process ([0011]), the processing comprising: [the method of claim 1, rejected in line with the analysis above]
With respect to claim 12, Hirota discloses:
A system for augmented machine learning ([Abstract]), comprising:
a processing circuitry ([0032])
a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to ([0034]): [perform the method of claim 1, rejected in line with the analysis above]
With respect to claims 13-15 and 20-21, they are functionally parallel to claims 2-4 and 9-10 respectively, with the exception that the steps are performed by the processing circuitry of claim 12. Accordingly, the claims are rejected in line with claims 12, 2-4, and 9-10.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Hirota in view of Kirichenko et. al Last Layer Re-Training Is Sufficient For Robustness to Spurious Correlations (Hereinafter, “Kirichenko”)
With respect to claim 5, Hirota teaches the method of claim 1.
Hirota does not explicitly teach the further limitations of claim 5.
However, Kirichenko, in the same field of endeavor of model debiasing, teaches:
wherein the trained machine learning model has a plurality of weights ([1] “While both the relevant and spurious features are learned, the spurious features can be highly weighted in the final classification layer of the model, leading to poor predictions on the minority groups…we simply retrain the last layer of a classification model trained with standard Empirical Risk Minimization (ERM)…”), further comprising
applying the trained machine learning model to at least one evenly distributed data set ([2] “In this paper, we study the effect of spurious correlations on the features learned by standard neural networks, and based on our findings propose a simple way of reducing the reliance on spurious features assuming access to a small set of data where the groups are equally represented”) in order to produce a set of outputs with respect to the at least one evenly distributed data set (Appendix, B “For the logistic regression model, we first extract the embeddings from the penultimate layer of the network, then use the logistic regression implementation (sklearn.linear_model.LogisticRegression) from the scikit-learn package (Pedregosa et al., 2011). We use ℓ1 regularization and tune the inverse regularization strength parameter C in the range {0.1,10,100,1000}. For more details on Deep Feature Reweighting implementation and tuning, see section C.”), each evenly distributed data set including an evenly distributed set of visual content samples (Appendix A “In DFR ,we subsample the reweighting dataset to be group balanced instead of using class- or group-balanced sampling. Specifically, we produce a dataset where the number of examples in each group is the same, and only use these datapoints. This detail is hugely important, as group-balanced sampling does not produce classifiers robust to spurious correlations (e.g. see RWG and SUBG methods in Idrissi et al., 2021).”; Appendix C.1, Table 10 “We train the logistic regression model on the validation data several times with random subsets of the data (we take all of the data from the smallest group, and subsample the other groups randomly to have the same number of datapoints) and average the weights of the learned models. Averaging more than 1 linear model leads to improved performance.”)
wherein the trained machine learning model outputs a classification for each visual sample among each evenly distributed data set (Fig. 1; [2] “For example, in the Waterbirds dataset (Sagawa et al., 2019), the task is to classify whether an image shows a landbird or a waterbird. The groups correspond to images of waterbirds on water background (G1), waterbirds on land background (G2), landbirds on water background (G3) and landbirds on land background (G4). See Figure 6 for a visual description of the Waterbirds data.”)
adjusting at least one weight plurality of plurality of weights based on the set of outputs with respect to the at least one evenly distributed data set (Fig. 1; [1] “With DFR, we simply reweight these features by retraining the last linear layer on a small dataset where the backgrounds are not spuriously correlated with the foreground. The resulting DFR model primarily relies on the foreground and performs much better on images with confusing backgrounds”; Appendix C.1, Table 10)
It would have been obvious to one of ordinary skill in the art as of the effective filing date of the claimed invention, to modify Hirota to include the limitations of weight adjustment amongst evenly distributed data sets, as taught by Kirichenko. Doing so would have the advantage of reducing bias, which is the ultimate goal of both methods. The systems readily integrate, as the Hirota also features weighting ([0047]; [0054]; [0057]).
With respect to claim 16, it is functionally parallel to claim 5, with the exception that the steps are performed by the processing circuitry of claim 12. Accordingly, the claim is rejected in line with claims 12 and 5.
Claims 6, 8, 17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Hirota in view of Liusie et. al Mitigating Word Bias in Zero-shot Prompt-based Classifiers (Hereinafter, “Liusie”)
With respect to claim 6, Hirota teaches the method of claim 1.
Hirota does not explicitly teach the further limitations of claim 6.
However, Liusie, in the same field of endeavor of bias reduction, teaches:
wherein the generative machine learning model is a first generative machine learning model, wherein the trained machine learning model has a plurality of thresholds ([2]; [3.1] “The oracle upper-bound performance, found by optimising the optimal accuracy threshold via equation 4 (optimal)”; [3.2] “Figure 4 shows a scatter plot of the weights found by the optimal threshold search α∗ (equation 4)”), further comprising:
PNG
media_image2.png
377
490
media_image2.png
Greyscale
querying a second generative machine learning model for distribution data of a population set ([2] “Prompt-based classifiers”; [2] “Null-Input Approximation”, “Inspired by Zhao et al. (2021), we consider a resource-free approximation of the word prior (equation 8) by considering the output word probabilities of the null input ∅ (i.e. an empty string).”; equations 8 and 9)
adjusting at least one threshold of the plurality of thresholds based on the distribution data of the population set ([3.2] equations 8 and 9 output used in equation 3 and 4 for finding optimal threshold).
With respect to claim 8, Hirota and Liusie teach:
The method of claim 6, wherein the second generative machine learning model is a large language model ([2]; [2] “Null-Input Approximation”)
PNG
media_image3.png
181
525
media_image3.png
Greyscale
With respect to claims 17 and 19, they are functionally parallel to claims 6 and 8 respectively, with the exception that the steps are performed by the processing circuitry of claim 12. Accordingly, the claims are rejected in line with claims 12, 6, and 8.
Claims 7 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Hirota and Liusie in view of Yang et. al (US 20220207707 A1) (Hereinafter, “Yang”)
With respect to claim 7, Hirota and Liusie teach the method of claim 6.
Hirota and Liusie do not explicitly teach the further limitations of claim 7.
However, Yang, in the same field of endeavor of image model training, teaches:
wherein the at least one threshold is adjusted such that the trained machine learning model achieves a predetermined precision rate ([0035] “In one embodiment, the threshold is adjusted according to a preset recall rate or a preset accuracy rate. The preset accuracy rate is a proportion of the number of correctly detected defect sample images to the number of all detected defect sample images, and the preset recall rate is a proportion of the number of correctly detected defect sample images to the number of all real defect sample images”)
It would have been obvious to one of ordinary skill in the art as of the effective filing date of the claimed invention, to modify Hirota and Liusie to include the limitations of threshold adjustment, as taught by Yang. Doing so would offer increased accuracy. The systems readily integrate, as Hirota and Liusie are already configured to contain adjustable thresholds.
With respect to claim 18, it is functionally parallel to claim 7, with the exception that the steps are performed by the processing circuitry of claim 12. Accordingly, the claim is rejected in line with claims 12 and 7.
Additional References
Additionally cited references (see attached PTO-892) otherwise not relied upon above have been made of record in view of the manner in which they evidence the general state of the art.
Inquiry
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NOAH WILLIAM BOYAR whose telephone number is 571-272-8392. The examiner can normally be reached 10:00 – 6:00 EST, Monday – Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at 571-272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NOAH W BOYAR/Examiner, Art Unit 2669
/IAN L LEMIEUX/Primary Examiner, Art Unit 2669
Prosecution Insights