Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5, 7, 10-12, 14, 16, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Nia (US 20220129791 A1), in view of Carreira (US 20250103856 A1) and Ho (US 20240100693 A1).
Regarding claim 1, Nia discloses “generating, by a processor, a plurality of perturbed instances, wherein each of the plurality of perturbed instances is generated by perturbing one or more encoded features of a[n] artificial intelligence (AI)… model instance;” (See [0025]; data samples (perturbed instances) are generated by perturbing the feature values of a target data sample)
“determining, by the processor and for each of the plurality of perturbed instances, a distance between one or more encoded features of each perturbed instance and a corresponding one or more of the one or more encoded features of the multimodal AI model instance;” (See [0064]; LIME is used to weight the data samples (perturbed instances) to determine a distance between one or more encoded features)
“converting each distance to a weight using a kernel function to obtain a plurality of weights;” (See [0091]; weights are calculated based on the distances of each target data sample (perturbed instance) and yields a plurality of weights)
“determining for each respective weight of the plurality of weights, a… Shapley value corresponding to a modality associated with the respective weight” (See [0013], [0023]; LIME is a model explainer used to determine weights for data samples, and Shapley values for the respective weight can be determined using Kernel SHAP, an expansion of LIME)
“outputting an interpretable surrogate model based on the plurality of final weights.” (See [0061], [0095]; surrogate ML models are generated from the training data set, and they are based on the plurality of final generated weights for the data samples).
Nia fails to explicitly disclose, “AI model instance” is a “multimodal AI model instance” and “a Shapley value” is a “modality-specific Shapley value”. Nia fails to further explicitly disclose, “and post-weighting the respective weight with the modality-specific Shapley value associated with the weight to obtain a plurality of final weights;”. As noted above however, Nia teaches "Shapley values" used in a similar manner. See [0023]; Shapley values for each weight can be determined using Kernel SHAP, an extension of LIME.
Carreira teaches “a multimodal AI model instance” (See [0060]; the AI instance can be multimodal, featuring a combination of different types of data). Carreira also teaches “modality-specific… value” (See [0060]; Where the entity is multimodal, the associated data can be modality-specific values). Carreira further teaches “post-weighting the respective weight by multiplying the respective weight by the modality-specific … value associated with the weight to obtain a plurality of final weights;” (See [0065]; a set of embeddings (weights) is updated by generating a new weight for each embedding, and then multiplying each embedding with its corresponding new weight. Each embedding is then replaced by the product of the old embedding and its corresponding new weight).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Nia and Carreira before them to modify Nia to be a multimodal AI model instance, to use modality-specific values, and post-weighting each weight to obtain final weights. One would be motivated to modify Nia into a multimodal AI model instance in order to give an AI model a more comprehensive understanding of a data set, as being able to take multiple types of input, see e.g., [0060], where Carreira describes multiple examples of input data types, such as image, video, and audio data, would broaden an AI model’s context of a given task, as opposed to being offered only a single type of data, such as text, to accomplish a task. One would also be motivated to modify Nia to use modality-specific Shapley values to ensure that the Shapley values used are consistent with the AI model instance’s data types, see e.g., [0060], where Carreira describes that data that is associated with a modality-specific features can be fixed to that specific modality so that their modality can be easily identified. Additionally, one would be motivated to modify Nia to post-weight each weight to obtain final weights to be able to update the set of weights in the model to prepare it for use in other models or algorithms.
Nia fails to explicitly disclose, “wherein the modality-specific Shapley value associated with a first respective weight, of the plurality of weights, corresponds to a first modality, and wherein the modality-specific Shapley value associated with a second respective weight, of the plurality of weights, corresponds to a second modality that is different from the first modality”. As noted above however, Nia teaches "Shapley values" used in a similar manner. See [0023]; Shapley values for each weight can be determined using Kernel SHAP, an extension of LIME.
Ho teaches “wherein the modality-specific… value associated with a first respective weight, of the plurality of weights, corresponds to a first modality, and wherein the modality-specific… value associated with a second respective weight, of the plurality of weights, corresponds to a second modality that is different from the first modality” (See [0121]; Ho discloses using a first modality with a first respective weight and a second modality that is distinct from the first modality with a second respective weight).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Nia and Ho before them to modify Nia to use a first modality with a first respective weight and a second modality that is distinct from the first modality with a second respective weight. One would be motivated to modify Nia to do so because using two different modalities allows ML models to be trained on a wider variety of data types, see e.g., [0014] and [0016], where Ho teaches using RGB images as a first modality and depth images as a second modality for training a robot’s ML models for predicting robotic actions based on visual data.
Regarding claim 2, Nia discloses “generating the plurality of perturbed instances comprises: independently perturbing the one or more encoded features of each perturbed instance according to the modality of each of the one or more encoded features of the multimodal AI model instance.” (See [0025]; Kernel SHAP generates a data sample by perturbing the features of each perturbed instance).
Regarding claim 3, Nia fails to explicitly disclose, “generating the plurality of perturbed instances is performed using at least one of a pretrained autoencoder or a generative adversarial network.”.
Carreira teaches “generating is performed using at least one of a pretrained autoencoder or a generative adversarial network.” (See [0104], an autoencoder is shown being used for generating a data set).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Nia and Carreira before them to modify Nia to generate an instance using an autoencoder. One would be motivated to do so in order to process the data set of a multimodal AI instance by perturbing the features of an instance to generate a perturbed instance, see e.g., [0104], where Carreira describes that an auto encoder is used to process the data set to generate a corresponding data set.
Regarding claim 5, Nia discloses “the kernel function” (See [0091]; a kernel function is used to convert the distance of a data sample to a weight).
Nia fails to explicitly disclose, “is a modality-specific kernel function corresponding to a modality of encoded features for which the distance is determined”.
Carreira teaches “is a modality-specific kernel function corresponding to a modality of encoded features for which the distance is determined” (See [0060]; if the entity is multimodal, each type or domain of data can also be associated with one or more modality-specific features).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Nia and Carreira before them to modify Nia to specify using a modality-specific kernel function when converting distances. One would be motivated to do so because the AI model being used makes use of a multimodal AI model instance, so it would be advantageous to use a kernel function that considers the multimodal nature of the model.
Regarding claim 7, Nia discloses “iteratively tuning hyperparameters of the interpretable surrogate model by comparing true explanations with outputs generated by the interpretable surrogate model.” (See [0061], [0166]; hyperparameters of the model are iteratively tuned when a determination to generate a new set of hyperparameter specifications is made. By comparing the output of a black-box model to the output of a trained surrogate model, Nia compares true explanations with outputs generated by the surrogate model).
Regarding claim 10 and 19, these claims are similar in scope to claim 1.
Regarding claim 11, this claim is similar in scope to claim 2.
Regarding claim 12, this claim is similar in scope to claim 3.
Regarding claim 14, this claim is similar in scope to claim 5.
Regarding claim 16 and 20, these claims are similar in scope to claim 7.
Claim Rejections - 35 USC § 103
Claims 4, 13 are rejected under 35 U.S.C. 103 as being unpatentable over Nia (US 20220129791 A1), in view of Carreira (US 20250103856 A1) and Ho (US 20240100693 A1), and further view of Mathews (US 20210097176 A1).
Regarding claim 4, Nia discloses “determining the distance… comprises: using a modality-specific distance metric corresponding to a modality of encoded features for which the distance is determined.” (See [0064]; a distance is determined by determining a radius of a hypersphere, where the hypersphere is the plurality of known data samples and the radius is the distance between one or more instances).
Nia fails to explicitly disclose, “the distance between the one or more encoded features of each perturbed instance and the corresponding one or more of the one or more encoded features of the multimodal AI model instance”.
Mathews teaches “the distance between the one or more encoded features of each perturbed instance and the corresponding one or more of the one or more encoded features of the multimodal AI model instance” (See [0033]; Mathews discloses finding the distance between a perturbed input and its corresponding initial input from the model training dataset)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Nia and Mathews before them to modify Nia to find the distance between features of each perturbed instance and the corresponding features of each AI model instance. One would be motivated to do so in order to obtain the difference between a perturbed instance and its original instance from the AI model and use the difference to determine a certain output, see e.g., [0091], where Mathews teaches determining a difference between an input from a model training dataset and the perturbed version of the input to determine which layers need to be replaced with a tree-based structure.
Regarding claim 13, this claim is similar in scope to claim 4.
Claim Rejections - 35 USC § 103
Claims 6, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Nia (US 20220129791 A1), in view of Carreira (US 20250103856 A1), and further view of Ribeiro (LIME: Local Interpretable Model-agnostic Explanations).
Regarding claim 6, Nia fails to explicitly disclose, “the interpretable surrogate model is a sparse linear model comprising weights corresponding to feature importance values”.
However, Ribeiro discloses “the interpretable surrogate model is a sparse linear model comprising weights corresponding to feature importance values”. (See [Page 1, Paragraph 1]; LIME is an interpretable surrogate model that is a sparse linear model. Because LIME is interpretable, its weights are meaningful, which indicates that they can be used to indicate values that are important (feature importance values)).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Nia and Ribeiro before them to modify Nia to use a sparse linear model for the surrogate model. One would be motivated to do so to improve the AI model with an explanation algorithm that does not require a complex model to function, see e.g., [Page 1, Paragraph 1], where Ribeiro describes using a simpler model to help with explaining the AI model’s outputs.
Regarding claim 15, this claim is similar in scope to claim 15.
Claim Rejections - 35 USC § 103
Claims 8, 9, 17, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Nia (US 20220129791 A1), in view of Carreira (US 20250103856 A1), and further view of Sweeney (US 20230374589 A1).
Regarding claim 8, Nia fails to explicitly disclose, “true explanations comprise true feature attribution weights generated using a logistic regression model”.
Sweeney teaches “true explanations comprise true feature attribution weights generated using a logistic regression model” (See [0040]; hyperparameter optimization is accomplished with logistic regressions by showing how a risk score (true feature attribution weight) is determined by using a parameter-based method to generate a risk score, which includes using logistic regression).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Nia and Sweeney before them to modify Nia to incorporate logistic regressions to accomplish optimizing the hyperparameters to generate true feature attribution weights. One would be motivated to do so for the purpose of verifying if the model is outputting correct data, see e.g., [0040] where Sweeney describes how to calculate a risk score by using linear regression, a parameter-based method for understanding how variables and hyperparameters are influenced by different inputs.
Regarding claim 9, Nia fails to explicitly disclose, “values for the hyperparameters are determined based upon a Pearson correlation coefficient or a normalized discounted cumulative gain.”.
Sweeney teaches “values for the hyperparameters are determined based upon a Pearson correlation coefficient or a normalized discounted cumulative gain.” (See [0186]; values for the hyperparameters are determined based on a Pearson correlation by sorting the absolute value of a group of genes' Pearson correlation and then using a classifier model to rank the genes to determine values for the hyperparameters).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Nia and Sweeney before them to modify Nia to determine hyperparameter values using a Pearson correlation coefficient. One would be motivated to do so to find hyperparameters that have a higher correlation with the desired results for the purpose of iteratively tuning the hyperparameters of the model, see e.g., [0186], where Sweeney describes using the results of the Pearson correlation with class labels to determine which values would be used for the hyperparameters.
Regarding claim 17, this claim is similar in scope to claim 8.
Regarding claim 18, this claim is similar in scope to claim 9.
Response to Arguments
Applicant's arguments regarding the 35 USC 103 rejection are moot in view of the new grounds of rejection necessitated by applicant's amendments.
The rejection of Claim 1 under 35 U.S.C. 103 has been maintained. Similarly, the rejection of Claims 10 and 19 under 35 U.S.C. 103 have been maintained.
The rejection of Claims 2-7 under 35 U.S.C. 103, which depend directly from Claim 1 have been maintained.
The rejection of Claims 8 and 9 under 35 U.S.C. 103, which depend directly from Claim 7 have been maintained.
The rejection of Claims 11-16 under 35 U.S.C. 103, which depend directly from Claim 10 have been maintained.
The rejection of Claims 17 and 18 under 35 U.S.C. 103, which depend directly from Claim 16 have been maintained.
The rejection of Claim 20 under 35 U.S.C. 103, which depends directly from Claim 19 has been maintained.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID KIM whose telephone number is (571)272-4331. The examiner can normally be reached 7:30 AM - 4:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Ell can be reached at (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.K./Examiner, Art Unit 2141
/BEN M RIFKIN/Primary Examiner, Art Unit 2123
Prosecution Insights