DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on April 12, is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea (mental process) without significantly more.
Claim 1:
Regarding claim 1, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “A system for mitigating biases during testing of a machine learning model, comprising:”, and a system or machine is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a mental process but for recitation of generic computer components:
“and evaluating at least one of the set of outputs against a respective expected output, wherein the respective expected output is determined based at least in part upon a historical record associated with the set of real-world input data;” (this is a mental process, a person could mentally evaluate a set of outputs against a respective expected output, he expected output can also be mentally determined based on a historical record associated with a set of real world data, see MPEP § 2106.04(a)(2)(III)),
“determine that more than a threshold percentage of outputs from among the set of outputs differ from respective expected outputs;” (this is a mental process, a person could mentally determine that more than a determined threshold percentage of outputs from a set of outputs differ from respective expected outputs, see MPEP § 2106.04(a)(2)(III)),
“in response determining that more than the threshold percentage of outputs from among the set of outputs differ from respective expected outputs, determine that the machine learning model is biased;” (this is a mental process, a person could mentally evaluate determining a model being biased based on a determination of a threshold percentage of outputs from a set of outputs differ from respective expected outputs, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process and mathematical concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
“A system for mitigating biases during testing of a machine learning model, comprising: a memory…” (A system comprising a memory is considered generic computer component being used as tool to perform functions of the judicial exception – see MPEP § 2106.05(f)),
“configured to store a machine learning model and a training dataset, wherein the training dataset comprises a set of datapoints;” (Storing a machine learning model and a training dataset comprising of datapoints is considered insignificant extra-solution activity of mere data gathering – see MPEP § 2106.05(g)),
“and a processor, operably coupled to the memory, and configured to:” (Using a processor is considered generic computer component being used as tool to perform functions of the judicial exception – see MPEP § 2106.05(f)),
“access the machine learning model, wherein the machine learning model is trained using the training dataset;” (Accessing a model trained using a training dataset is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)),
“test the machine learning model, wherein testing the machine learning model comprises:” (testing a machine learning model is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)),
“inputting a set of real-world input data to the machine learning model;” (Inputting a set of real-world input data to the machine learning model is considered insignificant extra-solution activity of mere data gathering – see MPEP § 2106.05(g)),
“receiving a set of outputs from the machine learning model;” (receiving a set of outputs from the machine learning model is considered insignificant extra-solution activity of mere data gathering – see MPEP § 2106.05(g)),
“and in response to determining that the machine learning model is biased, perform one or more corrective actions, wherein the one or more corrective actions comprise adjusting one or more parameters associated with the machine learning model, wherein the one or more parameters comprise a weight value or a bias value.” (performing corrective actions by adjusting parameters is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)),
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, additional elements iv, and vi recites generic computer component being used as tool to perform functions of the judicial exception, additional elements vii, viii and xi recites mere instructions to apply an exception using generic computer, and additional elements v, ix, and x recites insignificant extra-solution activity of mere data gathering, which is a well understood routine and conventional activity, see receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362, which is not indicative of significantly more.
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim 2:
Regarding claim 2, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis to claim 1. Further, claim 2 recites the following additional elements:
“The system of Claim 1, wherein the one or more corrective actions further comprise: identifying a second datapoint, within the training dataset, that is associated with an incorrect label compared to a counterpart expected datapoint;” (this is a mental process, a person could mentally identify another datapoint within a training dataset associated with an incorrect label in comparison to a counterpart expected datapoint, see MPEP § 2106.04(a)(2)(III)),
“identifying a second label associated with the counterpart expected datapoint;” (this is a mental process, a person can mentally identify another label associated with a counterpart expected datapoint, see MPEP § 2106.04(a)(2)(III)),
“associating the second datapoint to the second label;” (this is a mental process, a person can mentally evaluate associating a second datapoint to a second label, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“and retraining the machine learning model using a revised training dataset that comprises the second datapoint associated with the second label.” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 3:
Regarding claim 3, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis to claim 1. Further, claim 3 recites the following additional elements:
“The system of Claim 1, wherein the one or more corrective actions further comprise: identifying a third datapoint, within the training dataset, that is incompatible with the machine learning model;” (this is a mental process, a person could mentally identify another datapoint within a training dataset, that is incompatible with a model, see MPEP § 2106.04(a)(2)(III)),
“generating an updated third datapoint with a first data structure that is compatible with the machine learning model;” (this is a mental process, a person can mentally evaluate generating an updated third datapoint of a first data structure that is compatible with a model, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“and retraining the machine learning model using a revised training dataset that comprises the updated third datapoint.” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 4:
Regarding claim 4, it is dependent upon claim 3, and thereby incorporates the limitations of, and corresponding analysis to claim 3. Further, claim 4 recites the following additional elements:
“The system of Claim 3, wherein determining that the third datapoint is incompatible with the machine learning model comprises: determining that the machine learning model is configured to accept the first data structure;” (this is a mental process, a person could mentally determine if a model can be configured to accept a data structure, see MPEP § 2106.04(a)(2)(III)),
“determining that the third datapoint is associated with a second data structure;” (this is a mental process, a person can mentally determine a third datapoint being associated with a second data structure, see MPEP § 2106.04(a)(2)(III)),
“and determining that the second data structure does not correspond with the first data structure.” (this is a mental process, a person can mentally determine that a second data structure does not correspond to a first data structure, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 5:
Regarding claim 5, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis to claim 1. Further, claim 5 recites the following additional elements:
“The system of Claim 1, wherein the one or more corrective actions further comprise: determining that a fifth label associated with a fifth datapoint, within the training dataset, is incompatible with the machine learning model;” (this is a mental process, a person could mentally determine that a fifth label associated with a fifth datapoint within a training dataset, is incompatible with a model, see MPEP § 2106.04(a)(2)(III)),
“generating an updated fifth label with a third data structure that is compatible with the machine learning model;” (this is a mental process, a person can mentally evaluate generating an updated fifth label of a third data structure that is compatible with a model, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“and retraining the machine learning model using a revised training dataset that comprises fifth datapoint associated with the updated fifth label.” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 6:
Regarding claim 6, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis to claim 1. Further, claim 6 recites the following additional elements:
“The system of Claim 1, wherein the one or more corrective actions further comprise: accessing a set of expected datapoints that is expected to be present in the training dataset;” (this is a mental process, a person could mentally access and observe a set of expected datapoints that are expected to be present in a training dataset, see MPEP § 2106.04(a)(2)(III)),
“comparing each of the set of datapoints with a counterpart expected datapoint from among the set of expected datapoints;” (this is a mental process, a person can mentally compare each set of datapoints with a counterpart expected datapoint from , see MPEP § 2106.04(a)(2)(III)),
“determining that a fourth datapoint is missing from the training dataset based at least in part upon determining that the fourth datapoint is found in the set of expected datapoints;” (this is a mental process, a person can mentally determine a fourth datapoint is missing from a training dataset based on that datapoint being found in a set of expected datapoints, see MPEP § 2106.04(a)(2)(III)),
“adding the fourth datapoint to the training dataset;” (this is a mental process, a person can mentally add a fourth datapoint to a training dataset, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“and retraining the machine learning model using a revised training dataset that comprises the fourth datapoint.” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 7:
Regarding claim 7, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis to claim 1. Further, claim 7 recites the following additional elements:
“The system of Claim 1, wherein the one or more corrective actions further comprise: identifying a first datapoint, within the training dataset, that is missing a first label;” (this is a mental process, a person could mentally identify a first datapoint within a training dataset that is missing a first label, see MPEP § 2106.04(a)(2)(III)),
“adding the first label to the first datapoint;” (this is a mental process, a person can mentally add a first label to a first datapoint, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“and retraining the machine learning model using a revised training dataset that comprises the first datapoint associated with the first label.” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 8:
Regarding claim 8, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “A method for mitigating biases during testing of a machine learning model, comprising:”, and a method or process is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a mental process but for recitation of generic computer components:
“and evaluating at least one of the set of outputs against a respective expected output, wherein the respective expected output is determined based at least in part upon a historical record associated with the set of real-world input data;” (this is a mental process, a person could mentally evaluate a set of outputs against a respective expected output, he expected output can also be mentally determined based on a historical record associated with a set of real world data, see MPEP § 2106.04(a)(2)(III)),
“determine that more than a threshold percentage of outputs from among the set of outputs differ from respective expected outputs;” (this is a mental process, a person could mentally determine that more than a determined threshold percentage of outputs from a set of outputs differ from respective expected outputs, see MPEP § 2106.04(a)(2)(III)),
“in response determining that more than the threshold percentage of outputs from among the set of outputs differ from respective expected outputs, determine that the machine learning model is biased;” (this is a mental process, a person could mentally evaluate determining a model being biased based on a determination of a threshold percentage of outputs from a set of outputs differ from respective expected outputs, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process and mathematical concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
“A method for mitigating biases during testing of a machine learning model, comprising: storing a machine learning model and a training dataset, wherein the training dataset comprises a set of datapoints;” (Storing a machine learning model and a training dataset comprising of datapoints is considered insignificant extra-solution activity of mere data gathering – see MPEP § 2106.05(g)),
“accessing the machine learning model, wherein the machine learning model is trained using the training dataset;” (Accessing a model trained using a training dataset is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)),
“testing the machine learning model, wherein testing the machine learning model comprises:” (testing a machine learning model is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)),
“inputting a set of real-world input data to the machine learning model;” (Inputting a set of real-world input data to the machine learning model is considered insignificant extra-solution activity of mere data gathering – see MPEP § 2106.05(g)),
“receiving a set of outputs from the machine learning model;” (receiving a set of outputs from the machine learning model is considered insignificant extra-solution activity of mere data gathering – see MPEP § 2106.05(g)),
“and in response to determining that the machine learning model is biased, perform one or more corrective actions, wherein the one or more corrective actions comprise adjusting one or more parameters associated with the machine learning model, wherein the one or more parameters comprise a weight value or a bias value.” (performing corrective actions by adjusting parameters is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)),
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, additional elements v, vi, and ix recites mere instructions to apply an exception using generic computer, and additional elements iv, vii, and viii recites insignificant extra-solution activity of mere data gathering, which is a well understood routine and conventional activity, see receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362, which is not indicative of significantly more.
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim 9:
Regarding claim 9, it is dependent upon claim 8, and thereby incorporates the limitations of, and corresponding analysis to claim 8. Further, claim 9 recites the following additional elements:
“The method of Claim 8, wherein the one or more corrective actions further comprise: identifying a second datapoint, within the training dataset, that is associated with an incorrect label compared to a counterpart expected datapoint;” (this is a mental process, a person could mentally identify another datapoint within a training dataset associated with an incorrect label in comparison to a counterpart expected datapoint, see MPEP § 2106.04(a)(2)(III)),
“identifying a second label associated with the counterpart expected datapoint;” (this is a mental process, a person can mentally identify another label associated with a counterpart expected datapoint, see MPEP § 2106.04(a)(2)(III)),
“associating the second datapoint to the second label;” (this is a mental process, a person can mentally evaluate associating a second datapoint to a second label, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“and retraining the machine learning model using a revised training dataset that comprises the second datapoint associated with the second label.” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 10:
Regarding claim 10, it is dependent upon claim 8, and thereby incorporates the limitations of, and corresponding analysis to claim 8. Further, claim 10 recites the following additional elements:
“The method of Claim 8, wherein the one or more corrective actions further comprise: identifying a third datapoint, within the training dataset, that is incompatible with the machine learning model;” (this is a mental process, a person could mentally identify another datapoint within a training dataset, that is incompatible with a model, see MPEP § 2106.04(a)(2)(III)),
“generating an updated third datapoint with a first data structure that is compatible with the machine learning model;” (this is a mental process, a person can mentally evaluate generating an updated third datapoint of a first data structure that is compatible with a model, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“and retraining the machine learning model using a revised training dataset that comprises the updated third datapoint.” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 11:
Regarding claim 11, it is dependent upon claim 10, and thereby incorporates the limitations of, and corresponding analysis to claim 10. Further, claim 11 recites the following additional elements:
“The method of Claim 8, wherein determining that the third datapoint is incompatible with the machine learning model comprises: determining that the machine learning model is configured to accept the first data structure;” (this is a mental process, a person could mentally determine if a model can be configured to accept a data structure, see MPEP § 2106.04(a)(2)(III)),
“determining that the third datapoint is associated with a second data structure;” (this is a mental process, a person can mentally determine a third datapoint being associated with a second data structure, see MPEP § 2106.04(a)(2)(III)),
“and determining that the second data structure does not correspond with the first data structure.” (this is a mental process, a person can mentally determine that a second data structure does not correspond to a first data structure, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 12:
Regarding claim 12, it is dependent upon claim 8, and thereby incorporates the limitations of, and corresponding analysis to claim 8. Further, claim 12 recites the following additional elements:
“The method of Claim 8, wherein the one or more corrective actions further comprise: determining that a fifth label associated with a fifth datapoint, within the training dataset, is incompatible with the machine learning model;” (this is a mental process, a person could mentally determine that a fifth label associated with a fifth datapoint within a training dataset, is incompatible with a model, see MPEP § 2106.04(a)(2)(III)),
“generating an updated fifth label with a third data structure that is compatible with the machine learning model;” (this is a mental process, a person can mentally evaluate generating an updated fifth label of a third data structure that is compatible with a model, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“and retraining the machine learning model using a revised training dataset that comprises fifth datapoint associated with the updated fifth label.” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 13:
Regarding claim 13, it is dependent upon claim 8, and thereby incorporates the limitations of, and corresponding analysis to claim 8. Further, claim 13 recites the following additional elements:
“The method of Claim 8, wherein the one or more corrective actions further comprise: accessing a set of expected datapoints that is expected to be present in the training dataset;” (this is a mental process, a person could mentally access and observe a set of expected datapoints that are expected to be present in a training dataset, see MPEP § 2106.04(a)(2)(III)),
“comparing each of the set of datapoints with a counterpart expected datapoint from among the set of expected datapoints;” (this is a mental process, a person can mentally compare each set of datapoints with a counterpart expected datapoint from , see MPEP § 2106.04(a)(2)(III)),
“determining that a fourth datapoint is missing from the training dataset based at least in part upon determining that the fourth datapoint is found in the set of expected datapoints;” (this is a mental process, a person can mentally determine a fourth datapoint is missing from a training dataset based on that datapoint being found in a set of expected datapoints, see MPEP § 2106.04(a)(2)(III)),
“adding the fourth datapoint to the training dataset;” (this is a mental process, a person can mentally add a fourth datapoint to a training dataset, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“and retraining the machine learning model using a revised training dataset that comprises the fourth datapoint.” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 14:
Regarding claim 14, it is dependent upon claim 8, and thereby incorporates the limitations of, and corresponding analysis to claim 8. Further, claim 14 recites the following additional elements:
“The method of Claim 8, wherein the one or more corrective actions further comprise: identifying a first datapoint, within the training dataset, that is missing a first label;” (this is a mental process, a person could mentally identify a first datapoint within a training dataset that is missing a first label, see MPEP § 2106.04(a)(2)(III)),
“adding the first label to the first datapoint;” (this is a mental process, a person can mentally add a first label to a first datapoint, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“and retraining the machine learning model using a revised training dataset that comprises the first datapoint associated with the first label.” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 15:
Regarding claim 15, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “A non-transitory computer-readable medium storing instructions”, the claim does not fall within at least one of the four categories of patent eligible subject matter because there is no definition of non-transitory computer readable medium in the applicant’s specification. Therefore, under BRI non-transitory computer readable medium could include signals making the claim signals per se. As such the claim is rejected under failing to fall into the one of the statutory categories of the patent eligible subject matter.
In step 2A prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a mental process but for recitation of generic computer components:
“and evaluating at least one of the set of outputs against a respective expected output, wherein the respective expected output is determined based at least in part upon a historical record associated with the set of real-world input data;” (this is a mental process, a person could mentally evaluate a set of outputs against a respective expected output, he expected output can also be mentally determined based on a historical record associated with a set of real world data, see MPEP § 2106.04(a)(2)(III)),
“determine that more than a threshold percentage of outputs from among the set of outputs differ from respective expected outputs;” (this is a mental process, a person could mentally determine that more than a determined threshold percentage of outputs from a set of outputs differ from respective expected outputs, see MPEP § 2106.04(a)(2)(III)),
“in response determining that more than the threshold percentage of outputs from among the set of outputs differ from respective expected outputs, determine that the machine learning model is biased;” (this is a mental process, a person could mentally evaluate determining a model being biased based on a determination of a threshold percentage of outputs from a set of outputs differ from respective expected outputs, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under the broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process and mathematical concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
“A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to:” (A non-transitory computer-readable medium executed by a processor is considered generic computer component being used as tool to perform functions of the judicial exception – see MPEP § 2106.05(f)),
“store a machine learning model and a training dataset, wherein the training dataset comprises a set of datapoints;” (Storing a machine learning model and a training dataset comprising of datapoints is considered insignificant extra-solution activity of mere data gathering – see MPEP § 2106.05(g)),
“access the machine learning model, wherein the machine learning model is trained using the training dataset;” (Accessing a model trained using a training dataset is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)),
“test the machine learning model, wherein testing the machine learning model comprises:” (testing a machine learning model is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)),
“inputting a set of real-world input data to the machine learning model;” (Inputting a set of real-world input data to the machine learning model is considered insignificant extra-solution activity of mere data gathering – see MPEP § 2106.05(g)),
“receiving a set of outputs from the machine learning model;” (receiving a set of outputs from the machine learning model is considered insignificant extra-solution activity of mere data gathering – see MPEP § 2106.05(g)),
“and in response to determining that the machine learning model is biased, perform one or more corrective actions, wherein the one or more corrective actions comprise adjusting one or more parameters associated with the machine learning model, wherein the one or more parameters comprise a weight value or a bias value.” (performing corrective actions by adjusting parameters is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)),
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, additional element iv recites generic computer component being used as tool to perform functions of the judicial exception, additional elements vi, vii, and x recites mere instructions to apply an exception using generic computer, and additional elements v, viii, and ix recites insignificant extra-solution activity of mere data gathering, which is a well understood routine and conventional activity, see receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362, which is not indicative of significantly more.
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim 16:
Regarding claim 16, it is dependent upon claim 15, and thereby incorporates the limitations of, and corresponding analysis to claim 15. Further, claim 16 recites the following additional elements:
“The non-transitory computer-readable medium of Claim 15, wherein the one or more corrective actions further comprise: identifying a second datapoint, within the training dataset, that is associated with an incorrect label compared to a counterpart expected datapoint;” (this is a mental process, a person could mentally identify another datapoint within a training dataset associated with an incorrect label in comparison to a counterpart expected datapoint, see MPEP § 2106.04(a)(2)(III)),
“identifying a second label associated with the counterpart expected datapoint;” (this is a mental process, a person can mentally identify another label associated with a counterpart expected datapoint, see MPEP § 2106.04(a)(2)(III)),
“associating the second datapoint to the second label;” (this is a mental process, a person can mentally evaluate associating a second datapoint to a second label, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“and retraining the machine learning model using a revised training dataset that comprises the second datapoint associated with the second label.” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 17:
Regarding claim 17, it is dependent upon claim 15, and thereby incorporates the limitations of, and corresponding analysis to claim 15. Further, claim 17 recites the following additional elements:
“The non-transitory computer-readable medium of Claim 15, wherein the one or more corrective actions further comprise: identifying a third datapoint, within the training dataset, that is incompatible with the machine learning model;” (this is a mental process, a person could mentally identify another datapoint within a training dataset, that is incompatible with a model, see MPEP § 2106.04(a)(2)(III)),
“generating an updated third datapoint with a first data structure that is compatible with the machine learning model;” (this is a mental process, a person can mentally evaluate generating an updated third datapoint of a first data structure that is compatible with a model, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“and retraining the machine learning model using a revised training dataset that comprises the updated third datapoint.” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 18:
Regarding claim 18, it is dependent upon claim 17, and thereby incorporates the limitations of, and corresponding analysis to claim 17. Further, claim 18 recites the following additional elements:
“The non-transitory computer-readable medium of Claim 17, wherein determining that the third datapoint is incompatible with the machine learning model comprises: determining that the machine learning model is configured to accept the first data structure;” (this is a mental process, a person could mentally determine if a model can be configured to accept a data structure, see MPEP § 2106.04(a)(2)(III)),
“determining that the third datapoint is associated with a second data structure;” (this is a mental process, a person can mentally determine a third datapoint being associated with a second data structure, see MPEP § 2106.04(a)(2)(III)),
“and determining that the second data structure does not correspond with the first data structure.” (this is a mental process, a person can mentally determine that a second data structure does not correspond to a first data structure, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 19:
Regarding claim 19, it is dependent upon claim 15, and thereby incorporates the limitations of, and corresponding analysis to claim 15. Further, claim 19 recites the following additional elements:
“The non-transitory computer-readable medium of Claim 15, wherein the one or more corrective actions further comprise: determining that a fifth label associated with a fifth datapoint, within the training dataset, is incompatible with the machine learning model;” (this is a mental process, a person could mentally determine that a fifth label associated with a fifth datapoint within a training dataset, is incompatible with a model, see MPEP § 2106.04(a)(2)(III)),
“generating an updated fifth label with a third data structure that is compatible with the machine learning model;” (this is a mental process, a person can mentally evaluate generating an updated fifth label of a third data structure that is compatible with a model, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“and retraining the machine learning model using a revised training dataset that comprises fifth datapoint associated with the updated fifth label.” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 20:
Regarding claim 20, it is dependent upon claim 15, and thereby incorporates the limitations of, and corresponding analysis to claim 15. Further, claim 20 recites the following additional elements:
“determine that an accuracy score associated with the machine learning model is less than a threshold score, wherein determining that the machine learning model is biased is in response to determining that the accuracy score associated with the machine learning model is less than the threshold score.” (this is a mental process, a person could mentally determine that an accuracy score in association with a model is less than a threshold score where the determination is in response to determining that a accuracy score associated with the model is less than the threshold score, see MPEP § 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process concept grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
“The non-transitory computer-readable medium of Claim 15, wherein the instructions further cause the processor to…” (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP § 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 8, 15, and 20 are rejected under 35 U.S.C. 102(a)(1) and 102(a)(2) as being anticipated by Fathi A. et al, (US. Patent Application Publication 20250238621 A1) effectively filed on January 26, 2024, (hereafter Fathi).
Claim 1:
Regarding claim 1, Fathi teaches “A system for mitigating biases during testing of a machine learning model, comprising: a memory configured to store a machine learning model and a training dataset, wherein the training dataset comprises a set of datapoints; and a processor, operably coupled to the memory, and configured to: access the machine learning model, wherein the machine learning model is trained using the training dataset;”
See Fathi in paragraph [0007] describing, “In yet another embodiment, a system may include one or more processors, and memory having instructions that, when executed by the one or more processors, cause the one or more processors to: determine an AI model to test, wherein the AI model is a trained deep learning model such that a gradient of the trained deep learning model with respect to each token of the sentence is calculable, receive a text sentence and an identification of a class, calculate a bias direction with respect to the class for an embedding model used by an artificial intelligence model to be analyzed, and for each token in the text sentence, calculate a protected gradient score.” Further, see Fathi in paragraph [0023] describing, “Within the diagram of system 100, blocks 110-114 describe a model and its training. Training data of block 110 is used to train a base model to form a trained model 112. The trained model 112 is the model for which bias will be detected.” Further, see Fathi in paragraph [0069] describing, “In many examples, the training data includes pairs of input and desired output given the input.” Here, Fathi establishes the training data comprising of datapoints with the pairs of input and desired output.
Further, Fathi teaches “test the machine learning model, wherein testing the machine learning model comprises: inputting a set of real-world input data to the machine learning model;”
See Fathi in paragraph [0007] describing, “In yet another embodiment, a system may include one or more processors, and memory having instructions that, when executed by the one or more processors, cause the one or more processors to: determine an AI model to test”. Further, see Fathi in paragraph [0068] describing, “Operation 422 includes deploying the trained model 402 for use in production, such as providing the trained model 402 with real-world input data and produce output data used in a real-world process.”
Further, Fathi teaches “receiving a set of outputs from the machine learning model;”
See Fathi in paragraph [0018] describing, “In an embodiment, upon a generative AI model creating a new derivate work it first commences its training process. In this embodiment, the generative AI model is built for the purpose of monitoring copyrighted data use. In this embodiment, as the generative AI model generates derivative works, it continually tracks the use of copyrighted data, ensuring no loss of information.” Here, Fathi establishes copyright information for a derivative of AI generated work being tracked, the derivative can be seen as a snippet. Further, see Fathi in paragraph [0025] describing, “In an embodiment, the method includes employing a copyright attribution report generator. In this embodiment, the copyright attribution report generator compiles essential information, including the newly created work, its unique digital creative DNA profile, and traced influences to generate a copyright attribution report. In this embodiment, the report serves as a comprehensive record for copyright attribution purposes.” Here, Fathi further establishes generating copyright information.
Further, Fathi teaches “and evaluating at least one of the set of outputs against a respective expected output, wherein the respective expected output is determined based at least in part upon a historical record associated with the set of real-world input data;”
See Fathi in paragraph [0068] describing, “Operation 416 includes providing a portion of the training data to the model 402. This can include providing the training data in a format usable by the model 402. The framework 400 (e.g., via the interface 404) can cause the model 402 to produce an output based on the input. Operation 418 can follow operation 416. Operation 418 includes comparing the expected output with the actual output.” Further, see Fathi in paragraph [0070] describing, “ Where implementations involve personal or corporate data, that data can be stored in a manner consistent with relevant laws and with a defined privacy policy. In certain circumstances, the data can be decentralized, anonymized, or fuzzed to reduce the amount of accurate private data that is stored or accessible at a particular computer.” Here, Fathi establishes the data used in implementations using involving personal or corporate data which is being interpreted as the historical record associated with a set of real-world input data, as established in previous limitation the model uses real-world data to produce output data.
Further, Fathi teaches “determine that more than a threshold percentage of outputs from among the set of outputs differ from respective expected outputs;”
See Fathi in paragraph [0068] describing, “Operation 418 includes comparing the expected output with the actual output. In an example, this can include applying a loss function to determine the difference between expected and actual. This value can be used to determine how training is progressing. Operation 420 can follow operation 418. Operation 420 includes updating the model 402 based on the result of the comparison. This can take any of a variety of forms depending on the nature of the model 402. Where the model 402 includes weights, the weights can be modified to increase the likelihood that the model 402 will produce correct output given an input. Depending on the model 402, backpropagation or other techniques can be used to update the model 402. Operation 422 can follow operation 420. Operation 422 includes determining whether a stopping criterion has been reached, such as based on the output of the loss function (e.g., actual value or change in value over time). In addition or instead, whether the stopping criterion has been reached can be determined based on a number of training epochs that have occurred or an amount of training data that has been used. In some examples, satisfaction of the stopping criterion can include If the stopping criterion has not been satisfied, the flow of the method can return to operation 414. If the stopping criterion has been satisfied, the flow can move to operation 422.”
Further, Fathi teaches “in response determining that more than the threshold percentage of outputs from among the set of outputs differ from respective expected outputs, determine that the machine learning model is biased;”
See Fathi in paragraph [0048] describing, “In examples, the process 200 is repeated until a bias of the model is below a threshold level of bias (or above a threshold level of fairness). Once bias is below a bias threshold (or above a fairness threshold), the AI model (or a remediated AI model) may be deployed in production or otherwise used.”
Further, Fathi teaches “and in response to determining that the machine learning model is biased, perform one or more corrective actions, wherein the one or more corrective actions comprise adjusting one or more parameters associated with the machine learning model, wherein the one or more parameters comprise a weight value or a bias value.”
See Fathi in paragraph [0068] describing, “Operation 420 includes updating the model 402 based on the result of the comparison. This can take any of a variety of forms depending on the nature of the model 402. Where the model 402 includes weights, the weights can be modified to increase the likelihood that the model 402 will produce correct output given an input. Depending on the model 402, backpropagation or other techniques can be used to update the model 402.
Claim 8:
Regarding claim 8, Fathi teaches “A method for mitigating biases during testing of a machine learning model, comprising: storing a machine learning model and a training dataset, wherein the training dataset comprises a set of datapoints; accessing the machine learning model, wherein the machine learning model is trained using the training dataset;”
See Fathi in paragraph [0007] describing, “In yet another embodiment, a system may include one or more processors, and memory having instructions that, when executed by the one or more processors, cause the one or more processors to: determine an AI model to test, wherein the AI model is a trained deep learning model such that a gradient of the trained deep learning model with respect to each token of the sentence is calculable, receive a text sentence and an identification of a class, calculate a bias direction with respect to the class for an embedding model used by an artificial intelligence model to be analyzed, and for each token in the text sentence, calculate a protected gradient score.” Further, see Fathi in paragraph [0023] describing, “Within the diagram of system 100, blocks 110-114 describe a model and its training. Training data of block 110 is used to train a base model to form a trained model 112. The trained model 112 is the model for which bias will be detected.” Further, see Fathi in paragraph [0069] describing, “In many examples, the training data includes pairs of input and desired output given the input.” Here, Fathi establishes the training data comprising of datapoints with the pairs of input and desired output.
Further, Fathi teaches “testing the machine learning model, wherein testing the machine learning model comprises: inputting a set of real-world input data to the machine learning model;”
See Fathi in paragraph [0007] describing, “In yet another embodiment, a system may include one or more processors, and memory having instructions that, when executed by the one or more processors, cause the one or more processors to: determine an AI model to test”. Further, see Fathi in paragraph [0068] describing, “Operation 422 includes deploying the trained model 402 for use in production, such as providing the trained model 402 with real-world input data and produce output data used in a real-world process.”
Further, Fathi teaches “receiving a set of outputs from the machine learning model;”
See Fathi in paragraph [0018] describing, “In an embodiment, upon a generative AI model creating a new derivate work it first commences its training process. In this embodiment, the generative AI model is built for the purpose of monitoring copyrighted data use. In this embodiment, as the generative AI model generates derivative works, it continually tracks the use of copyrighted data, ensuring no loss of information.” Here, Fathi establishes copyright information for a derivative of AI generated work being tracked, the derivative can be seen as a snippet. Further, see Fathi in paragraph [0025] describing, “In an embodiment, the method includes employing a copyright attribution report generator. In this embodiment, the copyright attribution report generator compiles essential information, including the newly created work, its unique digital creative DNA profile, and traced influences to generate a copyright attribution report. In this embodiment, the report serves as a comprehensive record for copyright attribution purposes.” Here, Fathi further establishes generating copyright information.
Further, Fathi teaches “and evaluating at least one of the set of outputs against a respective expected output, wherein the respective expected output is determined based at least in part upon a historical record associated with the set of real-world input data;”
See Fathi in paragraph [0068] describing, “Operation 416 includes providing a portion of the training data to the model 402. This can include providing the training data in a format usable by the model 402. The framework 400 (e.g., via the interface 404) can cause the model 402 to produce an output based on the input. Operation 418 can follow operation 416. Operation 418 includes comparing the expected output with the actual output.” Further, see Fathi in paragraph [0070] describing, “ Where implementations involve personal or corporate data, that data can be stored in a manner consistent with relevant laws and with a defined privacy policy. In certain circumstances, the data can be decentralized, anonymized, or fuzzed to reduce the amount of accurate private data that is stored or accessible at a particular computer.” Here, Fathi establishes the data used in implementations using involving personal or corporate data which is being interpreted as the historical record associated with a set of real-world input data, as established in previous limitation the model uses real-world data to produce output data.
Further, Fathi teaches “determining that more than a threshold percentage of outputs from among the set of outputs differ from respective expected outputs;”
See Fathi in paragraph [0068] describing, “Operation 418 includes comparing the expected output with the actual output. In an example, this can include applying a loss function to determine the difference between expected and actual. This value can be used to determine how training is progressing. Operation 420 can follow operation 418. Operation 420 includes updating the model 402 based on the result of the comparison. This can take any of a variety of forms depending on the nature of the model 402. Where the model 402 includes weights, the weights can be modified to increase the likelihood that the model 402 will produce correct output given an input. Depending on the model 402, backpropagation or other techniques can be used to update the model 402. Operation 422 can follow operation 420. Operation 422 includes determining whether a stopping criterion has been reached, such as based on the output of the loss function (e.g., actual value or change in value over time). In addition or instead, whether the stopping criterion has been reached can be determined based on a number of training epochs that have occurred or an amount of training data that has been used. In some examples, satisfaction of the stopping criterion can include If the stopping criterion has not been satisfied, the flow of the method can return to operation 414. If the stopping criterion has been satisfied, the flow can move to operation 422.”
Further, Fathi teaches “in response determining that more than the threshold percentage of outputs from among the set of outputs differ from respective expected outputs, determine that the machine learning model is biased;”
See Fathi in paragraph [0048] describing, “In examples, the process 200 is repeated until a bias of the model is below a threshold level of bias (or above a threshold level of fairness). Once bias is below a bias threshold (or above a fairness threshold), the AI model (or a remediated AI model) may be deployed in production or otherwise used.”
Further, Fathi teaches “and in response to determining that the machine learning model is biased, performing one or more corrective actions, wherein the one or more corrective actions comprise adjusting one or more parameters associated with the machine learning model, wherein the one or more parameters comprise a weight value or a bias value.”
See Fathi in paragraph [0068] describing, “Operation 420 includes updating the model 402 based on the result of the comparison. This can take any of a variety of forms depending on the nature of the model 402. Where the model 402 includes weights, the weights can be modified to increase the likelihood that the model 402 will produce correct output given an input. Depending on the model 402, backpropagation or other techniques can be used to update the model 402.
Claim 15:
Regarding claim 15, Fathi teaches “A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to: store a machine learning model and a training dataset, wherein the training dataset comprises a set of datapoints; access the machine learning model, wherein the machine learning model is trained using the training dataset;”
See Fathi in paragraph [0006] describing, “In another embodiment, a non-transitory computer readable medium having instructions that, when executed by one or more processors may cause the one or more processors to: determine an AI model to test, wherein the AI model is a trained deep learning model such that a gradient of the trained deep learning model with respect to each token of the sentence is calculable, receive a text sentence and an identification of a class, calculate a bias direction with respect to the class for an embedding model used by an artificial intelligence model to be analyzed, and for each token in the text sentence, calculate a protected gradient score.” Further, see Fathi in paragraph [0023] describing, “Within the diagram of system 100, blocks 110-114 describe a model and its training. Training data of block 110 is used to train a base model to form a trained model 112. The trained model 112 is the model for which bias will be detected.” Further, see Fathi in paragraph [0069] describing, “In many examples, the training data includes pairs of input and desired output given the input.” Here, Fathi establishes the training data comprising of datapoints with the pairs of input and desired output.
Further, Fathi teaches “test the machine learning model, wherein testing the machine learning model comprises: inputting a set of real-world input data to the machine learning model;”
See Fathi in paragraph [0007] describing, “In yet another embodiment, a system may include one or more processors, and memory having instructions that, when executed by the one or more processors, cause the one or more processors to: determine an AI model to test”. Further, see Fathi in paragraph [0068] describing, “Operation 422 includes deploying the trained model 402 for use in production, such as providing the trained model 402 with real-world input data and produce output data used in a real-world process.”
Further, Fathi teaches “receiving a set of outputs from the machine learning model;”
See Fathi in paragraph [0018] describing, “In an embodiment, upon a generative AI model creating a new derivate work it first commences its training process. In this embodiment, the generative AI model is built for the purpose of monitoring copyrighted data use. In this embodiment, as the generative AI model generates derivative works, it continually tracks the use of copyrighted data, ensuring no loss of information.” Here, Fathi establishes copyright information for a derivative of AI generated work being tracked, the derivative can be seen as a snippet. Further, see Fathi in paragraph [0025] describing, “In an embodiment, the method includes employing a copyright attribution report generator. In this embodiment, the copyright attribution report generator compiles essential information, including the newly created work, its unique digital creative DNA profile, and traced influences to generate a copyright attribution report. In this embodiment, the report serves as a comprehensive record for copyright attribution purposes.” Here, Fathi further establishes generating copyright information.
Further, Fathi teaches “and evaluating at least one of the set of outputs against a respective expected output, wherein the respective expected output is determined based at least in part upon a historical record associated with the set of real-world input data;”
See Fathi in paragraph [0068] describing, “Operation 416 includes providing a portion of the training data to the model 402. This can include providing the training data in a format usable by the model 402. The framework 400 (e.g., via the interface 404) can cause the model 402 to produce an output based on the input. Operation 418 can follow operation 416. Operation 418 includes comparing the expected output with the actual output.” Further, see Fathi in paragraph [0070] describing, “ Where implementations involve personal or corporate data, that data can be stored in a manner consistent with relevant laws and with a defined privacy policy. In certain circumstances, the data can be decentralized, anonymized, or fuzzed to reduce the amount of accurate private data that is stored or accessible at a particular computer.” Here, Fathi establishes the data used in implementations using involving personal or corporate data which is being interpreted as the historical record associated with a set of real-world input data, as established in previous limitation the model uses real-world data to produce output data.
Further, Fathi teaches “determine that more than a threshold percentage of outputs from among the set of outputs differ from respective expected outputs;”
See Fathi in paragraph [0068] describing, “Operation 418 includes comparing the expected output with the actual output. In an example, this can include applying a loss function to determine the difference between expected and actual. This value can be used to determine how training is progressing. Operation 420 can follow operation 418. Operation 420 includes updating the model 402 based on the result of the comparison. This can take any of a variety of forms depending on the nature of the model 402. Where the model 402 includes weights, the weights can be modified to increase the likelihood that the model 402 will produce correct output given an input. Depending on the model 402, backpropagation or other techniques can be used to update the model 402. Operation 422 can follow operation 420. Operation 422 includes determining whether a stopping criterion has been reached, such as based on the output of the loss function (e.g., actual value or change in value over time). In addition or instead, whether the stopping criterion has been reached can be determined based on a number of training epochs that have occurred or an amount of training data that has been used. In some examples, satisfaction of the stopping criterion can include If the stopping criterion has not been satisfied, the flow of the method can return to operation 414. If the stopping criterion has been satisfied, the flow can move to operation 422.”
Further, Fathi teaches “in response determining that more than the threshold percentage of outputs from among the set of outputs differ from respective expected outputs, determine that the machine learning model is biased;”
See Fathi in paragraph [0048] describing, “In examples, the process 200 is repeated until a bias of the model is below a threshold level of bias (or above a threshold level of fairness). Once bias is below a bias threshold (or above a fairness threshold), the AI model (or a remediated AI model) may be deployed in production or otherwise used.”
Further, Fathi teaches “and in response to determining that the machine learning model is biased, perform one or more corrective actions, wherein the one or more corrective actions comprise adjusting one or more parameters associated with the machine learning model, wherein the one or more parameters comprise a weight value or a bias value.”
See Fathi in paragraph [0068] describing, “Operation 420 includes updating the model 402 based on the result of the comparison. This can take any of a variety of forms depending on the nature of the model 402. Where the model 402 includes weights, the weights can be modified to increase the likelihood that the model 402 will produce correct output given an input. Depending on the model 402, backpropagation or other techniques can be used to update the model 402.
Claim 20:
Regarding claim 20, Fathi teaches the limitations of claim 15.
Further, Fathi teaches “The non-transitory computer-readable medium of Claim 15, wherein the instructions further cause the processor to determine that an accuracy score associated with the machine learning model is less than a threshold score, wherein determining that the machine learning model is biased is in response to determining that the accuracy score associated with the machine learning model is less than the threshold score.”
See Fathi in paragraph [0048] describing, “ In examples, the process 200 is repeated until a bias of the model is below a threshold level of bias (or above a threshold level of fairness). Once bias is below a bias threshold (or above a fairness threshold), the AI model (or a remediated AI model) may be deployed in production or otherwise used.”
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2, 9, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Fathi et al., in view of Basu D. et al, (US. Patent Application Publication 20220300557 A1) effectively filed on March 16, 2021, (hereafter Basu).
Claim 2:
Regarding claim 2, Fathi teaches the limitations of claim 1.
Fathi does not appear to explicitly teach “The system of Claim 1, wherein the one or more corrective actions further comprise: identifying a second datapoint, within the training dataset, that is associated with an incorrect label compared to a counterpart expected datapoint; identifying a second label associated with the counterpart expected datapoint; associating the second datapoint to the second label; and retraining the machine learning model using a revised training dataset that comprises the second datapoint associated with the second label.”,
However in the same field of art, Basu teaches “The system of Claim 1, wherein the one or more corrective actions further comprise: identifying a second datapoint, within the training dataset, that is associated with an incorrect label compared to a counterpart expected datapoint;”
See Basu in paragraph [0072] describing, “The dataset may include a set of datapoints that includes at least a first and a second datapoint.” Further, see Basu in paragraph [0096] describing, “The classifier model may predict the first label for the first datapoint. A second label of the set of labels that the classifier model is likely to confuse with the first label may be identified. The identification of the second label may be based on instances of incorrect label predictions of the classifier model for the first label and/or the second label.”
Further, Basu teaches “identifying a second label associated with the counterpart expected datapoint;”
See Basu in paragraph [0096] describing, “The classifier model may predict the first label for the first datapoint. A second label of the set of labels that the classifier model is likely to confuse with the first label may be identified. The identification of the second label may be based on instances of incorrect label predictions of the classifier model for the first label and/or the second label. The classifier model may predict the second label for the second datapoint.”
Further, Basu teaches “associating the second datapoint to the second label;”
See Basu in paragraph [0096] describing, “The classifier model may predict the second label for the second datapoint.”
Further, Basu teaches “and retraining the machine learning model using a revised training dataset that comprises the second datapoint associated with the second label.”
See Basu in paragraph [0096] describing, “In some embodiments, the classifier model may be updated based on a third label that includes an aggregation of the first label and the second label. The updated classifier model may predict the third label for each of the first datapoint and the second datapoint. In various embodiments, the set of labels may be updated to include the third label and to exclude each of the first label and the second label. Updating the set of labels may include updating ground-truth assignments of the set of datapoints based on the updated set of labels.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Basu by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Basu’s teachings of improving performance of classifiers for models.
One of ordinary skill in the art would be motivated to do so because by integrating Basu’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection and classification, one of ordinary skill in the art would bring “enhanced methods and systems for quantifying a classifier's performance with reduced statistical uncertainty in the quantification.” (Basu, paragraph [0005]).
Claim 9:
Regarding claim 9, Fathi teaches the limitations of claim 8.
Fathi does not appear to explicitly teach “The method of Claim 8, wherein the one or more corrective actions further comprise: identifying a second datapoint, within the training dataset, that is associated with an incorrect label compared to a counterpart expected datapoint; identifying a second label associated with the counterpart expected datapoint; associating the second datapoint to the second label; and retraining the machine learning model using a revised training dataset that comprises the second datapoint associated with the second label.”,
However in the same field of art, Basu teaches “The method of Claim 8, wherein the one or more corrective actions further comprise: identifying a second datapoint, within the training dataset, that is associated with an incorrect label compared to a counterpart expected datapoint;”
See Basu in paragraph [0072] describing, “The dataset may include a set of datapoints that includes at least a first and a second datapoint.” Further, see Basu in paragraph [0096] describing, “The classifier model may predict the first label for the first datapoint. A second label of the set of labels that the classifier model is likely to confuse with the first label may be identified. The identification of the second label may be based on instances of incorrect label predictions of the classifier model for the first label and/or the second label.”
Further, Basu teaches “identifying a second label associated with the counterpart expected datapoint;”
See Basu in paragraph [0096] describing, “The classifier model may predict the first label for the first datapoint. A second label of the set of labels that the classifier model is likely to confuse with the first label may be identified. The identification of the second label may be based on instances of incorrect label predictions of the classifier model for the first label and/or the second label. The classifier model may predict the second label for the second datapoint.”
Further, Basu teaches “associating the second datapoint to the second label;”
See Basu in paragraph [0096] describing, “The classifier model may predict the second label for the second datapoint.”
Further, Basu teaches “and retraining the machine learning model using a revised training dataset that comprises the second datapoint associated with the second label.”
See Basu in paragraph [0096] describing, “In some embodiments, the classifier model may be updated based on a third label that includes an aggregation of the first label and the second label. The updated classifier model may predict the third label for each of the first datapoint and the second datapoint. In various embodiments, the set of labels may be updated to include the third label and to exclude each of the first label and the second label. Updating the set of labels may include updating ground-truth assignments of the set of datapoints based on the updated set of labels.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Basu by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Basu’s teachings of improving performance of classifiers for models.
One of ordinary skill in the art would be motivated to do so because by integrating Basu’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection and classification, one of ordinary skill in the art would bring “enhanced methods and systems for quantifying a classifier's performance with reduced statistical uncertainty in the quantification.” (Basu, paragraph [0005]).
Claim 16:
Regarding claim 16, Fathi teaches the limitations of claim 15.
Fathi does not appear to explicitly teach “The non-transitory computer-readable medium of Claim 15, wherein the one or more corrective actions further comprise: identifying a second datapoint, within the training dataset, that is associated with an incorrect label compared to a counterpart expected datapoint; identifying a second label associated with the counterpart expected datapoint; associating the second datapoint to the second label; and retraining the machine learning model using a revised training dataset that comprises the second datapoint associated with the second label.”,
However in the same field of art, Basu teaches “The non-transitory computer-readable medium of Claim 15, wherein the one or more corrective actions further comprise: identifying a second datapoint, within the training dataset, that is associated with an incorrect label compared to a counterpart expected datapoint;”
See Basu in paragraph [0072] describing, “The dataset may include a set of datapoints that includes at least a first and a second datapoint.” Further, see Basu in paragraph [0096] describing, “The classifier model may predict the first label for the first datapoint. A second label of the set of labels that the classifier model is likely to confuse with the first label may be identified. The identification of the second label may be based on instances of incorrect label predictions of the classifier model for the first label and/or the second label.”
Further, Basu teaches “identifying a second label associated with the counterpart expected datapoint;”
See Basu in paragraph [0096] describing, “The classifier model may predict the first label for the first datapoint. A second label of the set of labels that the classifier model is likely to confuse with the first label may be identified. The identification of the second label may be based on instances of incorrect label predictions of the classifier model for the first label and/or the second label. The classifier model may predict the second label for the second datapoint.”
Further, Basu teaches “associating the second datapoint to the second label;”
See Basu in paragraph [0096] describing, “The classifier model may predict the second label for the second datapoint.”
Further, Basu teaches “and retraining the machine learning model using a revised training dataset that comprises the second datapoint associated with the second label.”
See Basu in paragraph [0096] describing, “In some embodiments, the classifier model may be updated based on a third label that includes an aggregation of the first label and the second label. The updated classifier model may predict the third label for each of the first datapoint and the second datapoint. In various embodiments, the set of labels may be updated to include the third label and to exclude each of the first label and the second label. Updating the set of labels may include updating ground-truth assignments of the set of datapoints based on the updated set of labels.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Basu by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Basu’s teachings of improving performance of classifiers for models.
One of ordinary skill in the art would be motivated to do so because by integrating Basu’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection and classification, one of ordinary skill in the art would bring “enhanced methods and systems for quantifying a classifier's performance with reduced statistical uncertainty in the quantification.” (Basu, paragraph [0005]).
Claim(s) 3-7, 10-14, and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Fathi et al., in view of Desmond M. et al, (US. Patent Application Publication 20210174196 A1) effectively filed on December 10, 2019, (hereafter Desmond).
Claim 3:
Regarding claim 3, Fathi teaches the limitations of claim 1.
Fathi does not appear to explicitly teach “The system of Claim 1, wherein the one or more corrective actions further comprise: identifying a third datapoint, within the training dataset, that is incompatible with the machine learning model; generating an updated third datapoint with a first data structure that is compatible with the machine learning model; and retraining the machine learning model using a revised training dataset that comprises the updated third datapoint.”,
However in the same field of art, Desmond teaches “The system of Claim 1, wherein the one or more corrective actions further comprise: identifying a third datapoint, within the training dataset, that is incompatible with the machine learning model;”
See Desmond in paragraph [0014] describing, “The method also includes training a model based on the plurality of data inputs. The method also includes generating a plurality of vector representations based on the model. Each of the plurality of vector representations corresponds to a unique one of the plurality of data inputs. The method also includes clustering the plurality of vector representations into one or more clusters using a density clustering algorithm. The method also includes analyzing a vector space that includes the one or more clusters to identify at least one vector representation corresponding to an outlier data input and/or a mislabeled data input. The method also includes forming a new plurality of data inputs having associated labels by removing the outlier data input from the plurality of data inputs in response to identifying an outlier data input and relabeling the data input to have an associated label of a classification type of the predominant classification type of the vector representations in the same cluster as the vector representation corresponding to the mislabeled data input in response to identifying a mislabeled data input. The method also includes automatically retraining the model based on the new plurality of data inputs.” Here, Desmond establishes analyzing or identifying outlier data input w from a plurality of inputs used to train a model which is being interpreted as the identifying of a third datapoint that is incompatible with a model Further, see Desmond in paragraph [0030] describing, “Further, outliers can be instances of training data that do not meet any of the predetermined classification types (e.g., an image of a “snake” in the previous example having 5 classification types) or that do meet a predetermined classification type but are so few in number so as to not allow for an effective training with respect to that classification type.” Here, Desmond establishes outliers as incompatible training data, with the training data not meeting a classification type for effective training.
Further, Desmond teaches “generating an updated third datapoint with a first data structure that is compatible with the machine learning model;”
See Desmond in paragraph [0014] describing, “The method also includes analyzing a vector space that includes the one or more clusters to identify at least one vector representation corresponding to an outlier data input and/or a mislabeled data input. The method also includes forming a new plurality of data inputs having associated labels by removing the outlier data input from the plurality of data inputs in response to identifying an outlier data input and relabeling the data input to have an associated label of a classification type of the predominant classification type of the vector representations in the same cluster as the vector representation corresponding to the mislabeled data input in response to identifying a mislabeled data input.” Here, Desmond establishes generating an updated third datapoint with identifying and relabeling of the outlier data input. The predominant classification type of vector representations is being interpreted as the first data structure and labeling to make the input have an association label of a classification type makes it compatible with the model.
Further, Desmond teaches “and retraining the machine learning model using a revised training dataset that comprises the updated third datapoint.”
See Desmond in paragraph [0014] describing, “The method also includes automatically retraining the model based on the new plurality of data inputs.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Desmond by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Desmond’s teachings of improving ground truth quality for models.
One of ordinary skill in the art would be motivated to do so because by integrating Desmond’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection, one of ordinary skill in the art would bring “A non-limiting example computer-implemented method for automatically improving ground truth quality for modeling.” (Desmond, paragraph [0014]), and “the development of more accurate machine learning models while reducing the development time of such models” (Desmond, paragraph [0014]).
Claim 4:
Regarding claim 4, Fathi in view of Desmond teaches the limitations of claim 3.
Fathi does not appear to explicitly teach “The system of Claim 3, wherein determining that the third datapoint is incompatible with the machine learning model comprises: determining that the machine learning model is configured to accept the first data structure; determining that the third datapoint is associated with a second data structure; and determining that the second data structure does not correspond with the first data structure.”,
Further, Desmond teaches “The system of Claim 3, wherein determining that the third datapoint is incompatible with the machine learning model comprises: determining that the machine learning model is configured to accept the first data structure;”
See Desmond in paragraph [0013] describing, “determining a distribution of vector representations of the first classification type and of the second classification type among the one or more clusters, and responsive to determining that the distribution does not meet a predetermined level of homogeneity, determining that the plurality of data inputs has an ambiguous class structure in relation to the first classification type and the second classification type. The notification provides an indication of the ambiguous class structure. Advantages can also include the development of more accurate ground truth data classification for use in the automatic retraining of a machine learning model.”
Further, Desmond teaches “determining that the third datapoint is associated with a second data structure;”
See Desmond in paragraph [0056] describing, “According to some embodiments, the ground truth analysis engine 112 can identify one or more data inputs that are outliers by identifying vector representations within a vector space that were not clustered. For example, as shown in FIG. 5A, a vector representation having a third type of classification 506 is positioned on its own outside of the first cluster 510 and the second cluster 520, indicating that it is not similar to any other vector representation in the vector space.” Here, Desmond establishes an outlier, which was established to be seen as the third datapoint in previous limitations, being associated with a third type of classification separate from the first and second clusters, which can be seen as the second or another data structure here. The outlier here is the vector representation that is positioned on its own outside of the two clusters and this vector representation has a third classification type which makes it associated with it making the outlier associated with a second data structure. Further, see Desmond in paragraph [0054] describing, “For example, as shown in FIG. 5A, the first cluster 510 includes four instances of vector representations having a first type of classification 502 (e.g., “dog”) and one instance of a vector representation having a second type of classification 504 (e.g., “cat”), whereas the second cluster 520 includes only vector representations having the second type of classification 504.” Here, Desmond establishes the first and second clusters having their own classification types, classification types here have been interpreted as the data structures.
Further, Desmond teaches “and determining that the second data structure does not correspond with the first data structure.”
See Desmond in paragraph [0056] describing, “According to some embodiments, the ground truth analysis engine 112 can identify one or more data inputs that are outliers by identifying vector representations within a vector space that were not clustered. For example, as shown in FIG. 5A, a vector representation having a third type of classification 506 is positioned on its own outside of the first cluster 510 and the second cluster 520, indicating that it is not similar to any other vector representation in the vector space.” Here, Desmond establishes a third type of classification sperate from the first and second clusters, which can be seen as the second data structure here and it does not correspond with either cluster.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Desmond by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Desmond’s teachings of improving ground truth quality for models.
One of ordinary skill in the art would be motivated to do so because by integrating Desmond’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection, one of ordinary skill in the art would bring “A non-limiting example computer-implemented method for automatically improving ground truth quality for modeling.” (Desmond, paragraph [0014]), and “the development of more accurate machine learning models while reducing the development time of such models” (Desmond, paragraph [0014]).
Claim 5:
Regarding claim 5, Fathi teaches the limitations of claim 1.
Fathi does not appear to explicitly teach “The system of Claim 1, wherein the one or more corrective actions further comprise: determining that a fifth label associated with a fifth datapoint, within the training dataset, is incompatible with the machine learning model; generating an updated fifth label with a third data structure that is compatible with the machine learning model; and retraining the machine learning model using a revised training dataset that comprises the fifth datapoint associated with the updated fifth label.”,
However in the same field of art, Desmond teaches “The system of Claim 1, wherein the one or more corrective actions further comprise: determining that a fifth label associated with a fifth datapoint, within the training dataset, is incompatible with the machine learning model;”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can then analyze the clustered/partitioned vector representations to identify anomalous data inputs that have a negative impact on the accuracy of the specified model, such as data inputs that are mislabeled, that contribute to an ambiguous class structure or that are an outlier.” Here, Desmond establishes a mislabeled input which can be seen as the fifth label associated with a fifth datapoint as the input is from a plurality these inputs can be datapoints and a fifth input could be the mislabeled one and mislabel can be the fifth label of a data structure.
Further, Desmond teaches “generating an updated fifth label with a third data structure that is compatible with the machine learning model;”
See Desmond in paragraph [0056] describing, “According to some embodiments, the ground truth analysis engine 112 can identify one or more data inputs that are outliers by identifying vector representations within a vector space that were not clustered. For example, as shown in FIG. 5A, a vector representation having a third type of classification 506 is positioned on its own outside of the first cluster 510 and the second cluster 520, indicating that it is not similar to any other vector representation in the vector space. It can be desirable to examine outliers such as this to determine if they should be discarded for the purposes of training the model. In some embodiments, the processing system 100 can automatically remove outlier(s) from the ground truth data and retrain the model without them. In some embodiments, a notification provided to the user 121 by the processing system 100 can invite the user to examine the outlier(s) make a determination of whether to remove them from the training data, relabel the data input, create a new classification to account for the outlier or some other suitable action. According to some embodiments, the processing system 100 can generate recommended actions (e.g., remove an outlier from training data, modify the label of a mislabeled data input, etc.) and present the recommendations to the user 121 through a notification in a manner that can allow the user 121 to quickly review the relevant data inputs and accept, reject or modify the recommended change.” Here, Desmond establishes a third data structure or type of classification in association with an outlier which in previous limitations is seen as the mislabeled input established to be a fifth label and shows that the label can be modified to be compatible.
Further, Desmond teaches “and retraining the machine learning model using a revised training dataset that comprises the fifth datapoint associated with the updated fifth label.”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can generate notifications to a user 121 of user device 120 to inform the user 121 of any such anomalous data inputs and the user 121 can (e.g., via an interface of user device 120) then modify the ground truth data by deleting one or more selected data inputs from use in training the model, modifying the labels (e.g. relabel, add a label, delete a label or split a label into more than one label) of one or more selected data inputs following the user's inspection of the data point and its original label, or redefine class structures used with the ground truth data. According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.” Here, Desmond establishes anomalous data inputs which were established to include a mislabeled input seen as the fifth datapoint associated with a fifth label being relabeled to revise training data and then retraining a model.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Desmond by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Desmond’s teachings of improving ground truth quality for models.
One of ordinary skill in the art would be motivated to do so because by integrating Desmond’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection, one of ordinary skill in the art would bring “A non-limiting example computer-implemented method for automatically improving ground truth quality for modeling.” (Desmond, paragraph [0014]), and “the development of more accurate machine learning models while reducing the development time of such models” (Desmond, paragraph [0014]).
Claim 6:
Regarding claim 6, Fathi teaches the limitations of claim 1.
Fathi does not appear to explicitly teach “The system of Claim 1, wherein the one or more corrective actions further comprise: accessing a set of expected datapoints that is expected to be present in the training dataset; comparing each of the set of datapoints with a counterpart expected datapoint from among the set of expected datapoints; determining that a fourth datapoint is missing from the training dataset based at least in part upon determining that the fourth datapoint is found in the set of expected datapoints; adding the fourth datapoint to the training dataset; and retraining the machine learning model using a revised training dataset that comprises the fourth datapoint.”,
However in the same field of art, Desmond teaches “The system of Claim 1, wherein the one or more corrective actions further comprise: accessing a set of expected datapoints that is expected to be present in the training dataset;”
See Desmond in paragraph [0003] describing, “A non-limiting example computer-implemented method includes receiving a plurality of data inputs. Each of the plurality of data inputs has an associated label.” Here, Desmond establishes set of datapoints with the data inputs being received or accessed. Further, see Desmond in paragraph [0054] describing, “Data inputs that have the same classification type are expected to be clustered together, as they are expected to have similar aspects that would tend to have n-dimensional vectors that are in the same area of the vector space.” Here, Desmond establishes set of datapoints with the same classification type that are expected.
Further, Desmond teaches “comparing each of the set of datapoints with a counterpart expected datapoint from among the set of expected datapoints;”
See Desmond in paragraph [0031] describing, “one or more embodiments of the invention address the above-described shortcomings of the prior art by providing techniques for improving ground truth quality for generating accurate machine learning models that involve forming vector representations of each example of training set data for a particular model, clustering the vector representations, and analyzing the clusters within the vector space to identify anomalies to be corrected.” Here, Desmond establishes clustering of each of the vector representations which were established to be associated with data inputs or data point within a vector space for anomalies, which is being interpreted as the comparing of each of the datapoints with a counterpart.
Further, Desmond teaches “determining that a fourth datapoint is missing from the training dataset based at least in part upon determining that the fourth datapoint is found in the set of expected datapoints;”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can then analyze the clustered/partitioned vector representations to identify anomalous data inputs that have a negative impact on the accuracy of the specified model, such as data inputs that are mislabeled, that contribute to an ambiguous class structure or that are an outlier. The ground truth analysis engine 112 can generate notifications to a user 121 of user device 120 to inform the user 121 of any such anomalous data inputs and the user 121 can (e.g., via an interface of user device 120) then modify the ground truth data by deleting one or more selected data inputs from use in training the model, modifying the labels (e.g. relabel, add a label, delete a label or split a label into more than one label) of one or more selected data inputs following the user's inspection of the data point and its original label, or redefine class structures used with the ground truth data. According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.” Here, Desmond establishes determining of a missing datapoint from training dataset that is found in the set of expected datapoints with the outlier/mislabeled datapoint which because a plurality of data inputs or datapoints is being observed, a fourth data input could be the anomalous data input that is determined as missing or mislabeled.
Further, Desmond teaches “adding the fourth datapoint to the training dataset;”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can then analyze the clustered/partitioned vector representations to identify anomalous data inputs that have a negative impact on the accuracy of the specified model, such as data inputs that are mislabeled, that contribute to an ambiguous class structure or that are an outlier. The ground truth analysis engine 112 can generate notifications to a user 121 of user device 120 to inform the user 121 of any such anomalous data inputs and the user 121 can (e.g., via an interface of user device 120) then modify the ground truth data by deleting one or more selected data inputs from use in training the model, modifying the labels (e.g. relabel, add a label, delete a label or split a label into more than one label) of one or more selected data inputs following the user's inspection of the data point and its original label, or redefine class structures used with the ground truth data. According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.” Here, Desmond establishes adding of a data input which was mislabeled which is being interpreted as the fourth datapoint as established to training data.
Further, Desmond teaches “and retraining the machine learning model using a revised training dataset that comprises the fourth datapoint.”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can generate notifications to a user 121 of user device 120 to inform the user 121 of any such anomalous data inputs and the user 121 can (e.g., via an interface of user device 120) then modify the ground truth data by deleting one or more selected data inputs from use in training the model, modifying the labels (e.g. relabel, add a label, delete a label or split a label into more than one label) of one or more selected data inputs following the user's inspection of the data point and its original label, or redefine class structures used with the ground truth data. According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.” Here, Desmond establishes retraining model of a revised training data that comprises an added or modified data input which can be seen as a fourth datapoint as established.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Desmond by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Desmond’s teachings of improving ground truth quality for models.
One of ordinary skill in the art would be motivated to do so because by integrating Desmond’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection, one of ordinary skill in the art would bring “A non-limiting example computer-implemented method for automatically improving ground truth quality for modeling.” (Desmond, paragraph [0014]), and “the development of more accurate machine learning models while reducing the development time of such models” (Desmond, paragraph [0014]).
Claim 7:
Regarding claim 7, Fathi teaches the limitations of claim 1.
Fathi does not appear to explicitly teach “The system of Claim 1, wherein the one or more corrective actions further comprise: identifying a first datapoint, within the training dataset, that is missing a first label; adding the first label to the first datapoint; and retraining the machine learning model with a revised training dataset that comprises the first datapoint associated with the first label.”,
However in the same field of art, Desmond teaches “The system of Claim 1, wherein the one or more corrective actions further comprise: identifying a first datapoint, within the training dataset, that is missing a first label;”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can generate notifications to a user 121 of user device 120 to inform the user 121 of any such anomalous data inputs and the user 121 can (e.g., via an interface of user device 120) then modify the ground truth data by deleting one or more selected data inputs from use in training the model, modifying the labels (e.g. relabel, add a label, delete a label or split a label into more than one label) of one or more selected data inputs following the user's inspection of the data point and its original label, or redefine class structures used with the ground truth data. According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.”
Further, Desmond teaches “adding the first label to the first datapoint;”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can generate notifications to a user 121 of user device 120 to inform the user 121 of any such anomalous data inputs and the user 121 can (e.g., via an interface of user device 120) then modify the ground truth data by deleting one or more selected data inputs from use in training the model, modifying the labels (e.g. relabel, add a label, delete a label or split a label into more than one label) of one or more selected data inputs following the user's inspection of the data point and its original label, or redefine class structures used with the ground truth data. According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.”
Further, Desmond teaches “and retraining the machine learning model with a revised training dataset that comprises the first datapoint associated with the first label.”
See Desmond in paragraph [0035] describing, “According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Desmond by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Desmond’s teachings of improving ground truth quality for models.
One of ordinary skill in the art would be motivated to do so because by integrating Desmond’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection, one of ordinary skill in the art would bring “A non-limiting example computer-implemented method for automatically improving ground truth quality for modeling.” (Desmond, paragraph [0014]), and “the development of more accurate machine learning models while reducing the development time of such models” (Desmond, paragraph [0014]).
Claim 10:
Regarding claim 10, Fathi teaches the limitations of claim 8.
Fathi does not appear to explicitly teach “The method of Claim 8, wherein the one or more corrective actions further comprise: identifying a third datapoint, within the training dataset, that is incompatible with the machine learning model; generating an updated third datapoint with a first data structure that is compatible with the machine learning model; and retraining the machine learning model using a revised training dataset that comprises the updated third datapoint.”,
However in the same field of art, Desmond teaches “The method of Claim 8, wherein the one or more corrective actions further comprise: identifying a third datapoint, within the training dataset, that is incompatible with the machine learning model;”
See Desmond in paragraph [0014] describing, “The method also includes training a model based on the plurality of data inputs. The method also includes generating a plurality of vector representations based on the model. Each of the plurality of vector representations corresponds to a unique one of the plurality of data inputs. The method also includes clustering the plurality of vector representations into one or more clusters using a density clustering algorithm. The method also includes analyzing a vector space that includes the one or more clusters to identify at least one vector representation corresponding to an outlier data input and/or a mislabeled data input. The method also includes forming a new plurality of data inputs having associated labels by removing the outlier data input from the plurality of data inputs in response to identifying an outlier data input and relabeling the data input to have an associated label of a classification type of the predominant classification type of the vector representations in the same cluster as the vector representation corresponding to the mislabeled data input in response to identifying a mislabeled data input. The method also includes automatically retraining the model based on the new plurality of data inputs.” Here, Desmond establishes analyzing or identifying outlier data input w from a plurality of inputs used to train a model which is being interpreted as the identifying of a third datapoint that is incompatible with a model Further, see Desmond in paragraph [0030] describing, “Further, outliers can be instances of training data that do not meet any of the predetermined classification types (e.g., an image of a “snake” in the previous example having 5 classification types) or that do meet a predetermined classification type but are so few in number so as to not allow for an effective training with respect to that classification type.” Here, Desmond establishes outliers as incompatible training data, with the training data not meeting a classification type for effective training.
Further, Desmond teaches “generating an updated third datapoint with a first data structure that is compatible with the machine learning model;”
See Desmond in paragraph [0014] describing, “The method also includes analyzing a vector space that includes the one or more clusters to identify at least one vector representation corresponding to an outlier data input and/or a mislabeled data input. The method also includes forming a new plurality of data inputs having associated labels by removing the outlier data input from the plurality of data inputs in response to identifying an outlier data input and relabeling the data input to have an associated label of a classification type of the predominant classification type of the vector representations in the same cluster as the vector representation corresponding to the mislabeled data input in response to identifying a mislabeled data input.” Here, Desmond establishes generating an updated third datapoint with identifying and relabeling of the outlier data input. The predominant classification type of vector representations is being interpreted as the first data structure and labeling to make the input have an association label of a classification type makes it compatible with the model.
Further, Desmond teaches “and retraining the machine learning model using a revised training dataset that comprises the updated third datapoint.”
See Desmond in paragraph [0014] describing, “The method also includes automatically retraining the model based on the new plurality of data inputs.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Desmond by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Desmond’s teachings of improving ground truth quality for models.
One of ordinary skill in the art would be motivated to do so because by integrating Desmond’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection, one of ordinary skill in the art would bring “A non-limiting example computer-implemented method for automatically improving ground truth quality for modeling.” (Desmond, paragraph [0014]), and “the development of more accurate machine learning models while reducing the development time of such models” (Desmond, paragraph [0014]).
Claim 11:
Regarding claim 11, Fathi in view of Desmond teaches the limitations of claim 10.
Fathi does not appear to explicitly teach “The method of Claim 10, wherein determining that the third datapoint is incompatible with the machine learning model comprises: determining that the machine learning model is configured to accept the first data structure; determining that the third datapoint is associated with a second data structure; and determining that the second data structure does not correspond with the first data structure.”,
Further, Desmond teaches “The method of Claim 10, wherein determining that the third datapoint is incompatible with the machine learning model comprises: determining that the machine learning model is configured to accept the first data structure;”
See Desmond in paragraph [0013] describing, “determining a distribution of vector representations of the first classification type and of the second classification type among the one or more clusters, and responsive to determining that the distribution does not meet a predetermined level of homogeneity, determining that the plurality of data inputs has an ambiguous class structure in relation to the first classification type and the second classification type. The notification provides an indication of the ambiguous class structure. Advantages can also include the development of more accurate ground truth data classification for use in the automatic retraining of a machine learning model.”
Further, Desmond teaches “determining that the third datapoint is associated with a second data structure;”
See Desmond in paragraph [0056] describing, “According to some embodiments, the ground truth analysis engine 112 can identify one or more data inputs that are outliers by identifying vector representations within a vector space that were not clustered. For example, as shown in FIG. 5A, a vector representation having a third type of classification 506 is positioned on its own outside of the first cluster 510 and the second cluster 520, indicating that it is not similar to any other vector representation in the vector space.” Here, Desmond establishes an outlier, which was established to be seen as the third datapoint in previous limitations, being associated with a third type of classification separate from the first and second clusters, which can be seen as the second or another data structure here. The outlier here is the vector representation that is positioned on its own outside of the two clusters and this vector representation has a third classification type which makes it associated with it making the outlier associated with a second data structure. Further, see Desmond in paragraph [0054] describing, “For example, as shown in FIG. 5A, the first cluster 510 includes four instances of vector representations having a first type of classification 502 (e.g., “dog”) and one instance of a vector representation having a second type of classification 504 (e.g., “cat”), whereas the second cluster 520 includes only vector representations having the second type of classification 504.” Here, Desmond establishes the first and second clusters having their own classification types, classification types here have been interpreted as the data structures.
Further, Desmond teaches “and determining that the second data structure does not correspond with the first data structure.”
See Desmond in paragraph [0056] describing, “According to some embodiments, the ground truth analysis engine 112 can identify one or more data inputs that are outliers by identifying vector representations within a vector space that were not clustered. For example, as shown in FIG. 5A, a vector representation having a third type of classification 506 is positioned on its own outside of the first cluster 510 and the second cluster 520, indicating that it is not similar to any other vector representation in the vector space.” Here, Desmond establishes a third type of classification sperate from the first and second clusters, which can be seen as the second data structure here and it does not correspond with either cluster.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Desmond by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Desmond’s teachings of improving ground truth quality for models.
One of ordinary skill in the art would be motivated to do so because by integrating Desmond’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection, one of ordinary skill in the art would bring “A non-limiting example computer-implemented method for automatically improving ground truth quality for modeling.” (Desmond, paragraph [0014]), and “the development of more accurate machine learning models while reducing the development time of such models” (Desmond, paragraph [0014]).
Claim 12:
Regarding claim 12, Fathi teaches the limitations of claim 8.
Fathi does not appear to explicitly teach “The method of Claim 8, wherein the one or more corrective actions further comprise: determining that a fifth label associated with a fifth datapoint, within the training dataset, is incompatible with the machine learning model; generating an updated fifth label with a third data structure that is compatible with the machine learning model; and retraining the machine learning model using a revised training dataset that comprises the fifth datapoint associated with the updated fifth label.”,
However in the same field of art, Desmond teaches “The method of Claim 8, wherein the one or more corrective actions further comprise: determining that a fifth label associated with a fifth datapoint, within the training dataset, is incompatible with the machine learning model;”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can then analyze the clustered/partitioned vector representations to identify anomalous data inputs that have a negative impact on the accuracy of the specified model, such as data inputs that are mislabeled, that contribute to an ambiguous class structure or that are an outlier.” Here, Desmond establishes a mislabeled input which can be seen as the fifth label associated with a fifth datapoint as the input is from a plurality these inputs can be datapoints and a fifth input could be the mislabeled one and mislabel can be the fifth label of a data structure.
Further, Desmond teaches “generating an updated fifth label with a third data structure that is compatible with the machine learning model;”
See Desmond in paragraph [0056] describing, “According to some embodiments, the ground truth analysis engine 112 can identify one or more data inputs that are outliers by identifying vector representations within a vector space that were not clustered. For example, as shown in FIG. 5A, a vector representation having a third type of classification 506 is positioned on its own outside of the first cluster 510 and the second cluster 520, indicating that it is not similar to any other vector representation in the vector space. It can be desirable to examine outliers such as this to determine if they should be discarded for the purposes of training the model. In some embodiments, the processing system 100 can automatically remove outlier(s) from the ground truth data and retrain the model without them. In some embodiments, a notification provided to the user 121 by the processing system 100 can invite the user to examine the outlier(s) make a determination of whether to remove them from the training data, relabel the data input, create a new classification to account for the outlier or some other suitable action. According to some embodiments, the processing system 100 can generate recommended actions (e.g., remove an outlier from training data, modify the label of a mislabeled data input, etc.) and present the recommendations to the user 121 through a notification in a manner that can allow the user 121 to quickly review the relevant data inputs and accept, reject or modify the recommended change.” Here, Desmond establishes a third data structure or type of classification in association with an outlier which in previous limitations is seen as the mislabeled input established to be a fifth label and shows that the label can be modified to be compatible.
Further, Desmond teaches “and retraining the machine learning model using a revised training dataset that comprises the fifth datapoint associated with the updated fifth label.”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can generate notifications to a user 121 of user device 120 to inform the user 121 of any such anomalous data inputs and the user 121 can (e.g., via an interface of user device 120) then modify the ground truth data by deleting one or more selected data inputs from use in training the model, modifying the labels (e.g. relabel, add a label, delete a label or split a label into more than one label) of one or more selected data inputs following the user's inspection of the data point and its original label, or redefine class structures used with the ground truth data. According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.” Here, Desmond establishes anomalous data inputs which were established to include a mislabeled input seen as the fifth datapoint associated with a fifth label being relabeled to revise training data and then retraining a model.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Desmond by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Desmond’s teachings of improving ground truth quality for models.
One of ordinary skill in the art would be motivated to do so because by integrating Desmond’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection, one of ordinary skill in the art would bring “A non-limiting example computer-implemented method for automatically improving ground truth quality for modeling.” (Desmond, paragraph [0014]), and “the development of more accurate machine learning models while reducing the development time of such models” (Desmond, paragraph [0014]).
Claim 13:
Regarding claim 13, Fathi teaches the limitations of claim 8.
Fathi does not appear to explicitly teach “The method of Claim 8, wherein the one or more corrective actions further comprise: accessing a set of expected datapoints that is expected to be present in the training dataset; comparing each of the set of datapoints with a counterpart expected datapoint from among the set of expected datapoints; determining that a fourth datapoint is missing from the training dataset based at least in part upon determining that the fourth datapoint is found in the set of expected datapoints; adding the fourth datapoint to the training dataset; and retraining the machine learning model using a revised training dataset that comprises the fourth datapoint.”,
However in the same field of art, Desmond teaches “The method of Claim 8, wherein the one or more corrective actions further comprise: accessing a set of expected datapoints that is expected to be present in the training dataset;”
See Desmond in paragraph [0003] describing, “A non-limiting example computer-implemented method includes receiving a plurality of data inputs. Each of the plurality of data inputs has an associated label.” Here, Desmond establishes set of datapoints with the data inputs being received or accessed. Further, see Desmond in paragraph [0054] describing, “Data inputs that have the same classification type are expected to be clustered together, as they are expected to have similar aspects that would tend to have n-dimensional vectors that are in the same area of the vector space.” Here, Desmond establishes set of datapoints with the same classification type that are expected.
Further, Desmond teaches “comparing each of the set of datapoints with a counterpart expected datapoint from among the set of expected datapoints;”
See Desmond in paragraph [0031] describing, “one or more embodiments of the invention address the above-described shortcomings of the prior art by providing techniques for improving ground truth quality for generating accurate machine learning models that involve forming vector representations of each example of training set data for a particular model, clustering the vector representations, and analyzing the clusters within the vector space to identify anomalies to be corrected.” Here, Desmond establishes clustering of each of the vector representations which were established to be associated with data inputs or data point within a vector space for anomalies, which is being interpreted as the comparing of each of the datapoints with a counterpart.
Further, Desmond teaches “determining that a fourth datapoint is missing from the training dataset based at least in part upon determining that the fourth datapoint is found in the set of expected datapoints;”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can then analyze the clustered/partitioned vector representations to identify anomalous data inputs that have a negative impact on the accuracy of the specified model, such as data inputs that are mislabeled, that contribute to an ambiguous class structure or that are an outlier. The ground truth analysis engine 112 can generate notifications to a user 121 of user device 120 to inform the user 121 of any such anomalous data inputs and the user 121 can (e.g., via an interface of user device 120) then modify the ground truth data by deleting one or more selected data inputs from use in training the model, modifying the labels (e.g. relabel, add a label, delete a label or split a label into more than one label) of one or more selected data inputs following the user's inspection of the data point and its original label, or redefine class structures used with the ground truth data. According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.” Here, Desmond establishes determining of a missing datapoint from training dataset that is found in the set of expected datapoints with the outlier/mislabeled datapoint which because a plurality of data inputs or datapoints is being observed, a fourth data input could be the anomalous data input that is determined as missing or mislabeled.
Further, Desmond teaches “adding the fourth datapoint to the training dataset;”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can then analyze the clustered/partitioned vector representations to identify anomalous data inputs that have a negative impact on the accuracy of the specified model, such as data inputs that are mislabeled, that contribute to an ambiguous class structure or that are an outlier. The ground truth analysis engine 112 can generate notifications to a user 121 of user device 120 to inform the user 121 of any such anomalous data inputs and the user 121 can (e.g., via an interface of user device 120) then modify the ground truth data by deleting one or more selected data inputs from use in training the model, modifying the labels (e.g. relabel, add a label, delete a label or split a label into more than one label) of one or more selected data inputs following the user's inspection of the data point and its original label, or redefine class structures used with the ground truth data. According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.” Here, Desmond establishes adding of a data input which was mislabeled which is being interpreted as the fourth datapoint as established to training data.
Further, Desmond teaches “and retraining the machine learning model using a revised training dataset that comprises the fourth datapoint.”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can generate notifications to a user 121 of user device 120 to inform the user 121 of any such anomalous data inputs and the user 121 can (e.g., via an interface of user device 120) then modify the ground truth data by deleting one or more selected data inputs from use in training the model, modifying the labels (e.g. relabel, add a label, delete a label or split a label into more than one label) of one or more selected data inputs following the user's inspection of the data point and its original label, or redefine class structures used with the ground truth data. According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.” Here, Desmond establishes retraining model of a revised training data that comprises an added or modified data input which can be seen as a fourth datapoint as established.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Desmond by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Desmond’s teachings of improving ground truth quality for models.
One of ordinary skill in the art would be motivated to do so because by integrating Desmond’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection, one of ordinary skill in the art would bring “A non-limiting example computer-implemented method for automatically improving ground truth quality for modeling.” (Desmond, paragraph [0014]), and “the development of more accurate machine learning models while reducing the development time of such models” (Desmond, paragraph [0014]).
Claim 14:
Regarding claim 14, Fathi teaches the limitations of claim 8.
Fathi does not appear to explicitly teach “The method of Claim 8, wherein the one or more corrective actions further comprise: identifying a first datapoint, within the training dataset, that is missing a first label; adding the first label to the first datapoint; and retraining the machine learning model with a revised training dataset that comprises the first datapoint associated with the first label.”,
However in the same field of art, Desmond teaches “The method of Claim 8, wherein the one or more corrective actions further comprise: identifying a first datapoint, within the training dataset, that is missing a first label;”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can generate notifications to a user 121 of user device 120 to inform the user 121 of any such anomalous data inputs and the user 121 can (e.g., via an interface of user device 120) then modify the ground truth data by deleting one or more selected data inputs from use in training the model, modifying the labels (e.g. relabel, add a label, delete a label or split a label into more than one label) of one or more selected data inputs following the user's inspection of the data point and its original label, or redefine class structures used with the ground truth data. According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.”
Further, Desmond teaches “adding the first label to the first datapoint;”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can generate notifications to a user 121 of user device 120 to inform the user 121 of any such anomalous data inputs and the user 121 can (e.g., via an interface of user device 120) then modify the ground truth data by deleting one or more selected data inputs from use in training the model, modifying the labels (e.g. relabel, add a label, delete a label or split a label into more than one label) of one or more selected data inputs following the user's inspection of the data point and its original label, or redefine class structures used with the ground truth data. According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.”
Further, Desmond teaches “and retraining the machine learning model with a revised training dataset that comprises the first datapoint associated with the first label.”
See Desmond in paragraph [0035] describing, “According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Desmond by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Desmond’s teachings of improving ground truth quality for models.
One of ordinary skill in the art would be motivated to do so because by integrating Desmond’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection, one of ordinary skill in the art would bring “A non-limiting example computer-implemented method for automatically improving ground truth quality for modeling.” (Desmond, paragraph [0014]), and “the development of more accurate machine learning models while reducing the development time of such models” (Desmond, paragraph [0014]).
Claim 17:
Regarding claim 17, Fathi teaches the limitations of claim 19.
Fathi does not appear to explicitly teach “The non-transitory computer-readable medium of Claim 15, wherein the one or more corrective actions further comprise: identifying a third datapoint, within the training dataset, that is incompatible with the machine learning model; generating an updated third datapoint with a first data structure that is compatible with the machine learning model; and retraining the machine learning model using a revised training dataset that comprises the updated third datapoint.”,
However in the same field of art, Desmond teaches “The non-transitory computer-readable medium of Claim 15, wherein the one or more corrective actions further comprise: identifying a third datapoint, within the training dataset, that is incompatible with the machine learning model;”
See Desmond in paragraph [0014] describing, “The method also includes training a model based on the plurality of data inputs. The method also includes generating a plurality of vector representations based on the model. Each of the plurality of vector representations corresponds to a unique one of the plurality of data inputs. The method also includes clustering the plurality of vector representations into one or more clusters using a density clustering algorithm. The method also includes analyzing a vector space that includes the one or more clusters to identify at least one vector representation corresponding to an outlier data input and/or a mislabeled data input. The method also includes forming a new plurality of data inputs having associated labels by removing the outlier data input from the plurality of data inputs in response to identifying an outlier data input and relabeling the data input to have an associated label of a classification type of the predominant classification type of the vector representations in the same cluster as the vector representation corresponding to the mislabeled data input in response to identifying a mislabeled data input. The method also includes automatically retraining the model based on the new plurality of data inputs.” Here, Desmond establishes analyzing or identifying outlier data input w from a plurality of inputs used to train a model which is being interpreted as the identifying of a third datapoint that is incompatible with a model Further, see Desmond in paragraph [0030] describing, “Further, outliers can be instances of training data that do not meet any of the predetermined classification types (e.g., an image of a “snake” in the previous example having 5 classification types) or that do meet a predetermined classification type but are so few in number so as to not allow for an effective training with respect to that classification type.” Here, Desmond establishes outliers as incompatible training data, with the training data not meeting a classification type for effective training.
Further, Desmond teaches “generating an updated third datapoint with a first data structure that is compatible with the machine learning model;”
See Desmond in paragraph [0014] describing, “The method also includes analyzing a vector space that includes the one or more clusters to identify at least one vector representation corresponding to an outlier data input and/or a mislabeled data input. The method also includes forming a new plurality of data inputs having associated labels by removing the outlier data input from the plurality of data inputs in response to identifying an outlier data input and relabeling the data input to have an associated label of a classification type of the predominant classification type of the vector representations in the same cluster as the vector representation corresponding to the mislabeled data input in response to identifying a mislabeled data input.” Here, Desmond establishes generating an updated third datapoint with identifying and relabeling of the outlier data input. The predominant classification type of vector representations is being interpreted as the first data structure and labeling to make the input have an association label of a classification type makes it compatible with the model.
Further, Desmond teaches “and retraining the machine learning model using a revised training dataset that comprises the updated third datapoint.”
See Desmond in paragraph [0014] describing, “The method also includes automatically retraining the model based on the new plurality of data inputs.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Desmond by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Desmond’s teachings of improving ground truth quality for models.
One of ordinary skill in the art would be motivated to do so because by integrating Desmond’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection, one of ordinary skill in the art would bring “A non-limiting example computer-implemented method for automatically improving ground truth quality for modeling.” (Desmond, paragraph [0014]), and “the development of more accurate machine learning models while reducing the development time of such models” (Desmond, paragraph [0014]).
Claim 18:
Regarding claim 18, Fathi in view of Desmond teaches the limitations of claim 17.
Fathi does not appear to explicitly teach “The non-transitory computer-readable medium of Claim 15, wherein determining that the third datapoint is incompatible with the machine learning model comprises: determining that the machine learning model is configured to accept the first data structure; determining that the third datapoint is associated with a second data structure; and determining that the second data structure does not correspond with the first data structure.”,
Further, Desmond teaches “The non-transitory computer-readable medium of Claim 15, wherein determining that the third datapoint is incompatible with the machine learning model comprises: determining that the machine learning model is configured to accept the first data structure;”
See Desmond in paragraph [0013] describing, “determining a distribution of vector representations of the first classification type and of the second classification type among the one or more clusters, and responsive to determining that the distribution does not meet a predetermined level of homogeneity, determining that the plurality of data inputs has an ambiguous class structure in relation to the first classification type and the second classification type. The notification provides an indication of the ambiguous class structure. Advantages can also include the development of more accurate ground truth data classification for use in the automatic retraining of a machine learning model.”
Further, Desmond teaches “determining that the third datapoint is associated with a second data structure;”
See Desmond in paragraph [0056] describing, “According to some embodiments, the ground truth analysis engine 112 can identify one or more data inputs that are outliers by identifying vector representations within a vector space that were not clustered. For example, as shown in FIG. 5A, a vector representation having a third type of classification 506 is positioned on its own outside of the first cluster 510 and the second cluster 520, indicating that it is not similar to any other vector representation in the vector space.” Here, Desmond establishes an outlier, which was established to be seen as the third datapoint in previous limitations, being associated with a third type of classification separate from the first and second clusters, which can be seen as the second or another data structure here. The outlier here is the vector representation that is positioned on its own outside of the two clusters and this vector representation has a third classification type which makes it associated with it making the outlier associated with a second data structure. Further, see Desmond in paragraph [0054] describing, “For example, as shown in FIG. 5A, the first cluster 510 includes four instances of vector representations having a first type of classification 502 (e.g., “dog”) and one instance of a vector representation having a second type of classification 504 (e.g., “cat”), whereas the second cluster 520 includes only vector representations having the second type of classification 504.” Here, Desmond establishes the first and second clusters having their own classification types, classification types here have been interpreted as the data structures.
Further, Desmond teaches “and determining that the second data structure does not correspond with the first data structure.”
See Desmond in paragraph [0056] describing, “According to some embodiments, the ground truth analysis engine 112 can identify one or more data inputs that are outliers by identifying vector representations within a vector space that were not clustered. For example, as shown in FIG. 5A, a vector representation having a third type of classification 506 is positioned on its own outside of the first cluster 510 and the second cluster 520, indicating that it is not similar to any other vector representation in the vector space.” Here, Desmond establishes a third type of classification sperate from the first and second clusters, which can be seen as the second data structure here and it does not correspond with either cluster.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Desmond by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Desmond’s teachings of improving ground truth quality for models.
One of ordinary skill in the art would be motivated to do so because by integrating Desmond’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection, one of ordinary skill in the art would bring “A non-limiting example computer-implemented method for automatically improving ground truth quality for modeling.” (Desmond, paragraph [0014]), and “the development of more accurate machine learning models while reducing the development time of such models” (Desmond, paragraph [0014]).
Claim 19:
Regarding claim 19, Fathi teaches the limitations of claim 15.
Fathi does not appear to explicitly teach “The non-transitory computer-readable medium of Claim 15, wherein the one or more corrective actions further comprise: determining that a fifth label associated with a fifth datapoint, within the training dataset, is incompatible with the machine learning model; generating an updated fifth label with a third data structure that is compatible with the machine learning model; and retraining the machine learning model using a revised training dataset that comprises the fifth datapoint associated with the updated fifth label.”,
However in the same field of art, Desmond teaches “The non-transitory computer-readable medium of Claim 15, wherein the one or more corrective actions further comprise: determining that a fifth label associated with a fifth datapoint, within the training dataset, is incompatible with the machine learning model;”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can then analyze the clustered/partitioned vector representations to identify anomalous data inputs that have a negative impact on the accuracy of the specified model, such as data inputs that are mislabeled, that contribute to an ambiguous class structure or that are an outlier.” Here, Desmond establishes a mislabeled input which can be seen as the fifth label associated with a fifth datapoint as the input is from a plurality these inputs can be datapoints and a fifth input could be the mislabeled one and mislabel can be the fifth label of a data structure.
Further, Desmond teaches “generating an updated fifth label with a third data structure that is compatible with the machine learning model;”
See Desmond in paragraph [0056] describing, “According to some embodiments, the ground truth analysis engine 112 can identify one or more data inputs that are outliers by identifying vector representations within a vector space that were not clustered. For example, as shown in FIG. 5A, a vector representation having a third type of classification 506 is positioned on its own outside of the first cluster 510 and the second cluster 520, indicating that it is not similar to any other vector representation in the vector space. It can be desirable to examine outliers such as this to determine if they should be discarded for the purposes of training the model. In some embodiments, the processing system 100 can automatically remove outlier(s) from the ground truth data and retrain the model without them. In some embodiments, a notification provided to the user 121 by the processing system 100 can invite the user to examine the outlier(s) make a determination of whether to remove them from the training data, relabel the data input, create a new classification to account for the outlier or some other suitable action. According to some embodiments, the processing system 100 can generate recommended actions (e.g., remove an outlier from training data, modify the label of a mislabeled data input, etc.) and present the recommendations to the user 121 through a notification in a manner that can allow the user 121 to quickly review the relevant data inputs and accept, reject or modify the recommended change.” Here, Desmond establishes a third data structure or type of classification in association with an outlier which in previous limitations is seen as the mislabeled input established to be a fifth label and shows that the label can be modified to be compatible.
Further, Desmond teaches “and retraining the machine learning model using a revised training dataset that comprises the fifth datapoint associated with the updated fifth label.”
See Desmond in paragraph [0035] describing, “The ground truth analysis engine 112 can generate notifications to a user 121 of user device 120 to inform the user 121 of any such anomalous data inputs and the user 121 can (e.g., via an interface of user device 120) then modify the ground truth data by deleting one or more selected data inputs from use in training the model, modifying the labels (e.g. relabel, add a label, delete a label or split a label into more than one label) of one or more selected data inputs following the user's inspection of the data point and its original label, or redefine class structures used with the ground truth data. According to some embodiments, one of more of these corrective actions can be performed automatically by the processing system 100 (e.g., removing outliers from data used for training, relabeling an anomalous data point determined to be mislabeled, etc.) to create an improved set of data inputs and automatically retrain the model using the improved set of data inputs.” Here, Desmond establishes anomalous data inputs which were established to include a mislabeled input seen as the fifth datapoint associated with a fifth label being relabeled to revise training data and then retraining a model.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Fathi with the teachings of Desmond by using Fathi’s teachings of detecting bias with artificial intelligence models, and incorporate with Desmond’s teachings of improving ground truth quality for models.
One of ordinary skill in the art would be motivated to do so because by integrating Desmond’s frameworks into the methods of Fathi, which are both in the same field of art of bias detection, one of ordinary skill in the art would bring “A non-limiting example computer-implemented method for automatically improving ground truth quality for modeling.” (Desmond, paragraph [0014]), and “the development of more accurate machine learning models while reducing the development time of such models” (Desmond, paragraph [0014]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HASSAN R SESAY whose telephone number is (571)272-8493. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HASSAN RAMADAN SESAY/Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights