DETAILED ACTION
This action is responsive to the amendment filed on 03/03/2026. Claims 1-20 are pending in the case. Claims 1-3, 5-6, 8-10, 12-13, 15-16, and 18-19 are currently amended. Claims 1, 8, and 15 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/03/2026 has been entered.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, the claim recites the limitation “at least one data subset” in lines 12-13. The claim also recites “a plurality of data subsets” in line 6. It is unclear if applicant is attempting to recite a new claim element or attempting to refer to a previously recited claim element. For examination purposes, this limitation has been interpreted to mean “at least one data subset of the plurality of data subsets” referring to the previously recited claim element.
Claim 2 is rejected as being dependent upon a rejected base claim without curing any of the deficiencies.
Regarding claim 3, the claim recites “at least one data subset different from the plurality of data subsets of the second model” in lines 2-3. The parent claim recites “a plurality of data subsets” in line 6 and “a second model associated with at least one data subset” which is interpreted above as “a second model associated with at least one data subset of the plurality of data subsets”. It is unclear if applicant is attempting to recite a new claim element or attempting to refer to a previously recited claim element. For examination purposes, this limitation has been interpreted to mean “at least one data subset of the plurality of data subsets different from the at least one data subset associated with the second model” referring to the previously recited claim element.
Claims 4-7 are rejected as being dependent on a rejected base claim without curing any of the deficiencies.
Regarding claim 8, the claim recites the limitation “at least one data subset” in lines 16-17. The claim also recites “a plurality of data subsets” in line 8. It is unclear if applicant is attempting to recite a new claim element or attempting to refer to a previously recited claim element. For examination purposes, this limitation has been interpreted to mean “at least one data subset of the plurality of data subsets” referring to the previously recited claim element.
Claim 9 is rejected as being dependent upon a rejected base claim without curing any of the deficiencies.
Regarding claim 10, the claim recites “The computer system of claim 10” on line 1 thus depends on itself. The claim is held as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor regards as the invention. For examination purposes, the claim has been interpreted to be dependent upon claim 9, as it is directed to a computer system and recites “the one or more outputs” on line 3 which requires antecedent basis from claim 9. Thus, for examination purposes, line 1 of claim 10 has been interpreted to be “The computer system of claim 9”.
Further, the claim recites “at least one data subset different from the data subsets of the second model” in lines 3-4. The parent claim recites “a plurality of data subsets” in line 8 and “a second model associated with at least one data subset” which is interpreted above as “a second model associated with at least one data subset of the plurality of data subsets”. It is unclear if applicant is attempting to recite a new claim element or attempting to refer to a previously recited claim element. For examination purposes, this limitation has been interpreted to mean “at least one data subset of the plurality of data subsets different from the at least one data subset associated with the second model” referring to the previously recited claim element.
Under the interpretation of claim 10 being dependent on claim 9, claims 11-14 are rejected as being dependent upon a rejected base claim without curing any of the deficiencies.
Regarding claim 15, the claim recites the limitation “at least one data subset” in lines 20-21. The claim also recites “a plurality of data subsets” in line 12. It is unclear if applicant is attempting to recite a new claim element or attempting to refer to a previously recited claim element. For examination purposes, this limitation has been interpreted to mean “at least one data subset of the plurality of data subsets” referring to the previously recited claim element.
Regarding claim 16, the claim recites “at least one data subset different from the data subsets of the second model” in lines 3-4. The parent claim recites “a plurality of data subsets” in line 12 and “a second model associated with at least one data subset” which is interpreted above as “a second model associated with at least one data subset of the plurality of data subsets”. It is unclear if applicant is attempting to recite a new claim element or attempting to refer to a previously recited claim element. For examination purposes, this limitation has been interpreted to mean “at least one data subset of the plurality of data subsets different from the at least one data subset associated with the second model” referring to the previously recited claim element.
Claims 17-20 are rejected as being dependent upon a rejected base claim without curing any of the deficiencies.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1:
Step 1 Statutory Category: Claim 1 is directed to a method, which falls under one of the four statutory categories.
Step 2A Prong 1 Judicial Exception: Claim 1 recites, in part, “generating … a first model associated with a dataset”. This limitation, under the broadest reasonable interpretation, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites, “determining…a first model performance level associated with the first model based on a plurality of dataset metric values associated with the dataset”. This limitation, under the broadest reasonable interpretation, covers the recitation of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”, see MPEP § 2106.04(a)(2)(I)(C). Further, the claim recites, “identifying … a plurality of data subsets of the dataset based on the first model performance level failing to exceed a performance threshold”. This limitation is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case observation. See MPEP § 2106.04(a)(2)(III). Further, the claim recites, “calculating…a plurality of subset metric values associated with the plurality of data subsets”. This limitation is the abstract idea of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP § 2106.04(a)(2)(I)(C). Further, the claim recites: “determining, …, a separability of a plurality of labels associated with the plurality of data subsets, wherein the performance threshold is based on the separability”. This limitation is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case observation. See MPEP § 2106.04(a)(2)(III). Additionally, according to applicant’s specification paragraph 0044, this limitation covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites, “generating…a second model associated with at least one data subset based on the plurality of subset metric values and the plurality of labels”. This limitation, under the broadest reasonable interpretation, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites, “determining…an optimization association associated with the first model based on a second model performance level associated with the second model exceeding the performance threshold”. This limitation is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case observation. See MPEP § 2106.04(a)(2)(III). Finally, the claim recites: “generating…a third model based on a selection of at least one data subset of the plurality of subsets configured to determine the optimization indicates insufficient performance of the first model”. This limitation, under the broadest reasonable interpretation, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I).
Step 2A Prong 2 Integration into a practical application: This judicial exception is not integrated into a practical application. In particular, the claim recites that the method is “computer-implemented”. This is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Further, the claim recites: “using a computer device” and “by the computing device”. These are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Finally, the claim recites “an artificial intelligence model”. This is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h).
Step 2B Significantly more: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements: “using a computing device”, “by the computing device”, and the method being “computer-implemented” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. Further, “an artificial intelligence model” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 2, the rejection of claim 1 is incorporated, and further, the claim recites: “generating… a plurality of outputs of the first model based on comparing the plurality of subset metric values to a plurality of reference metrics; wherein the plurality of outputs are generated based on the first model performance level failing to exceed the performance threshold”. This limitation recites mathematical concepts in addition to those identified in the rejection of claim 1, thus recites a judicial exception.
Further, the claim recites, “by the computing device”. This is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept.
Regarding claim 3, the rejection of claim 2 is incorporated, and further, the claim recites: “generating …, the third model associated with at least one data subset different from the plurality of data subsets of the second model upon the plurality of outputs failing to exceed the performance threshold”. This limitation recites mathematical concepts in addition to those identified in the rejection of the parent claim.
Further, the claim recites, “by the computing device”. This is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept.
Regarding claim 4, the rejection of claim 2 is incorporated, and further, the claim recites: “performing…data profiling on the dataset, the profiling generating the plurality of subset metric values”. This limitation is a continuation of the “calculating…a plurality of subset metric values associated with the plurality of data subsets” limitation identified as an abstract idea in the parent claim, thus recites a judicial exception.
Further, the claim recites, “by the computing device”. This is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept.
Regarding claim 5, the rejection of claim 4 is incorporated, and further, the claim recites: “applying…a plurality of filters to the dataset resulting in a plurality of close codes associated with the dataset” and “mapping…the plurality of close codes to the plurality of labels associated with a plurality of tickets derived from the dataset”. These limitations are continuations of the “performing…data profiling on the dataset, the profiling generating the plurality of subset metric values” limitation identified as an abstract idea in the parent claim, thus recites a judicial exception.
Further, the claim recites, “by the computing device”. This is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept.
Regarding claim 6, the rejection of claim 4 is incorporated, and further, the claim recites: “continuously monitoring, …, the separability of a plurality of labels; wherein the separability comprises a text proportionality threshold and the second model is configured to process the at least one data subset based on the separability”. This limitation is a continuation of the “performing…data profiling on the dataset, the profiling generating the plurality of subset metric values” limitation identified as an abstract idea in the parent claim, thus recites a judicial exception.
Further, the claim recites, “by the computing device”. This is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept.
Regarding claim 7, the rejection of claim 1 is incorporated, and further, the claim recites: “removing…a plurality of undesired expressions from the dataset”. This limitation recites mental processes in addition to those identified in the rejection of claim 1, thus recites a judicial exception.
Further, the claim recites, “by the computing device”. This is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept.
Regarding claim 8:
Step 1 Statutory Category: Claim 8 is directed to a computer system, which falls under one of the four statutory categories.
Step 2A Prong 1 Judicial Exception: Claim 8 recites, in part, “generate a first model associated with a dataset”. This limitation, under the broadest reasonable interpretation, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites, “determine a first model performance level associated with the first model based on a plurality of dataset metric values associated with the dataset”. This limitation, under the broadest reasonable interpretation, covers the recitation of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”, see MPEP § 2106.04(a)(2)(I)(C). Further, the claim recites, “identify a plurality of data subsets of the dataset based on the first model performance level failing to exceed a performance threshold”. This limitation is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case observation. See MPEP § 2106.04(a)(2)(III). Further, the claim recites, “calculate a plurality of subset metric values associated with the plurality of data subsets”. This limitation is the abstract idea of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP § 2106.04(a)(2)(I)(C). Further, the claim recites: “determine a separability of a plurality of labels associated with the plurality of data subsets, wherein the performance threshold is based on the separability”. This limitation is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case observation. See MPEP § 2106.04(a)(2)(III). Additionally, according to applicant’s specification paragraph 0044, this limitation covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites, “generate a second model associated with at least one data subset based on the plurality of subset metric values and the plurality of labels”. This limitation, under the broadest reasonable interpretation, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites, “determine an optimization association associated with the first model based on a second model performance level associated with the second model exceeding the performance threshold”. This limitation is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case observation. See MPEP § 2106.04(a)(2)(III). Finally, the claim recites: “generate a third model based on a selection of at least one data subset of the plurality of data subsets configured to determine the optimization indicates insufficient performance of the first model”. This limitation, under the broadest reasonable interpretation, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I).
Step 2A Prong 2 Integration into a practical application: This judicial exception is not integrated into a practical application. In particular, the claim recites: “a computer system”, “one or more processors”, “one or more computer-readable memories”, and “program instructions”. These are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f).
Step 2B Significantly more: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements: “a computer system”, “one or more processors”, “one or more computer-readable memories”, and “program instructions” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 9, the rejection of claim 8 is incorporated, and further, claim 9 is substantially similar to claim 2 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 10, under the interpretation of claim 10 being dependent upon claim 9, the rejection of claim 9 is incorporated, and further, claim 10 is substantially similar to claim 3 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 11, the rejection of claim 10 is incorporated, and further, claim 11 is substantially similar to claim 4 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 12, the rejection of claim 11 is incorporated, and further, claim 12 is substantially similar to claim 5 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 13, the rejection of claim 11 is incorporated, and further, claim 13 is substantially similar to claim 6 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 14, the rejection of claim 9 is incorporated, and further, claim 14 is substantially similar to claim 7 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 15:
Step 1 Statutory Category: Claim 15 is directed to a machine, which falls under one of the four statutory categories.
Step 2A Prong 1 Judicial Exception: Claim 15 recites, in part, “generating … a first model associated with a dataset”. This limitation, under the broadest reasonable interpretation, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites, “determining…a first model performance level associated with the first model based on a plurality of dataset metric values associated with the dataset”. This limitation, under the broadest reasonable interpretation, covers the recitation of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”, see MPEP § 2106.04(a)(2)(I)(C). Further, the claim recites, “identifying … a plurality of data subsets of the dataset based on the first model performance level failing to exceed a performance threshold”. This limitation is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case observation. See MPEP § 2106.04(a)(2)(III). Further, the claim recites, “calculating…a plurality of subset metric values associated with the plurality of data subsets”. This limitation is the abstract idea of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP § 2106.04(a)(2)(I)(C). Further, the claim recites: “determining, …, a separability of a plurality of labels associated with the plurality of data subsets, wherein the performance threshold is based on the separability”. This limitation is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case observation. See MPEP § 2106.04(a)(2)(III). Additionally, according to applicant’s specification paragraph 0044, this limitation covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites, “generating…a second model associated with at least one data subset based on the plurality of subset metric values and the plurality of labels”. This limitation, under the broadest reasonable interpretation, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Finally, the claim recites, “determining…an optimization association associated with the first model based on a second model performance level associated with the second model exceeding the performance threshold”. This limitation is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case observation. See MPEP § 2106.04(a)(2)(III). Finally, the claim recites: “generating, …, a third model based on a selection of at least one data subset of the plurality of data subsets configured to determine the optimization indicates insufficient performance of the first model”. This limitation, under the broadest reasonable interpretation, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I).
Step 2A Prong 2 Integration into a practical application: This judicial exception is not integrated into a practical application. In particular, the claim recites: “a computing device”, “by the computing device”, “one or more non-transitory computer-readable storage media”, and “program instructions”. These are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f).
Step 2B Significantly more: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements: “a computing device”, “by the computing device”, “one or more non-transitory computer-readable storage media”, and “program instructions” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 16, the rejection of claim 15 is incorporated, and further, claim 16 is substantially similar to claims 3 and 10 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 17, the rejection of claim 16 is incorporated, and further, claim 17 is substantially similar to claims 4 and 11 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 18, the rejection of claim 17 is incorporated, and further, claim 18 is substantially similar to claims 5 and 12 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 19, the rejection of claim 17 is incorporated, and further, claim 19 is substantially similar to claims 6 and 13 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 20, the rejection of claim 15 is incorporated, and further, claim 20 is substantially similar to claims 7 and 14 respectively, and is rejected in the same manner and reasoning applying.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 8-11, and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Black, International Application Published Under the Patent Cooperation Treaty (PCT) No. WO 0058908, hereinafter referred to as “Black” in view of Baron et al., U.S. Patent Application Publication No. 20210350930, hereinafter referred to as “Baron” in further view of Breckenridge et al., U.S. Patent Application Publication No. 20120191630, hereinafter referred to as “Breckenridge”.
Regarding claim 1, Black teaches A … method … for data selection for use with an artificial intelligence model (Black, Abstract, Line 1, “A method and system for training an artificial neural network ('ANN') (10) are disclosed”; Black, Page 25, Lines 4-7, “A further embodiment of the improved training algorithm of the method of the present invention provides a more efficient method of choosing a representative training dataset”), the method comprising:
generating, …, a first model associated with a dataset (Black, Page 29, Line 32 – Page 30, Line 3, “At step 40 of FIGURE 8, along path A, the method of this invention uses the clustering technique to generate a first-pass training dataset. This first-pass training dataset is used to train the artificial neural network to a preset error goal at step 44” Training the ANN is considered to be “generating … a first model”, the dataset that clustering is performed on is considered to be the “dataset”, because the first model is trained using a training dataset derived from the original dataset, it is considered to be “associated with” the original dataset);
determining, …, a first model performance level associated with the first model based on a plurality of dataset metric values associated with the dataset (Black, Page 30, Lines 3-7, “At step 46, the ANN training method of this invention determines whether the intermediate error goal has been achieved for all the records in the group of data records and not just with respect to the training dataset” The “intermediate error” is considered to be the “performance level”; Black, Page 25, Lines 22-31, “Automated procedures involving data clustering are sometimes utilized (as they are here) . Clustering involves determining which data records are similar to, or at least distant from, one another. The similar ones are grouped together, generating "clusters ." The clusters may be used to generate centroids (averages) or to determine the closest actual data points to the centroids. Either way, each of these points are representative of the other points in the cluster and may be used to generate an initial training dataset” The “centroids” are considered to be “a plurality of dataset metric values” and because the ANN is trained on a training dataset comprised of the “centroids”, the performance level is determined “based on a plurality of dataset metric values associated with the dataset”)
identifying, …, a plurality of data subsets of the dataset based on the first model performance level failing to exceed a performance threshold (Black, Page 30, Lines 7-12, “If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset”; Black, Page 26, Line 20 – Page 27, Line 7, “When an otherwise trained ANN has difficulty with the mapping associated with a particular point, there are two possible explanations. Either there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset, or there are nearby data points that have quite different outputs. These "excuses" may be formulated into two equations for prediction difficulty: [Equation 8]. [Equation 9]. β.sub.1 and β.sub.2 in the above equations represent the particular input vectors from points from the training set that minimize/maximize their respective equations. α represents the input vector from a data point not yet included in the training set. D(x) represents the desired output vector associated with the input vector x. The points that are not yet included in the training dataset are cycled through until one is identified that possesses the greatest sum of the two difficulties” The process of cycling through “the points that are not yet included in the training dataset” is considered to be “identifying … a plurality of data subsets” because each point is considered to be added to the training dataset, which is itself a subset of the original dataset. It is performed “based on the first model failing to exceed a performance threshold” because this process is only performed “if the intermediate error goal has not been achieved”);
calculating, …, a plurality of subset metric values associated with the plurality of data subsets (Black, Page 30, Lines 7-12, “If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset”; Black, Page 26, Line 20 – Page 27, Line 7, “When an otherwise trained ANN has difficulty with the mapping associated with a particular point, there are two possible explanations. Either there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset, or there are nearby data points that have quite different outputs. These "excuses" may be formulated into two equations for prediction difficulty: [Equation 8]. [Equation 9]. β.sub.1 and β.sub.2 in the above equations represent the particular input vectors from points from the training set that minimize/maximize their respective equations. α represents the input vector from a data point not yet included in the training set. D(x) represents the desired output vector associated with the input vector x. The points that are not yet included in the training dataset are cycled through until one is identified that possesses the greatest sum of the two difficulties” The “difficultly criterion of equations (8) and (9)” is considered to be the “plurality of subset metric values”. They are considered to be “associated with the plurality of data subsets” because each subset is identified as the training dataset with the addition of another point from the original dataset, so the difficulty criterion calculated for each point is representative of the subset);
…
generating, …, a second model associated with at least one data subset based on the plurality of subset metric values … (Black, Page 30, Lines 12-15, “At step 48, the record from the group of data records is chosen that has the greatest learning difficulty, i.e., the record (data point) that exhibits the greatest prediction error”; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” The model is retrained with “the now expanded training dataset” which is considered to be “generating…a second model”. The subset is chosen based on “the greatest learning difficulty, i.e., the record (data point) that exhibits the greatest prediction error” which is considered to be the equivalent of the model being “associated with at least one data subset based on the plurality of subset metric values”);
determining, …, an optimization associated with the first model based on a second model performance level associated with the second model exceeding the performance threshold (Black, Page 30, Lines 27-31, “If at step 46 the intermediate error goal has been achieved for all records, the method of this embodiment of this invention proceeds to step 50. At step 50, the method of this embodiment determines whether the current error goal is less than or equal to the desired final error goal”; Black, Page 31, Lines 3-9, “If at step 50 the current error goal is less than or equal to the final error goal, the method of this embodiment of this invention proceeds to step 54, where training is complete. If not, then at step 52 the current intermediate error goal is reduced by a specified amount and the method of this invention returns to step 44 to train to the new current error goal” The “intermediate error” is considered to be the “second model performance level”. Based on the second model performance level exceeding the performance threshold, an optimization is determined; either the second model replaces the first model – which is considered to be “training is complete”, or the “intermediate error goal is reduced” which is considered the equivalent of “the performance level” is reduced, and training continues); and generating, …, a third model based on a selection of at least one data subset of the plurality of data subsets configured to determine the optimization indicates insufficient performance of the first model (Black, Figure 8; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” Once the subset for the second model is chosen, the method returns to step 44 to generate the second model; Black, Page 30, Lines 3-15, “At step 46, the ANN training method of this invention determines whether the intermediate error goal has been achieved for all the records in the group of data records and not just with respect to the training dataset. If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset. At step 48, the record from the group of data records is chosen that has the greatest learning difficulty, i.e., the record (data point) that exhibits the greatest prediction error”; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” If the second model fails to achieve the “intermediate error goal”, “a new data record” is added to the training dataset and the method returns to step 44 to generate the “third model”. The selection of this “new data record” is considered to be the selection of at least one data subset, because when it is added to the subset, a new subset is created. Further, because the “intermediate error goal” was failed, the subset was selected upon determining the optimization indicates “insufficient performance”).
Black does not explicitly teach that the method is computer implemented nor that it uses a computing device nor that each of the steps are performed by the computing device nor determining, …, a separability of a plurality of labels associated with the plurality of data subsets, wherein the performance threshold is based on the separability nor that the second model is generated based on the plurality of labels.
Baron teaches determining, …, a separability of a plurality of labels associated with the plurality of data subsets, wherein the performance threshold is based on the separability (Baron, Paragraph 0018, Lines 25-27, “In some examples, the performance metrics can be based on the area under curve (AUC) of the ROC curve of the model”; Baron, Paragraph 0070, Lines 10-16, “A value of 0.5 for the area under curve (AUC) of ROC, as represented by the dotted line, meaning that the true positive rate and the false positive rate are equal, means the machine learning model cannot discriminate and is undesirable, while a larger AUC exceeding 0.5, represented by the solid line, can indicate a higher of confidence in the prediction”; A person of ordinary skill in the art would recognize that AUC of the ROC is considered to be “a separability of a plurality of labels”);
And that the second model is generated based on the plurality of labels (Baron, Baron, Paragraph 0018, Lines 25-27, “In some examples, the performance metrics can be based on the area under curve (AUC) of the ROC curve of the model”; The AUC of the ROC is based on “the plurality of labels”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method of Black to include determining a separability of a plurality of labels associated with the plurality of data subsets and basing the performance threshold on that separability as taught by Baron. The motivation to do so would have been that AUC of the ROC provides a plot between corresponding true positive rates and the false positive rates and is threshold-independent (Baron, Paragraph 0070).
The proposed combination does not explicitly teach that the method is computer implemented nor that it uses a computing device nor that each of the steps are performed by the computing device.
Breckenridge teaches that the method is computer implemented and that it uses a computing device and that each of the steps are performed by the computing device (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”; Breckenridge, Paragraph 0053, Lines 10-14, “The components of the client computing system 202 and/or the predictive modeling system 206 can be implemented in multiple computers distributed over a network, such as a server farm, in one or more locations, or can be implemented in a single computer device”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method as taught by the proposed combination to include the hardware (a computing device) as taught by Breckenridge. The motivation for doing so would have been that any person of ordinary skill in the art would understand that it is necessary to implement a method that uses an artificial intelligence model on a computer infrastructure. Further, one would have been motivated to make such a combination in order to better verify the results of the method taught by Black.
Regarding claim 2, the rejection of claim 1 is incorporated, and further the proposed combination teaches generating by the computing device (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”; Breckenridge, Paragraph 0053, Lines 10-14, “The components of the client computing system 202 and/or the predictive modeling system 206 can be implemented in multiple computers distributed over a network, such as a server farm, in one or more locations, or can be implemented in a single computer device”), a plurality of outputs of the first model based on comparing the plurality of subset metric values to a plurality of reference metrics; wherein the plurality of outputs are generated based on the first model performance failing to exceed the performance threshold (Black, Page 46, Lines 22-24, “FIGURE 14 is a simplified block diagram showing representative applications for an ANN trained with the training algorithm of the present invention”; Black, Figure 14 shows the ANN generating “one or more outputs” of the ANN model such as 400, Stock Market Predictions; Black, Page 25, Lines 22-31, “Automated procedures involving data clustering are sometimes utilized (as they are here) . Clustering involves determining which data records are similar to, or at least distant from, one another. The similar ones are grouped together, generating "clusters ." The clusters may be used to generate centroids (averages) or to determine the closest actual data points to the centroids. Either way, each of these points are representative of the other points in the cluster and may be used to generate an initial training dataset”; The first model is trained using the “initial training dataset” which is determined by comparing the plurality of subset metric values of the first model to other clusters/centroids which are considered to be the “plurality of reference metrics”; Black, Figure 8, Step 44, The first model is “train[ed] to current error goal” and thus produced outputs while failing to exceed the performance threshold).
Regarding claim 3, the rejection of claim 2 is incorporated, and further, the proposed combination teaches generating by the computing device (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”; Breckenridge, Paragraph 0053, Lines 10-14, “The components of the client computing system 202 and/or the predictive modeling system 206 can be implemented in multiple computers distributed over a network, such as a server farm, in one or more locations, or can be implemented in a single computer device”), the third model associated with at least one data subset different from the plurality of data subsets of the second model upon the plurality of outputs failing to exceed the performance threshold (Black, Figure 8; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” Once the subset for the second model is chosen, the method returns to step 44 to generate the second model; Black, Page 30, Lines 3-15, “At step 46, the ANN training method of this invention determines whether the intermediate error goal has been achieved for all the records in the group of data records and not just with respect to the training dataset. If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset. At step 48, the record from the group of data records is chosen that has the greatest learning difficulty, i.e., the record (data point) that exhibits the greatest prediction error”; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” If the second model fails to achieve the “intermediate error goal” which is considered to be the “second model performance level”, “a new data record” is added to the training dataset and the method returns to step 44 to generate the “third model”. The subset for the third model must be different from the subset of the second model because a new point is added to the training dataset that was not present in the training dataset of the second model).
Regarding claim 4, the rejection of claim 2 is incorporated, and further, the proposed combination teaches wherein calculating a plurality of subset metric values comprises: performing, by the computing device (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”; Breckenridge, Paragraph 0053, Lines 10-14, “The components of the client computing system 202 and/or the predictive modeling system 206 can be implemented in multiple computers distributed over a network, such as a server farm, in one or more locations, or can be implemented in a single computer device”), data profiling on the dataset, the profiling generating the plurality of subset metric values (Black, Page 26, Line 20 – Page 27, Line 7, “When an otherwise trained ANN has difficulty with the mapping associated with a particular point, there are two possible explanations. Either there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset, or there are nearby data points that have quite different outputs. These "excuses" may be formulated into two equations for prediction difficulty: [Equation 8]. [Equation 9]. β.sub.1 and β.sub.2 in the above equations represent the particular input vectors from points from the training set that minimize/maximize their respective equations. α represents the input vector from a data point not yet included in the training set. D(x) represents the desired output vector associated with the input vector x. The points that are not yet included in the training dataset are cycled through until one is identified that possesses the greatest sum of the two difficulties” The two equations determine if “there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset” or “there are nearby data points that have quite different outputs” this process is considered to be “data profiling” and results in a value for each point which are considered to be the “plurality of subset metric values”).
Regarding claim 8, Black teaches A computer system for data selection for models (Black, Abstract, Line 1, “A method and system for training an artificial neural network ('ANN') (10) are disclosed”; Black, Page 25, Lines 4-7, “A further embodiment of the improved training algorithm of the method of the present invention provides a more efficient method of choosing a representative training dataset”)
…generate a first model associated with a dataset (Black, Page 29, Line 32 – Page 30, Line 3, “At step 40 of FIGURE 8, along path A, the method of this invention uses the clustering technique to generate a first-pass training dataset. This first-pass training dataset is used to train the artificial neural network to a preset error goal at step 44” Training the ANN is considered to be “generate a first model”, the dataset that clustering is performed on is considered to be the “dataset”, because the first model is trained using a training dataset derived from the original dataset, it is considered to be “associated with” the original dataset);
…determine a first model performance level associated with the first model based on a plurality of dataset metric values associated with the dataset (Black, Page 30, Lines 3-7, “At step 46, the ANN training method of this invention determines whether the intermediate error goal has been achieved for all the records in the group of data records and not just with respect to the training dataset” The “intermediate error” is considered to be the “performance level”; Black, Page 25, Lines 22-31, “Automated procedures involving data clustering are sometimes utilized (as they are here) . Clustering involves determining which data records are similar to, or at least distant from, one another. The similar ones are grouped together, generating "clusters ." The clusters may be used to generate centroids (averages) or to determine the closest actual data points to the centroids. Either way, each of these points are representative of the other points in the cluster and may be used to generate an initial training dataset” The “centroids” are considered to be “a plurality of dataset metric values” and because the ANN is trained on a training dataset comprised of the “centroids”, the performance level is determined “based on a plurality of dataset metric values associated with the dataset”);
…identify a plurality of data subsets of the dataset based on the first model performance level failing to exceed a performance threshold (Black, Page 30, Lines 7-12, “If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset”; Black, Page 26, Line 20 – Page 27, Line 7, “When an otherwise trained ANN has difficulty with the mapping associated with a particular point, there are two possible explanations. Either there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset, or there are nearby data points that have quite different outputs. These "excuses" may be formulated into two equations for prediction difficulty: [Equation 8]. [Equation 9]. β.sub.1 and β.sub.2 in the above equations represent the particular input vectors from points from the training set that minimize/maximize their respective equations. α represents the input vector from a data point not yet included in the training set. D(x) represents the desired output vector associated with the input vector x. The points that are not yet included in the training dataset are cycled through until one is identified that possesses the greatest sum of the two difficulties” The process of cycling through “the points that are not yet included in the training dataset” is considered to be “identify a plurality of data subsets” because each point is considered to be added to the training dataset, which is itself a subset of the original dataset. It is performed “based on the first model failing to exceed a performance threshold” because this process is only performed “if the intermediate error goal has not been achieved”);
…calculate a plurality of subset metric values associated with the plurality of data subsets (Black, Page 30, Lines 7-12, “If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset”; Black, Page 26, Line 20 – Page 27, Line 7, “When an otherwise trained ANN has difficulty with the mapping associated with a particular point, there are two possible explanations. Either there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset, or there are nearby data points that have quite different outputs. These "excuses" may be formulated into two equations for prediction difficulty: [Equation 8]. [Equation 9]. β.sub.1 and β.sub.2 in the above equations represent the particular input vectors from points from the training set that minimize/maximize their respective equations. α represents the input vector from a data point not yet included in the training set. D(x) represents the desired output vector associated with the input vector x. The points that are not yet included in the training dataset are cycled through until one is identified that possesses the greatest sum of the two difficulties” The “difficultly criterion of equations (8) and (9)” is considered to be the “plurality of subset metric values”. They are considered to be “associated with the plurality of data subsets” because each subset is identified as the training dataset with the addition of another point from the original dataset, so the difficulty criterion calculated for each point is representative of the subset);
…
…generate a second model associated with at least one data subset based on the plurality of subset metric values … (Black, Page 30, Lines 12-15, “At step 48, the record from the group of data records is chosen that has the greatest learning difficulty, i.e., the record (data point) that exhibits the greatest prediction error”; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” The model is retrained with “the now expanded training dataset” which is considered to be “generate a second model”. The subset is chosen based on “the greatest learning difficulty, i.e., the record (data point) that exhibits the greatest prediction error” which is considered to be the equivalent of the model being “associated with at least one data subset based on the plurality of subset metric values”); and
…determine an optimization associated with the first model based on a second model performance level associated with the second model exceeding the performance threshold (Black, Page 30, Lines 27-31, “If at step 46 the intermediate error goal has been achieved for all records, the method of this embodiment of this invention proceeds to step 50. At step 50, the method of this embodiment determines whether the current error goal is less than or equal to the desired final error goal”; Black, Page 31, Lines 3-9, “If at step 50 the current error goal is less than or equal to the final error goal, the method of this embodiment of this invention proceeds to step 54, where training is complete. If not, then at step 52 the current intermediate error goal is reduced by a specified amount and the method of this invention returns to step 44 to train to the new current error goal” The “intermediate error” is considered to be the “second model performance level”. Based on the second model performance level exceeding the performance threshold, an optimization is determined; either the second model replaces the first model – which is considered to be “training is complete”, or the “intermediate error goal is reduced” which is considered the equivalent of “the performance level” is reduced, and training continues);
…generate a third model based on a selection of at least one data subset of the plurality of data subsets configured to determine the optimization indicates insufficient performance of the first model (Black, Figure 8; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” Once the subset for the second model is chosen, the method returns to step 44 to generate the second model; Black, Page 30, Lines 3-15, “At step 46, the ANN training method of this invention determines whether the intermediate error goal has been achieved for all the records in the group of data records and not just with respect to the training dataset. If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset. At step 48, the record from the group of data records is chosen that has the greatest learning difficulty, i.e., the record (data point) that exhibits the greatest prediction error”; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” If the second model fails to achieve the “intermediate error goal”, “a new data record” is added to the training dataset and the method returns to step 44 to generate the “third model”. The selection of this “new data record” is considered to be the selection of at least one data subset, because when it is added to the subset, a new subset is created. Further, because the “intermediate error goal” was failed, the subset was selected upon determining the optimization indicates “insufficient performance”).
Black does not explicitly teach the computer system comprising: one or more processors, one or more computer-readable memories, and program instructions stored on at least one of the one or more computer-readable memories for execution by at least one of the one or more processors to cause the computer system to… nor that each of the steps are performed by program instructions nor determine a separability of a plurality of labels associated with the plurality of data subsets, wherein the performance threshold is based on the separability nor that the second model is generated based on the plurality of labels.
Baron teaches determine a separability of a plurality of labels associated with the plurality of data subsets, wherein the performance threshold is based on the separability (Baron, Paragraph 0018, Lines 25-27, “In some examples, the performance metrics can be based on the area under curve (AUC) of the ROC curve of the model”; Baron, Paragraph 0070, Lines 10-16, “A value of 0.5 for the area under curve (AUC) of ROC, as represented by the dotted line, meaning that the true positive rate and the false positive rate are equal, means the machine learning model cannot discriminate and is undesirable, while a larger AUC exceeding 0.5, represented by the solid line, can indicate a higher of confidence in the prediction”; A person of ordinary skill in the art would recognize that AUC of the ROC is considered to be “a separability of a plurality of labels”);
And that the second model is generated based on the plurality of labels (Baron, Baron, Paragraph 0018, Lines 25-27, “In some examples, the performance metrics can be based on the area under curve (AUC) of the ROC curve of the model”; The AUC of the ROC is based on “the plurality of labels”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method of Black to include determining a separability of a plurality of labels associated with the plurality of data subsets and basing the performance threshold on that separability as taught by Baron. The motivation to do so would have been that AUC of the ROC provides a plot between corresponding true positive rates and the false positive rates and is threshold-independent (Baron, Paragraph 0070).
The proposed combination does not explicitly teach the computer system comprising: one or more processors, one or more computer-readable memories, and program instructions stored on at least one of the one or more computer-readable memories for execution by at least one of the one or more processors to cause the computer system to… nor that each of the steps are performed by program instructions.
Breckenridge teaches that the method is the computer system comprising: one or more processors, one or more computer-readable memories, and program instructions stored on at least one of the one or more computer-readable memories for execution by at least one of the one or more processors to cause the computer system to… and that each of the steps are performed by program instructions (Breckenridge, Paragraph 0097, “Various implementations of the systems and techniques described here may be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations may include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system” The “storage system” is considered to be the “computer-readable memory” Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method as taught by Black to include the hardware (one or more processors, one or more computer-readable memories, and program instructions) as taught by Breckenridge. The motivation for doing so would have been that any person of ordinary skill in the art would understand that it is necessary to implement a method that uses an artificial intelligence model on a computer infrastructure. Further, one would have been motivated to make such a combination in order to better verify the results of the method taught by Black.
Regarding claim 9, the rejection of claim 8 is incorporated, and further the proposed combination teaches generate a plurality of outputs of the first model based on program instructions (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”) to compare the plurality of subset metric values to a plurality of reference metrics; wherein the plurality of outputs are generated based on the first model performance failing to exceed the performance threshold (Black, Page 46, Lines 22-24, “FIGURE 14 is a simplified block diagram showing representative applications for an ANN trained with the training algorithm of the present invention”; Black, Figure 14 shows the ANN generating “one or more outputs” of the ANN model such as 400, Stock Market Predictions; Black, Page 25, Lines 22-31, “Automated procedures involving data clustering are sometimes utilized (as they are here) . Clustering involves determining which data records are similar to, or at least distant from, one another. The similar ones are grouped together, generating "clusters ." The clusters may be used to generate centroids (averages) or to determine the closest actual data points to the centroids. Either way, each of these points are representative of the other points in the cluster and may be used to generate an initial training dataset”; The first model is trained using the “initial training dataset” which is determined by comparing the plurality of subset metric values of the first model to other clusters/centroids which are considered to be the “plurality of reference metrics”; Black, Figure 8, Step 44, The first model is “train[ed] to current error goal” and thus produced outputs while failing to exceed the performance threshold).
Regarding claim 10, under the interpretation of claim 10 being dependent upon claim 9, the rejection of claim 9 is incorporated, and further, the proposed combination teaches generate the third model associated with at least one data subset different from the data subset of the second model upon the plurality of outputs failing to exceed the performance threshold (Black, Figure 8; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” Once the subset for the second model is chosen, the method returns to step 44 to generate the second model; Black, Page 30, Lines 3-15, “At step 46, the ANN training method of this invention determines whether the intermediate error goal has been achieved for all the records in the group of data records and not just with respect to the training dataset. If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset. At step 48, the record from the group of data records is chosen that has the greatest learning difficulty, i.e., the record (data point) that exhibits the greatest prediction error”; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” If the second model fails to achieve the “intermediate error goal” which is considered to be the “second model performance level”, “a new data record” is added to the training dataset and the method returns to step 44 to generate the “third model”. The subset for the third model must be different from the subset of the second model because a new point is added to the training dataset that was not present in the training dataset of the second model).
Regarding claim 11, the rejection of claim 10 is incorporated, and further, the proposed combination teaches wherein program instructions to calculate a plurality of subset metric values further comprises program instructions (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”) to: perform data profiling on the dataset, the profiling generating the plurality of subset metric values (Black, Page 26, Line 20 – Page 27, Line 7, “When an otherwise trained ANN has difficulty with the mapping associated with a particular point, there are two possible explanations. Either there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset, or there are nearby data points that have quite different outputs. These "excuses" may be formulated into two equations for prediction difficulty: [Equation 8]. [Equation 9]. β.sub.1 and β.sub.2 in the above equations represent the particular input vectors from points from the training set that minimize/maximize their respective equations. α represents the input vector from a data point not yet included in the training set. D(x) represents the desired output vector associated with the input vector x. The points that are not yet included in the training dataset are cycled through until one is identified that possesses the greatest sum of the two difficulties” The two equations determine if “there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset” or “there are nearby data points that have quite different outputs” this process is considered to be “data profiling” and results in a value for each point which are considered to be the “plurality of subset metric values”).
Regarding claim 15, Black teaches …for data selection…a method (Black, Abstract, Line 1, “A method and system for training an artificial neural network ('ANN') (10) are disclosed”; Black, Page 25, Lines 4-7, “A further embodiment of the improved training algorithm of the method of the present invention provides a more efficient method of choosing a representative training dataset”)
generating, …, a first model associated with a dataset (Black, Page 29, Line 32 – Page 30, Line 3, “At step 40 of FIGURE 8, along path A, the method of this invention uses the clustering technique to generate a first-pass training dataset. This first-pass training dataset is used to train the artificial neural network to a preset error goal at step 44” Training the ANN is considered to be “generating … a first model”, the dataset that clustering is performed on is considered to be the “dataset”, because the first model is trained using a training dataset derived from the original dataset, it is considered to be “associated with” the original dataset);
determining, …, a first model performance level associated with the first model based on a plurality of dataset metric values associated with the dataset (Black, Page 30, Lines 3-7, “At step 46, the ANN training method of this invention determines whether the intermediate error goal has been achieved for all the records in the group of data records and not just with respect to the training dataset” The “intermediate error” is considered to be the “performance level”; Black, Page 25, Lines 22-31, “Automated procedures involving data clustering are sometimes utilized (as they are here) . Clustering involves determining which data records are similar to, or at least distant from, one another. The similar ones are grouped together, generating "clusters ." The clusters may be used to generate centroids (averages) or to determine the closest actual data points to the centroids. Either way, each of these points are representative of the other points in the cluster and may be used to generate an initial training dataset” The “centroids” are considered to be “a plurality of dataset metric values” and because the ANN is trained on a training dataset comprised of the “centroids”, the performance level is determined “based on a plurality of dataset metric values associated with the dataset”)
identifying, …, a plurality of data subsets of the dataset based on the first model performance level failing to exceed a performance threshold (Black, Page 30, Lines 7-12, “If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset”; Black, Page 26, Line 20 – Page 27, Line 7, “When an otherwise trained ANN has difficulty with the mapping associated with a particular point, there are two possible explanations. Either there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset, or there are nearby data points that have quite different outputs. These "excuses" may be formulated into two equations for prediction difficulty: [Equation 8]. [Equation 9]. β.sub.1 and β.sub.2 in the above equations represent the particular input vectors from points from the training set that minimize/maximize their respective equations. α represents the input vector from a data point not yet included in the training set. D(x) represents the desired output vector associated with the input vector x. The points that are not yet included in the training dataset are cycled through until one is identified that possesses the greatest sum of the two difficulties” The process of cycling through “the points that are not yet included in the training dataset” is considered to be “identifying … a plurality of data subsets” because each point is considered to be added to the training dataset, which is itself a subset of the original dataset. It is performed “based on the first model failing to exceed a performance threshold” because this process is only performed “if the intermediate error goal has not been achieved”);
calculating, …, a plurality of subset metric values associated with the plurality of data subsets (Black, Page 30, Lines 7-12, “If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset”; Black, Page 26, Line 20 – Page 27, Line 7, “When an otherwise trained ANN has difficulty with the mapping associated with a particular point, there are two possible explanations. Either there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset, or there are nearby data points that have quite different outputs. These "excuses" may be formulated into two equations for prediction difficulty: [Equation 8]. [Equation 9]. β.sub.1 and β.sub.2 in the above equations represent the particular input vectors from points from the training set that minimize/maximize their respective equations. α represents the input vector from a data point not yet included in the training set. D(x) represents the desired output vector associated with the input vector x. The points that are not yet included in the training dataset are cycled through until one is identified that possesses the greatest sum of the two difficulties” The “difficultly criterion of equations (8) and (9)” is considered to be the “plurality of subset metric values”. They are considered to be “associated with the plurality of data subsets” because each subset is identified as the training dataset with the addition of another point from the original dataset, so the difficulty criterion calculated for each point is representative of the subset);
…
generating, …, a second model associated with at least one data subset based on the plurality of subset metric values… (Black, Page 30, Lines 12-15, “At step 48, the record from the group of data records is chosen that has the greatest learning difficulty, i.e., the record (data point) that exhibits the greatest prediction error”; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” The model is retrained with “the now expanded training dataset” which is considered to be “generating…a second model”. The subset is chosen based on “the greatest learning difficulty, i.e., the record (data point) that exhibits the greatest prediction error” which is considered to be the equivalent of the model being “associated with at least one data subset based on the plurality of subset metric values”);
determining, …, an optimization associated with the first model based on a second model performance level associated with the second model exceeding the performance threshold (Black, Page 30, Lines 27-31, “If at step 46 the intermediate error goal has been achieved for all records, the method of this embodiment of this invention proceeds to step 50. At step 50, the method of this embodiment determines whether the current error goal is less than or equal to the desired final error goal”; Black, Page 31, Lines 3-9, “If at step 50 the current error goal is less than or equal to the final error goal, the method of this embodiment of this invention proceeds to step 54, where training is complete. If not, then at step 52 the current intermediate error goal is reduced by a specified amount and the method of this invention returns to step 44 to train to the new current error goal” The “intermediate error” is considered to be the “second model performance level”. Based on the second model performance level exceeding the performance threshold, an optimization is determined; either the second model replaces the first model – which is considered to be “training is complete”, or the “intermediate error goal is reduced” which is considered the equivalent of “the performance level” is reduced, and training continues); and
generating, …, a third model based on a selection of at least one data subset of the plurality of data subsets configured to determine the optimization indicates insufficient performance of the first model (Black, Figure 8; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” Once the subset for the second model is chosen, the method returns to step 44 to generate the second model; Black, Page 30, Lines 3-15, “At step 46, the ANN training method of this invention determines whether the intermediate error goal has been achieved for all the records in the group of data records and not just with respect to the training dataset. If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset. At step 48, the record from the group of data records is chosen that has the greatest learning difficulty, i.e., the record (data point) that exhibits the greatest prediction error”; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” If the second model fails to achieve the “intermediate error goal”, “a new data record” is added to the training dataset and the method returns to step 44 to generate the “third model”. The selection of this “new data record” is considered to be the selection of at least one data subset, because when it is added to the subset, a new subset is created. Further, because the “intermediate error goal” was failed, the subset was selected upon determining the optimization indicates “insufficient performance”).
Black does not explicitly teach that the method is A computer program product using a computing device for data selection, comprising: one or more non-transitory computer-readable storage media and program instructions stored on the one or more non-transitory computer-readable storage media, the program instructions, when executed by the computing device, cause the computing device to perform… nor that each of the steps are performed by the computing device nor determining, …, a separability of a plurality of labels associated with the plurality of data subsets, wherein the performance threshold is based on the separability nor that the second model is generated based on the plurality of labels.
Baron teaches determining, …, a separability of a plurality of labels associated with the plurality of data subsets, wherein the performance threshold is based on the separability (Baron, Paragraph 0018, Lines 25-27, “In some examples, the performance metrics can be based on the area under curve (AUC) of the ROC curve of the model”; Baron, Paragraph 0070, Lines 10-16, “A value of 0.5 for the area under curve (AUC) of ROC, as represented by the dotted line, meaning that the true positive rate and the false positive rate are equal, means the machine learning model cannot discriminate and is undesirable, while a larger AUC exceeding 0.5, represented by the solid line, can indicate a higher of confidence in the prediction”; A person of ordinary skill in the art would recognize that AUC of the ROC is considered to be “a separability of a plurality of labels”);
And that the second model is generated based on the plurality of labels (Baron, Baron, Paragraph 0018, Lines 25-27, “In some examples, the performance metrics can be based on the area under curve (AUC) of the ROC curve of the model”; The AUC of the ROC is based on “the plurality of labels”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method of Black to include determining a separability of a plurality of labels associated with the plurality of data subsets and basing the performance threshold on that separability as taught by Baron. The motivation to do so would have been that AUC of the ROC provides a plot between corresponding true positive rates and the false positive rates and is threshold-independent (Baron, Paragraph 0070).
The proposed combination does not explicitly teach that the method is A computer program product using a computing device for data selection, comprising: one or more non-transitory computer-readable storage media and program instructions stored on the one or more non-transitory computer-readable storage media, the program instructions, when executed by the computing device, cause the computing device to perform… nor that each of the steps are performed by the computing device.
Breckenridge teaches that the method is A computer program product using a computing device for data selection, comprising: one or more non-transitory computer-readable storage media and program instructions stored on the one or more non-transitory computer-readable storage media, the program instructions, when executed by the computing device, cause the computing device to perform… and that each of the steps are performed by the computing device (Breckenridge, Paragraph 0097, “Various implementations of the systems and techniques described here may be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations may include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system” The “storage system” is considered to be the “non-transitory computer-readable storage media” Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method as taught by Black to include the hardware (a computer program product, a computing device, one or more non-transitory computer-readable storage media, and program instructions) as taught by Breckenridge. The motivation for doing so would have been that any person of ordinary skill in the art would understand that it is necessary to implement a method that uses an artificial intelligence model on a computer infrastructure. Further, one would have been motivated to make such a combination in order to better verify the results of the method taught by Black.
Regarding claim 16, the rejection of claim 15 is incorporated, and further, the proposed combination teaches generating by the computing device (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”; Breckenridge, Paragraph 0053, Lines 10-14, “The components of the client computing system 202 and/or the predictive modeling system 206 can be implemented in multiple computers distributed over a network, such as a server farm, in one or more locations, or can be implemented in a single computer device”), the third model associated with at least one data subset different from the data subsets of the second model upon the plurality of outputs failing to exceed the performance threshold (Black, Figure 8; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” Once the subset for the second model is chosen, the method returns to step 44 to generate the second model; Black, Page 30, Lines 3-15, “At step 46, the ANN training method of this invention determines whether the intermediate error goal has been achieved for all the records in the group of data records and not just with respect to the training dataset. If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset. At step 48, the record from the group of data records is chosen that has the greatest learning difficulty, i.e., the record (data point) that exhibits the greatest prediction error”; Black, Page 30, Lines 19-21, “The ANN training method of this invention proceeds once again to step 44 to train to the current error goal with the now expanded training dataset” If the second model fails to achieve the “intermediate error goal” which is considered to be the “second model performance level”, “a new data record” is added to the training dataset and the method returns to step 44 to generate the “third model”. The subset for the third model must be different from the subset of the second model because a new point is added to the training dataset that was not present in the training dataset of the second model).
Regarding claim 17, the rejection of claim 16 is incorporated, and further, the proposed combination teaches wherein calculating a plurality of subset metric values comprises: performing, by the computing device (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”; Breckenridge, Paragraph 0053, Lines 10-14, “The components of the client computing system 202 and/or the predictive modeling system 206 can be implemented in multiple computers distributed over a network, such as a server farm, in one or more locations, or can be implemented in a single computer device”), data profiling on the dataset, the profiling generating the plurality of subset metric values (Black, Page 26, Line 20 – Page 27, Line 7, “When an otherwise trained ANN has difficulty with the mapping associated with a particular point, there are two possible explanations. Either there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset, or there are nearby data points that have quite different outputs. These "excuses" may be formulated into two equations for prediction difficulty: [Equation 8]. [Equation 9]. β.sub.1 and β.sub.2 in the above equations represent the particular input vectors from points from the training set that minimize/maximize their respective equations. α represents the input vector from a data point not yet included in the training set. D(x) represents the desired output vector associated with the input vector x. The points that are not yet included in the training dataset are cycled through until one is identified that possesses the greatest sum of the two difficulties” The two equations determine if “there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset” or “there are nearby data points that have quite different outputs” this process is considered to be “data profiling” and results in a value for each point which are considered to be the “plurality of subset metric values”).
Claims 5, 7, 12, 14, and 18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Black in view of Baron in view of Breckenridge in further view of Paramesh et al., Automated IT Service Desk Systems Using Machine Learning Techniques. In: Nagabhushan, P., Guru, D., Shekar, B., Kumar, Y. (eds) Data Analytics and Learning. Lecture Notes in Networks and Systems, vol 43. Springer, Singapore. https://doi.org/10.1007/978-981-13-2514-4_28, hereinafter referred to as “Paramesh”.
Regarding claim 5, the rejection of claim 4 is incorporated, and further the proposed combination teaches that the steps of applying and mapping are performed by the computing device (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”; Breckenridge, Paragraph 0053, Lines 10-14, “The components of the client computing system 202 and/or the predictive modeling system 206 can be implemented in multiple computers distributed over a network, such as a server farm, in one or more locations, or can be implemented in a single computer device”).
The proposed combination thus far does not explicitly teach applying…, a plurality of filters to the dataset resulting in a plurality of close codes associated with the dataset; mapping…, the plurality of close codes to a plurality of labels associated with a plurality of tickets derived from the dataset.
Paramesh teaches applying…, a plurality of filters to the dataset resulting in a plurality of close codes associated with the dataset (Paramesh, Page 339-340, Section 4.2.1, teaches four filters to filter out unwanted text data; Paramesh, Page 336, Section 3.2, Line 4, “each ticket description is represented by a vector”; Paramesh, Page 336, Section 3.2, Lines 5-8, “feature vector construction is done on the tickets using TF-IDF term weighting scheme. After the TF-IDF matrix is constructed using the ticket data, feature selection by Chi-Squared test is done as a part of dimensionality reduction”; Paramesh, Page 341, Section 4.3, Lines 5-12, “each document (ticket description) d ∈ D of the training dataset D is represented as a feature vector x(d) = (xd,1, . . . , xd,m). Each vector element specifies the unique term in the document corpus and is represented using tf-idf weighting scheme. Tf-idf specifies the importance of particular word in the document. Formally, tf-idf is given by tf-idf
(
t
,
d
,
D
)
=
t
f
(
t
,
d
)
×
l
o
g
|
D
|
|
{
d
∈
D
:
t
∈
d
}
|
. Here tf (t, d) is the word count of the term t in the document d, |D| is the total count of documents in the dataset D, and |{d ∈ D : t ∈ d}| is the number of documents containing the term t. Feature vector representation is then followed by compression to lower the dimensionality by using chi-square test” The filters used for pre-processing and the “TF-IDF matrix” and “chi-square test” are considered to be the “plurality of filters” and the “feature vectors” are considered to be the “close codes”);
mapping…, the plurality of close codes to the plurality of labels associated with a plurality of tickets derived from the dataset (Paramesh, Page 335, Fig. 2 Solution diagram for classification of unlabeled tickets, the figure shows the trained model taking input text and outputting ticket labels; Paramesh, Page 336, Section 3.3, Lines 1-2, “Once the data preprocessing and vectorization is done, next step is to build the classifier models to categorize the tickets with unknown labels” The tickets are represented using a feature vector, which is considered to be the “close codes”, and to “categorize” them is considered to be assigning them a label, thus “Categorize the tickets” is considered to be “mapping the plurality of close codes to a plurality of labels”; Paramesh, Page 341, Section 4.4, Lines 1-4, “After the feature vector representation and proper feature selection, our next task is to build the accurate classifier. Our dataset would be split using 80 : 20 percentage split ratio with 80% of tickets used for training the classifier and the remaining instances used for testing the classification accuracy”).
It would have been obvious, before the effective filing date of the invention, to have modified the data selection method of the proposed combination to include using filters on the dataset resulting in close codes and mapping the close codes to labels associated with tickets derived from the dataset as taught by Paramesh. The motivation for doing so would have been the ability to use the data selection method taught by the proposed combination for the application of an automated service desk system with improved productivity, end user experience, and reduced ticket resolution time (Paramesh, Abstract, Lines 8-11, “By mining historical ticket descriptions and label, we have built a classifier model to classify the new tickets. A benefit of building such an automated service desk system includes improved productivity, end user experience and reduced resolution time”).
Regarding claim 7, the rejection of claim 1 is incorporated, and further, the proposed combination teaches that the removing step is performed by the computing device (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”; Breckenridge, Paragraph 0053, Lines 10-14, “The components of the client computing system 202 and/or the predictive modeling system 206 can be implemented in multiple computers distributed over a network, such as a server farm, in one or more locations, or can be implemented in a single computer device”).
The proposed combination thus far does not explicitly teach removing,…, a plurality of undesired expressions from the dataset.
Paramesh teaches removing,…, a plurality of undesired expressions from the dataset (Paramesh, Page 336, Section 3.1, Lines 3-7, “The service desk dataset considered for this research had lot of unclean data such as stop words, functional words, phone numbers, email address, date, time, numbers, special characters, etc. The preprocessing block removes all such unwanted data since they do not contribute in identifying the category of the ticket and only the relevant features are retained”).
It would have been obvious, to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method of the proposed combination to include removing a plurality of undesired expressions from the dataset as taught by Paramesh. The motivation for doing so would have been that removing the undesired expressions ensures the data is suitable for data mining, and the undesired expressions do not contribute in identifying the output (Paramesh, Page 336, Section 3.1, Lines 1-2, “The preprocessing of training data is necessary to handle all noisy and unwanted data and to make sure that the data is suitable for data mining”; Paramesh, Page 336, Section 3.1, Lines 5-7, “The preprocessing block removes all such unwanted data since they do not contribute in identifying the category of the ticket”).
Regarding claim 12, the rejection of claim 11 is incorporated.
The proposed combination thus far does not explicitly teach apply a plurality of filters to the dataset resulting in a plurality of close codes associated with the dataset; map the plurality of close codes to the plurality of labels associated with a plurality of tickets derived from the dataset.
Paramesh teaches apply a plurality of filters to the dataset resulting in a plurality of close codes associated with the dataset (Paramesh, Page 339-340, Section 4.2.1, teaches four filters to filter out unwanted text data; Paramesh, Page 336, Section 3.2, Line 4, “each ticket description is represented by a vector”; Paramesh, Page 336, Section 3.2, Lines 5-8, “feature vector construction is done on the tickets using TF-IDF term weighting scheme. After the TF-IDF matrix is constructed using the ticket data, feature selection by Chi-Squared test is done as a part of dimensionality reduction”; Paramesh, Page 341, Section 4.3, Lines 5-12, “each document (ticket description) d ∈ D of the training dataset D is represented as a feature vector x(d) = (xd,1, . . . , xd,m). Each vector element specifies the unique term in the document corpus and is represented using tf-idf weighting scheme. Tf-idf specifies the importance of particular word in the document. Formally, tf-idf is given by tf-idf
(
t
,
d
,
D
)
=
t
f
(
t
,
d
)
×
l
o
g
|
D
|
|
{
d
∈
D
:
t
∈
d
}
|
. Here tf (t, d) is the word count of the term t in the document d, |D| is the total count of documents in the dataset D, and |{d ∈ D : t ∈ d}| is the number of documents containing the term t. Feature vector representation is then followed by compression to lower the dimensionality by using chi-square test” The filters used for pre-processing and the “TF-IDF matrix” and “chi-square test” are considered to be the “plurality of filters” and the “feature vectors” are considered to be the “close codes”);
map the plurality of close codes to the plurality of labels associated with a plurality of tickets derived from the dataset (Paramesh, Page 335, Fig. 2 Solution diagram for classification of unlabeled tickets, the figure shows the trained model taking input text and outputting ticket labels; Paramesh, Page 336, Section 3.3, Lines 1-2, “Once the data preprocessing and vectorization is done, next step is to build the classifier models to categorize the tickets with unknown labels” The tickets are represented using a feature vector, which is considered to be the “close codes”, and to “categorize” them is considered to be assigning them a label, thus “Categorize the tickets” is considered to be “mapping the plurality of close codes to a plurality of labels”; Paramesh, Page 341, Section 4.4, Lines 1-4, “After the feature vector representation and proper feature selection, our next task is to build the accurate classifier. Our dataset would be split using 80 : 20 percentage split ratio with 80% of tickets used for training the classifier and the remaining instances used for testing the classification accuracy”).
It would have been obvious, before the effective filing date of the invention, to have modified the data selection method of the proposed combination to include using filters on the dataset resulting in close codes and mapping the close codes to labels associated with tickets derived from the dataset as taught by Paramesh. The motivation for doing so would have been the ability to use the data selection method taught by the proposed combination for the application of an automated service desk system with improved productivity, end user experience, and reduced ticket resolution time (Paramesh, Abstract, Lines 8-11, “By mining historical ticket descriptions and label, we have built a classifier model to classify the new tickets. A benefit of building such an automated service desk system includes improved productivity, end user experience and reduced resolution time”).
Regarding claim 14, the rejection of claim 9 is incorporated.
The proposed combination thus far does not explicitly teach remove a plurality of undesired expressions from the dataset.
Paramesh teaches remove a plurality of undesired expressions from the dataset (Paramesh, Page 336, Section 3.1, Lines 3-7, “The service desk dataset considered for this research had lot of unclean data such as stop words, functional words, phone numbers, email address, date, time, numbers, special characters, etc. The preprocessing block removes all such unwanted data since they do not contribute in identifying the category of the ticket and only the relevant features are retained”).
It would have been obvious, to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method of the proposed combination to include removing a plurality of undesired expressions from the dataset as taught by Paramesh. The motivation for doing so would have been that removing the undesired expressions ensures the data is suitable for data mining, and the undesired expressions do not contribute in identifying the output (Paramesh, Page 336, Section 3.1, Lines 1-2, “The preprocessing of training data is necessary to handle all noisy and unwanted data and to make sure that the data is suitable for data mining”; Paramesh, Page 336, Section 3.1, Lines 5-7, “The preprocessing block removes all such unwanted data since they do not contribute in identifying the category of the ticket”).
Regarding claim 18, the rejection of claim 17 is incorporated, and further, the proposed combination teaches that the steps of applying and mapping are performed by the computing device (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”; Breckenridge, Paragraph 0053, Lines 10-14, “The components of the client computing system 202 and/or the predictive modeling system 206 can be implemented in multiple computers distributed over a network, such as a server farm, in one or more locations, or can be implemented in a single computer device”).
The proposed combination thus far does not explicitly teach applying…, a plurality of filters to the dataset resulting in a plurality of close codes associated with the dataset; and mapping…, the plurality of close codes to the plurality of labels associated with a plurality of tickets derived from the dataset.
Paramesh teaches applying…, a plurality of filters to the dataset resulting in a plurality of close codes associated with the dataset (Paramesh, Page 339-340, Section 4.2.1, teaches four filters to filter out unwanted text data; Paramesh, Page 336, Section 3.2, Line 4, “each ticket description is represented by a vector”; Paramesh, Page 336, Section 3.2, Lines 5-8, “feature vector construction is done on the tickets using TF-IDF term weighting scheme. After the TF-IDF matrix is constructed using the ticket data, feature selection by Chi-Squared test is done as a part of dimensionality reduction”; Paramesh, Page 341, Section 4.3, Lines 5-12, “each document (ticket description) d ∈ D of the training dataset D is represented as a feature vector x(d) = (xd,1, . . . , xd,m). Each vector element specifies the unique term in the document corpus and is represented using tf-idf weighting scheme. Tf-idf specifies the importance of particular word in the document. Formally, tf-idf is given by tf-idf
(
t
,
d
,
D
)
=
t
f
(
t
,
d
)
×
l
o
g
|
D
|
|
{
d
∈
D
:
t
∈
d
}
|
. Here tf (t, d) is the word count of the term t in the document d, |D| is the total count of documents in the dataset D, and |{d ∈ D : t ∈ d}| is the number of documents containing the term t. Feature vector representation is then followed by compression to lower the dimensionality by using chi-square test” The filters used for pre-processing and the “TF-IDF matrix” and “chi-square test” are considered to be the “plurality of filters” and the “feature vectors” are considered to be the “close codes”); and
mapping…, the plurality of close codes to a plurality of labels associated with the plurality of tickets derived from the dataset (Paramesh, Page 335, Fig. 2 Solution diagram for classification of unlabeled tickets, the figure shows the trained model taking input text and outputting ticket labels; Paramesh, Page 336, Section 3.3, Lines 1-2, “Once the data preprocessing and vectorization is done, next step is to build the classifier models to categorize the tickets with unknown labels” The tickets are represented using a feature vector, which is considered to be the “close codes”, and to “categorize” them is considered to be assigning them a label, thus “Categorize the tickets” is considered to be “mapping the plurality of close codes to a plurality of labels”; Paramesh, Page 341, Section 4.4, Lines 1-4, “After the feature vector representation and proper feature selection, our next task is to build the accurate classifier. Our dataset would be split using 80 : 20 percentage split ratio with 80% of tickets used for training the classifier and the remaining instances used for testing the classification accuracy”).
It would have been obvious, before the effective filing date of the invention, to have modified the data selection method of the proposed combination to include using filters on the dataset resulting in close codes and mapping the close codes to labels associated with tickets derived from the dataset as taught by Paramesh. The motivation for doing so would have been the ability to use the data selection method taught by the proposed combination for the application of an automated service desk system with improved productivity, end user experience, and reduced ticket resolution time (Paramesh, Abstract, Lines 8-11, “By mining historical ticket descriptions and label, we have built a classifier model to classify the new tickets. A benefit of building such an automated service desk system includes improved productivity, end user experience and reduced resolution time”).
Regarding claim 20, the rejection of claim 15 is incorporated, and further, the proposed combination teaches that the removing step is performed by the computing device (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”; Breckenridge, Paragraph 0053, Lines 10-14, “The components of the client computing system 202 and/or the predictive modeling system 206 can be implemented in multiple computers distributed over a network, such as a server farm, in one or more locations, or can be implemented in a single computer device”).
The proposed combination thus far does not explicitly teach removing,…, a plurality of undesired expressions from the dataset.
Paramesh teaches removing,…, a plurality of undesired expressions from the dataset (Paramesh, Page 336, Section 3.1, Lines 3-7, “The service desk dataset considered for this research had lot of unclean data such as stop words, functional words, phone numbers, email address, date, time, numbers, special characters, etc. The preprocessing block removes all such unwanted data since they do not contribute in identifying the category of the ticket and only the relevant features are retained”).
It would have been obvious, to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method of the proposed combination to include removing a plurality of undesired expressions from the dataset as taught by Paramesh. The motivation for doing so would have been that removing the undesired expressions ensures the data is suitable for data mining, and the undesired expressions do not contribute in identifying the output (Paramesh, Page 336, Section 3.1, Lines 1-2, “The preprocessing of training data is necessary to handle all noisy and unwanted data and to make sure that the data is suitable for data mining”; Paramesh, Page 336, Section 3.1, Lines 5-7, “The preprocessing block removes all such unwanted data since they do not contribute in identifying the category of the ticket”).
Claims 6, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Black in view of Baron in further view of Breckenridge in further view of Beshr, The What, Why, and How of Model Drift, 10/26/2021, https://towardsdatascience.com/the-what-why-and-how-of-model-drift-38c2af0e97ee/, hereinafter referred to as “Beshr” in further view of Revina et al., IT Ticket Classification: The Simpler, the Better, 2020, IEEE Access. 8. 193380-193395. 10.1109/ACCESS.2020.3032840, hereinafter referred to as “Revina”.
Regarding claim 6, the rejection of claim 4 is incorporated, and further, the proposed combination teaches wherein the separability comprises a … threshold (Baron, Paragraph 0018, Lines 25-27, “In some examples, the performance metrics can be based on the area under curve (AUC) of the ROC curve of the model”; Baron, Paragraph 0070, Lines 10-16, “A value of 0.5 for the area under curve (AUC) of ROC, as represented by the dotted line, meaning that the true positive rate and the false positive rate are equal, means the machine learning model cannot discriminate and is undesirable, while a larger AUC exceeding 0.5, represented by the solid line, can indicate a higher of confidence in the prediction”; Baron, Paragraph 0028, Lines 1-8, “In a case where the machine learning models are trained to output a survival rate of a patient, a true positive occurs when the machine learning model outputs that the survival probability of the patient, at a given time, exceeds a probability threshold and the patient actually survives at that given time. A false positive occurs when the survival probability of the patient exceeds the probability threshold, but the patient does not survive at that given time”) and the second model is configured to process the at least one data subset based on the separability (Black, Page 30, Lines 7-12, “If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset”; Black, Page 26, Line 20 – Page 27, Line 7, “When an otherwise trained ANN has difficulty with the mapping associated with a particular point, there are two possible explanations. Either there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset, or there are nearby data points that have quite different outputs. These "excuses" may be formulated into two equations for prediction difficulty: [Equation 8]. [Equation 9]. β.sub.1 and β.sub.2 in the above equations represent the particular input vectors from points from the training set that minimize/maximize their respective equations. α represents the input vector from a data point not yet included in the training set. D(x) represents the desired output vector associated with the input vector x. The points that are not yet included in the training dataset are cycled through until one is identified that possesses the greatest sum of the two difficulties” Because the subsets are chosen based on there being “no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset”, or there being “nearby data points that have quite different outputs” and the “outputs” are considered to be the “labels”, the second model is configured to process the subset “based on the separability” because the selection of the subset is dependent on the labels).
Further, the proposed combination teaches that the step of determining is performed by the computing device (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”; Breckenridge, Paragraph 0053, Lines 10-14, “The components of the client computing system 202 and/or the predictive modeling system 206 can be implemented in multiple computers distributed over a network, such as a server farm, in one or more locations, or can be implemented in a single computer device”).
The proposed combination thus far does not explicitly teach continuously monitoring … the separability of the plurality of labels nor the threshold being a text proportionality threshold.
Beshr teaches continuously monitoring … the separability of the plurality of labels (Beshr, Page 5, Paragraph 2, Lines 2-4, “What we have uncovered so far on model drift highlights the importance of continuously monitoring the performance of models on incoming data streams during production as well”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method of the proposed combination to include continuously monitoring the separability of the plurality of labels as taught by Beshr. The motivation to do so would have been to detect concept drift and allow the ability to fix the concept drift before the performance degrades too far (Beshr, Page 3, Concept Drift; Beshr, Page 5, Detecting Model Drift; Beshr, Pages 5-6, Model Drift Remedies).
The proposed combination does not explicitly teach the threshold being a text proportionality threshold.
Revina teaches the threshold being a text proportionality threshold (Revina, Page 193384, Section 2, Paragraph 2, Lines 1-8, “Each of the words is associated with positive, negative, or neutral sentiment. Words with valence scores greater than 0 are considered positive, whereas words with a valence score less than 0 are considered negative. All other words are considered to have a neutral sentiment. We determine the proportion of words with negative, neutral, and positive sentiment for each IT ticket text and use these values as features”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method of the proposed combination to include the threshold being a text proportionality threshold as taught by Revina. The motivation to do so would have been that the lexicon-based sentiment analysis is known to increase prediction quality (Revina, Page 3, Section 3, Paragraph 2).
Regarding claim 13, the rejection of claim 11 is incorporated, and further, the proposed combination teaches wherein the separability comprises a … threshold (Baron, Paragraph 0018, Lines 25-27, “In some examples, the performance metrics can be based on the area under curve (AUC) of the ROC curve of the model”; Baron, Paragraph 0070, Lines 10-16, “A value of 0.5 for the area under curve (AUC) of ROC, as represented by the dotted line, meaning that the true positive rate and the false positive rate are equal, means the machine learning model cannot discriminate and is undesirable, while a larger AUC exceeding 0.5, represented by the solid line, can indicate a higher of confidence in the prediction”; Baron, Paragraph 0028, Lines 1-8, “In a case where the machine learning models are trained to output a survival rate of a patient, a true positive occurs when the machine learning model outputs that the survival probability of the patient, at a given time, exceeds a probability threshold and the patient actually survives at that given time. A false positive occurs when the survival probability of the patient exceeds the probability threshold, but the patient does not survive at that given time”) and the second model is configured to process the at least one data subset based on the separability (Black, Page 30, Lines 7-12, “If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset”; Black, Page 26, Line 20 – Page 27, Line 7, “When an otherwise trained ANN has difficulty with the mapping associated with a particular point, there are two possible explanations. Either there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset, or there are nearby data points that have quite different outputs. These "excuses" may be formulated into two equations for prediction difficulty: [Equation 8]. [Equation 9]. β.sub.1 and β.sub.2 in the above equations represent the particular input vectors from points from the training set that minimize/maximize their respective equations. α represents the input vector from a data point not yet included in the training set. D(x) represents the desired output vector associated with the input vector x. The points that are not yet included in the training dataset are cycled through until one is identified that possesses the greatest sum of the two difficulties” Because the subsets are chosen based on there being “no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset”, or there being “nearby data points that have quite different outputs” and the “outputs” are considered to be the “labels”, the second model is configured to process the subset “based on the separability” because the selection of the subset is dependent on the labels).
Further, the proposed combination teaches that the step of determining is performed by the computing device (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”; Breckenridge, Paragraph 0053, Lines 10-14, “The components of the client computing system 202 and/or the predictive modeling system 206 can be implemented in multiple computers distributed over a network, such as a server farm, in one or more locations, or can be implemented in a single computer device”).
The proposed combination thus far does not explicitly teach continuously monitoring … the separability of the plurality of labels nor the threshold being a text proportionality threshold.
Beshr teaches continuously monitoring … the separability of the plurality of labels (Beshr, Page 5, Paragraph 2, Lines 2-4, “What we have uncovered so far on model drift highlights the importance of continuously monitoring the performance of models on incoming data streams during production as well”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method of the proposed combination to include continuously monitoring the separability of the plurality of labels as taught by Beshr. The motivation to do so would have been to detect concept drift and allow the ability to fix the concept drift before the performance degrades too far (Beshr, Page 3, Concept Drift; Beshr, Page 5, Detecting Model Drift; Beshr, Pages 5-6, Model Drift Remedies).
The proposed combination does not explicitly teach the threshold being a text proportionality threshold.
Revina teaches the threshold being a text proportionality threshold (Revina, Page 193384, Section 2, Paragraph 2, Lines 1-8, “Each of the words is associated with positive, negative, or neutral sentiment. Words with valence scores greater than 0 are considered positive, whereas words with a valence score less than 0 are considered negative. All other words are considered to have a neutral sentiment. We determine the proportion of words with negative, neutral, and positive sentiment for each IT ticket text and use these values as features”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method of the proposed combination to include the threshold being a text proportionality threshold as taught by Revina. The motivation to do so would have been that the lexicon-based sentiment analysis is known to increase prediction quality (Revina, Page 3, Section 3, Paragraph 2).
Regarding claim 19, the rejection of claim 17 is incorporated, and further, the proposed combination teaches wherein the separability comprises a … threshold (Baron, Paragraph 0018, Lines 25-27, “In some examples, the performance metrics can be based on the area under curve (AUC) of the ROC curve of the model”; Baron, Paragraph 0070, Lines 10-16, “A value of 0.5 for the area under curve (AUC) of ROC, as represented by the dotted line, meaning that the true positive rate and the false positive rate are equal, means the machine learning model cannot discriminate and is undesirable, while a larger AUC exceeding 0.5, represented by the solid line, can indicate a higher of confidence in the prediction”; Baron, Paragraph 0028, Lines 1-8, “In a case where the machine learning models are trained to output a survival rate of a patient, a true positive occurs when the machine learning model outputs that the survival probability of the patient, at a given time, exceeds a probability threshold and the patient actually survives at that given time. A false positive occurs when the survival probability of the patient exceeds the probability threshold, but the patient does not survive at that given time”) and the second model is configured to process the at least one data subset based on the separability (Black, Page 30, Lines 7-12, “If the intermediate error goal has not been achieved for all of the data records, at step 48 a new data record (point) is added according to the difficulty criterion of equations (8) and (9) discussed above for selecting a new data point to add to the training dataset”; Black, Page 26, Line 20 – Page 27, Line 7, “When an otherwise trained ANN has difficulty with the mapping associated with a particular point, there are two possible explanations. Either there are no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset, or there are nearby data points that have quite different outputs. These "excuses" may be formulated into two equations for prediction difficulty: [Equation 8]. [Equation 9]. β.sub.1 and β.sub.2 in the above equations represent the particular input vectors from points from the training set that minimize/maximize their respective equations. α represents the input vector from a data point not yet included in the training set. D(x) represents the desired output vector associated with the input vector x. The points that are not yet included in the training dataset are cycled through until one is identified that possesses the greatest sum of the two difficulties” Because the subsets are chosen based on there being “no nearby (i.e., exhibiting similar inputs) data points with similar outputs included in the training dataset”, or there being “nearby data points that have quite different outputs” and the “outputs” are considered to be the “labels”, the second model is configured to process the subset “based on the separability” because the selection of the subset is dependent on the labels).
Further, the proposed combination teaches that the step of determining is performed by the computing device (Breckenridge, Paragraph 0053, Lines 1-6, “Components of the client computing system 202 and/or the predictive modeling system 206, e.g., the model training module 212, model selection module 210 and trained predictive model 218, can be realized by instructions that upon execution cause one or more computers to carry out the operations described”; Breckenridge, Paragraph 0053, Lines 10-14, “The components of the client computing system 202 and/or the predictive modeling system 206 can be implemented in multiple computers distributed over a network, such as a server farm, in one or more locations, or can be implemented in a single computer device”).
The proposed combination thus far does not explicitly teach continuously monitoring … the separability of the plurality of labels nor the threshold being a text proportionality threshold.
Beshr teaches continuously monitoring … the separability of the plurality of labels (Beshr, Page 5, Paragraph 2, Lines 2-4, “What we have uncovered so far on model drift highlights the importance of continuously monitoring the performance of models on incoming data streams during production as well”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method of the proposed combination to include continuously monitoring the separability of the plurality of labels as taught by Beshr. The motivation to do so would have been to detect concept drift and allow the ability to fix the concept drift before the performance degrades too far (Beshr, Page 3, Concept Drift; Beshr, Page 5, Detecting Model Drift; Beshr, Pages 5-6, Model Drift Remedies).
The proposed combination does not explicitly teach the threshold being a text proportionality threshold.
Revina teaches the threshold being a text proportionality threshold (Revina, Page 193384, Section 2, Paragraph 2, Lines 1-8, “Each of the words is associated with positive, negative, or neutral sentiment. Words with valence scores greater than 0 are considered positive, whereas words with a valence score less than 0 are considered negative. All other words are considered to have a neutral sentiment. We determine the proportion of words with negative, neutral, and positive sentiment for each IT ticket text and use these values as features”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the data selection method of the proposed combination to include the threshold being a text proportionality threshold as taught by Revina. The motivation to do so would have been that the lexicon-based sentiment analysis is known to increase prediction quality (Revina, Page 3, Section 3, Paragraph 2).
Response to Arguments
Applicant’s arguments regarding the 35 U.S.C. 101 rejections of the claims have been fully considered but are unpersuasive.
Applicant first argues, in page 13, final paragraph – page 14, paragraph 1, that “the implementation of data selection and data profiling ascertain differences between classes that indicate whether a given model is performing according to an environment-specific standard utilizing machine learning logic and the iterative training thereof is an apparent improvement”. It is important to note that an improvement in the abstract idea itself (e.g. a recited mathematical concept or mental process) is not an improvement in technology, see MPEP 2106.05(a)(II).
Applicant next argues, on page 14, final paragraph – page 15, paragraph 2 of the response, that “the instantly claimed invention goes far beyond a mathematical function” and that the human mind cannot perform any aspect of the claimed invention. Examiner respectfully disagrees. Applicant specifically notes “ascertain machine learning model performance” cannot be performed in the human mind, however this limitation is not recited in the rejected claims, and applicant has not directed the examiner to any limitations in which they believe reflect this feature. Further, applicant specifically points to “determine separability of labels specific to datasets during the training phase”. The broadest reasonable interpretation of this limitation includes using a mental judgement to determine separability of labels, which is considered to be a mental process. Finally, applicant points to “generate outputs of machine learning models based on comparing data subset metric values to reference metrics”. This limitation was not identified as a mental process in the 35 U.S.C. 101 analysis, but rather a mathematical concept. For a more in depth analysis see the updated 35 U.S.C. 101 rejection above.
Applicant next argues, on page 15, paragraph 3 of the response, that “the limitations preclude the training of both neural networks and machine learning models from being performed in the mind are similar in that both essentially require surrender of control to the machine itself”. Examiner respectfully disagrees. Mere physical or tangible implementation of an exception is not in itself an inventive concept and does not guarantee eligibility, see MPEP 2106.05(I)(A).
Applicant’s arguments regarding the 35 U.S.C. 103 rejections of the claims have been fully considered but are unpersuasive as they are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant's arguments regarding the remainder of the claims rely upon the arguments asserted with respect to the independent claims, and are thus unpersuasive.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Achim Schilling et al., Quantifying the separability of data classes in neural networks, Neural Networks, Volume 139, 2021, Pages 278-293, ISSN 0893-6080, https://doi.org/10.1016/j.neunet.2021.03.035 discloses a method that measures how well different data classes separate in each given layer of a neural network.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOLLY CLARKE SIPPEL whose telephone number is (571)272-3270. The examiner can normally be reached Monday - Friday, 7:30 a.m. - 4:30 p.m. ET..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571)272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.C.S./ Examiner, Art Unit 2122
/KAKALI CHAKI/ Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights