DETAILED ACTION
This is the initial Office action based on the application submitted on March 22, 2024.
Claims 1-20 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation Under 35 USC § 112(f)
The following is a quotation of 35 U.S.C. 112(f):
(f) ELEMENT IN CLAIM FOR A COMBINATION.—An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f):
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) because the claim limitations use a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation is: “a testing and validation platform configured for […]” in Claims 11-18.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f), they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If the Applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f), the Applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Claim Interpretation: Under the broadest reasonable interpretation (BRI), the limitations of Claim 1 are presumed to have their plain meaning consistent with the specification as it would be interpreted by one of ordinary skill in the art. See MPEP § 2111.
Step 1: Claim 1 is directed to a method, which is a process (a series of steps or
acts), and falls within one of the statutory categories of invention.
Step 2A, Prong One: Claim 1 recites the limitations:
analyzing, […], a received dataset by conducting at least one statistical test of the dataset, the dataset comprising data representative of samples each comprising features and a corresponding target;
validating, […], subsets of the dataset, the subsets comprising an expert dataset, a training dataset, and a validation dataset and/or a test dataset, wherein each subset comprises data representative of distinct samples; and
[…], by using the expert dataset to determine at least one metric for each sample of the expert dataset and comparing each metric of the at least one metric to a corresponding defined trust interval.
These recited steps, under the broadest reasonable interpretation (BRI), cover performance of the steps in the human mind alone or with the aid of pen and paper. That is, other than reciting:
providing a testing and validation platform for machine learning lifecycle testing and validation.
at the testing and validation platform.
by the testing and validation platform.
testing, by the testing and validation platform and after the machine learning model is trained, the machine learning model […].
These recited steps, under the broadest reasonable interpretation (BRI), cover performance of the steps in the human mind alone or with the aid of pen and paper. That is, the limitations in (a), (b) and (c) can be reasonably interpreted as mental processes that can be practically performed in the human mind alone using observation, evaluation, judgment, and opinion or with the aid of pen and paper. For example, the limitation (a) for the analyzing step, a human can read dataset information stored in a database using observation, evaluation, judgment, and opinion to analyze the dataset. The limitation (b) for the validating step, a human can read dataset information stored in a database using observation, evaluation, judgment, and opinion to validate the subsets of dataset. The limitation (c) for the determine and comparing step, a human can read dataset information stored in a database using observation, evaluation, judgment, and opinion to determine and compare the metric for the dataset to a to a corresponding defined trust interval. See MPEP § 2106.04(a)(2)(III).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the human mind alone or with the aid of pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A, Prong Two: This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements:
providing a testing and validation platform for machine learning lifecycle testing and validation.
at the testing and validation platform.
by the testing and validation platform.
testing, by the testing and validation platform and after the machine learning model is trained, the machine learning model […].
displaying, […], results of the analysis;
receiving, […], a selected at least one feature of the features for training a machine learning model;
The additional elements (1), (2), (3) and (4) are recited at a high-level of generality such that they amount to no more than mere instructions to apply the judicial exception using generic computer components. The testing and validation platform for machine learning lifecycle testing and validation and the machine learning model are used as tools to perform the analyzing, displaying, receiving, validating, testing, determine and comparing steps of the claim. See MPEP § 2106.05(f).
The additional element (5) and (6) are mere data outputting/gathering recited at a high level of generality, and thus are insignificant extra-solution activities. See MPEP § 2106.05(g). Furthermore, all uses of the recited judicial exception require such data outputting/gathering and, as such, the additional elements do not impose any meaningful limits on the claim. The additional elements amount to necessary data outputting/gathering. See MPEP § 2106.05.
Accordingly, even when viewed in combination, the additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as a combination do not amount to significantly more than the abstract idea. As discussed above with respect to integration of the abstract idea into a practical application, the claim recites the additional elements:
providing a testing and validation platform for machine learning lifecycle testing and validation.
at the testing and validation platform.
by the testing and validation platform.
testing, by the testing and validation platform and after the machine learning model is trained, the machine learning model […].
displaying, […], results of the analysis;
receiving, […], a selected at least one feature of the features for training a machine learning model;
The additional elements (1), (2), (3) and (4) amount to no more than mere instructions to apply the judicial exception using generic computer components. Mere instructions to apply a judicial exception using generic computer components cannot provide an inventive concept.
The additional element (5) and (6) simply append well-understood, routine, and conventional activity previously known to the industry, specified at a high level of generality, to the judicial exception is not indicative of an inventive concept. MPEP § 2106.05(d)(II) expressly states that the courts have recognized the computer functions of receiving or transmitting data over a network, e.g., using the Internet to gather data and presenting offers and gathering statistics as well‐understood, routine, and conventional computer functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as an insignificant extra-solution activity. Thus, a person of ordinary skill in the art would readily comprehend that it is well-understood, routine, and conventional in the computing art to displaying results of the analysis and receiving a selected at least one feature of the features for training a machine learning model. Therefore, the limitation remains insignificant extra-solution activity even upon reconsideration and does not amount to significantly more.
Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the additional elements as a combination adds nothing that is not already present when looking at the additional elements taken individually. Even when considered in combination, the additional elements represent mere instructions to apply a judicial exception using generic computer components, insignificant extra-solution activities, and therefore do not provide an inventive concept. The claim is not patent eligible.
Claims 2-10 are rejected under 35 U.S.C. 101 as directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more for at least the reasons stated above.
Claim 2 recites the limitation:
(a) if the metrics determined from the expert dataset are within the corresponding defined trust interval, testing, by the testing and validation platform, the machine learning model by determining at least one metric for each sample of the validation data set and/or the test dataset.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 3 recites the limitation:
(a) comparing the metrics determined from the expert dataset with the metrics determined from the validation data set and/or the test dataset.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 4 recites the limitation:
(a) defining clusters of the metrics determined from the expert data that determine cluster boundaries, defining clusters of the metrics determined from the validation data set and/or the test dataset, and comparing the clusters of the metrics determined from the expert dataset with the cluster boundaries.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 5 recites the limitation:
(a) determining that the machine learning model is safe if the clusters of the metrics determined from the validation data set and/or the test dataset are inside the cluster boundaries.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 6 recites the limitation:
(a) determining a score for the machine learning model based on the percentage of metrics determined from the validation data set and/or the test dataset that are inside the cluster boundaries.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 7 recites the limitation:
(a) iteratively testing the machine learning model by determining metrics from the expert dataset, determining metrics from the validation data set and/or the test dataset, and comparing the metrics from the expert dataset with the metrics from the validation data set and/or the test dataset.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 8 recites the limitation:
(a) wherein results from each test of each machine model iteration are saved for comparison.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 9 recites the limitation:
(a) wherein the at least one statistical test determines correlations between the features of the dataset and the displayed results include the determined correlations.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 10 recites the limitation:
(a) wherein the at least one statistical test identifies a degree of influence each of the features have on the at least one target and the displayed results include the features and the corresponding identified degrees of influence.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
These claims are dependent on Claim 1, but do not add any feature or subject matter that would solve the judicial exception deficiencies of Claim 1.
Claims 2-7, 9 and 10 recite further mental steps which can be practically performed in the human mind alone using observation, evaluation, judgment, and opinion or with the aid of pen and paper and thus, fail to make the claim any less abstract (see MPEP § 2106.04(a)(2)(III)).
Claim 2 and 7 recite further additional elements that amount to no more than mere instructions to apply the judicial exception using generic computer components. Mere instructions to apply a judicial exception using generic computer components cannot provide an inventive concept. See MPEP 2106.05(f).
Claim 8 recites further additional elements that fail to meaningfully limit the claim because it is mere data gathering/transmitting/outputting recited at a high level of generality, and thus is insignificant extra-solution activity (see MPEP § 2106.05(g)), and fails to integrated into practical application and they does not amount to significant more than the abstract idea.
Therefore, Claims 2-10 do not add any steps or additional elements, when considered both individually and as a combination, that would convert Claim 1 into patent-eligible subject matter.
Claims 1-10 are therefore not drawn to patent-eligible subject matter as they are directed to an abstract idea without significant more.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim Interpretation: Under the broadest reasonable interpretation (BRI), the limitations of Claim 11 are presumed to have their plain meaning consistent with the specification as it would be interpreted by one of ordinary skill in the art. See MPEP § 2111.
Step 1: Claim 11 is directed to a system, which is a machine and/or manufacture, and falls within one of the statutory categories of invention.
Step 2A, Prong One: Claim 11 recites the limitations:
a. analyzing a received dataset by conducting at least one statistical test of the dataset, the dataset comprising data representative of samples each comprising features and a corresponding target;
b. validating subsets of the dataset, the subsets comprising an expert dataset, a training dataset, and a validation dataset and/or a test dataset, wherein each subset comprises data representative of distinct samples; and
[…], by using the expert dataset to determine at least one metric for each sample of the expert dataset and comparing each metric of the at least one metric to a corresponding defined trust interval.
These recited steps, under the broadest reasonable interpretation (BRI), cover performance of the steps in the human mind alone or with the aid of pen and paper. That is, other than reciting:
providing a testing and validation platform for machine learning lifecycle testing and validation.
a testing and validation platform configured for:
testing the machine learning model and after the machine learning model is trained, […].
These recited steps, under the broadest reasonable interpretation (BRI), cover performance of the steps in the human mind alone or with the aid of pen and paper. That is, the limitations in (a), (b) and (c) can be reasonably interpreted as mental processes that can be practically performed in the human mind alone using observation, evaluation, judgment, and opinion or with the aid of pen and paper. For example, the limitation (a) for the analyzing step, a human can read dataset information stored in a database using observation, evaluation, judgment, and opinion to analyze the dataset. The limitation (b) for the validating step, a human can read dataset information stored in a database using observation, evaluation, judgment, and opinion to validate the subsets of dataset. The limitation (c) for the determine and comparing step, a human can read dataset information stored in a database using observation, evaluation, judgment, and opinion to determine and compare the metric for the dataset to a to a corresponding defined trust interval. See MPEP § 2106.04(a)(2)(III).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the human mind alone or with the aid of pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A, Prong Two: This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements:
1. providing a testing and validation platform for machine learning lifecycle testing and validation.
a testing and validation platform configured for:
testing the machine learning model and after the machine learning model is trained, […].
displaying results of the analysis;
receiving a selected at least one feature of the features for training a machine learning model;
The additional elements (1), (2) and (3) are recited at a high-level of generality such that they amount to no more than mere instructions to apply the judicial exception using generic computer components. The testing and validation platform for machine learning lifecycle testing and validation and the machine learning model are used as tools to perform the analyzing, displaying, receiving, validating, testing, determine and comparing steps of the claim. See MPEP § 2106.05(f).
The additional element (4) and (5) are mere data outputting/gathering recited at a high level of generality, and thus are insignificant extra-solution activities. See MPEP § 2106.05(g). Furthermore, all uses of the recited judicial exception require such data outputting/gathering and, as such, the additional elements do not impose any meaningful limits on the claim. The additional elements amount to necessary data outputting/gathering. See MPEP § 2106.05.
Accordingly, even when viewed in combination, the additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as a combination do not amount to significantly more than the abstract idea. As discussed above with respect to integration of the abstract idea into a practical application, the claim recites the additional elements:
1. providing a testing and validation platform for machine learning lifecycle testing and validation.
a testing and validation platform configured for:
testing the machine learning model and after the machine learning model is trained, […].
displaying results of the analysis;
receiving a selected at least one feature of the features for training a machine learning model;
The additional elements (1), (2) and (3) amount to no more than mere instructions to apply the judicial exception using generic computer components. Mere instructions to apply a judicial exception using generic computer components cannot provide an inventive concept.
The additional element (4) and (5) simply append well-understood, routine, and conventional activity previously known to the industry, specified at a high level of generality, to the judicial exception is not indicative of an inventive concept. MPEP § 2106.05(d)(II) expressly states that the courts have recognized the computer functions of receiving or transmitting data over a network, e.g., using the Internet to gather data and presenting offers and gathering statistics as well‐understood, routine, and conventional computer functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as an insignificant extra-solution activity. Thus, a person of ordinary skill in the art would readily comprehend that it is well-understood, routine, and conventional in the computing art to displaying results of the analysis and receiving a selected at least one feature of the features for training a machine learning model. Therefore, the limitation remains insignificant extra-solution activity even upon reconsideration and does not amount to significantly more.
Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the additional elements as a combination adds nothing that is not already present when looking at the additional elements taken individually. Even when considered in combination, the additional elements represent mere instructions to apply a judicial exception using generic computer components, insignificant extra-solution activities, and therefore do not provide an inventive concept. The claim is not patent eligible.
Claims 12-18 are rejected under 35 U.S.C. 101 as directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more for at least the reasons stated above.
Claim 12 recites the limitation:
(a) if the metrics determined from the expert dataset are within the corresponding defined trust interval, testing the machine learning model by determining at least one metric for each sample of the validation data set and/or the test dataset.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 13 recites the limitation:
(a) comparing the metrics determined from the expert dataset with the metrics determined from the validation data set and/or the test dataset.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 14 recites the limitation:
(a) defining clusters of the metrics determined from the expert data that determine cluster boundaries, defining clusters of the metrics determined from the validation data set and/or the test dataset, and comparing the clusters of the metrics determined from the expert dataset with the cluster boundaries.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 15 recites the limitation:
(a) determining that the machine learning model is safe if the clusters of the metrics determined from the validation data set and/or the test dataset are inside the cluster boundaries.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 16 recites the limitation:
(a) determining a score for the machine learning model based on the percentage of metrics determined from the validation data set and/or the test dataset that are inside the cluster boundaries.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 17 recites the limitation:
(a) iteratively testing the machine learning model by determining metrics from the expert dataset, determining metrics from the validation data set and/or the test dataset, and comparing the metrics from the expert dataset with the metrics from the validation data set and/or the test dataset.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 18 recites the limitation:
(a) the at least one statistical test identifies a degree of influence each of the features have on the at least one target and the displayed results include the features and the corresponding identified degrees of influence.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
These claims are dependent on Claim 11, but do not add any feature or subject matter that would solve the judicial exception deficiencies of Claim 11.
Claims 12-18 recite further mental steps which can be practically performed in the human mind alone using observation, evaluation, judgment, and opinion or with the aid of pen and paper and thus, fail to make the claim any less abstract (see MPEP § 2106.04(a)(2)(III)).
Claim 12 recite further additional elements that amount to no more than mere instructions to apply the judicial exception using generic computer components. Mere instructions to apply a judicial exception using generic computer components cannot provide an inventive concept. See MPEP 2106.05(f).
Therefore, Claims 12-18 do not add any steps or additional elements, when considered both individually and as a combination, that would convert Claim 11 into patent-eligible subject matter.
Claims 11-18 are therefore not drawn to patent-eligible subject matter as they are directed to an abstract idea without significant more.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim Interpretation: Under the broadest reasonable interpretation (BRI), the limitations of Claim 19 are presumed to have their plain meaning consistent with the specification as it would be interpreted by one of ordinary skill in the art. See MPEP § 2111.
Step 1: Claim 19 is directed to a non-transitory computer readable medium, which is an article of manufacture, and falls within one of the statutory categories of invention.
Step 2A, Prong One: Claim 19 recites the limitations:
a. analyzing a received dataset by conducting at least one statistical test of the dataset, the dataset comprising data representative of samples each comprising features and a corresponding target;
b. validating, […], subsets of the dataset, the subsets comprising an expert dataset, a training dataset, and a validation dataset and/or a test dataset, wherein each subset comprises data representative of distinct samples; and
c. […], by using the expert dataset to determine at least one metric for each sample of the expert dataset and comparing each metric of the at least one metric to a corresponding defined trust interval.
These recited steps, under the broadest reasonable interpretation (BRI), cover performance of the steps in the human mind alone or with the aid of pen and paper. That is, other than reciting:
1. A non-transitory computer readable medium having stored thereon executable instructions that when executed by at least one processor of at least one computer cause the at least one computer to perform steps comprising:
2. by the testing and validation platform.
3. testing the machine learning model and after the machine learning model is trained, […].
These recited steps, under the broadest reasonable interpretation (BRI), cover performance of the steps in the human mind alone or with the aid of pen and paper. That is, the limitations in (a), (b) and (c) can be reasonably interpreted as mental processes that can be practically performed in the human mind alone using observation, evaluation, judgment, and opinion or with the aid of pen and paper. For example, the limitation (a) for the analyzing step, a human can read dataset information stored in a database using observation, evaluation, judgment, and opinion to analyze the dataset. The limitation (b) for the validating step, a human can read dataset information stored in a database using observation, evaluation, judgment, and opinion to validate the subsets of dataset. The limitation (c) for the determine and comparing step, a human can read dataset information stored in a database using observation, evaluation, judgment, and opinion to determine and compare the metric for the dataset to a to a corresponding defined trust interval. See MPEP § 2106.04(a)(2)(III).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the human mind alone or with the aid of pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A, Prong Two: This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements:
1. A non-transitory computer readable medium having stored thereon executable instructions that when executed by at least one processor of at least one computer cause the at least one computer to perform steps comprising:
2. by the testing and validation platform.
3. testing the machine learning model and after the machine learning model is trained, […].
displaying results of the analysis;
receiving a selected at least one feature of the features for training a machine learning model;
The additional elements (1), (2) and (3) are recited at a high-level of generality such that they amount to no more than mere instructions to apply the judicial exception using generic computer components. The testing and validation platform for machine learning lifecycle testing and validation and the machine learning model are used as tools to perform the analyzing, displaying, receiving, validating, testing, determine and comparing steps of the claim. See MPEP § 2106.05(f).
The additional element (4) and (5) are mere data outputting/gathering recited at a high level of generality, and thus are insignificant extra-solution activities. See MPEP § 2106.05(g). Furthermore, all uses of the recited judicial exception require such data outputting/gathering and, as such, the additional elements do not impose any meaningful limits on the claim. The additional elements amount to necessary data outputting/gathering. See MPEP § 2106.05.
Accordingly, even when viewed in combination, the additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as a combination do not amount to significantly more than the abstract idea. As discussed above with respect to integration of the abstract idea into a practical application, the claim recites the additional elements:
1. A non-transitory computer readable medium having stored thereon executable instructions that when executed by at least one processor of at least one computer cause the at least one computer to perform steps comprising:
2. by the testing and validation platform.
3. testing the machine learning model and after the machine learning model is trained, […].
4. displaying results of the analysis;
5. receiving a selected at least one feature of the features for training a machine learning model;
The additional elements (1), (2) and (3) amount to no more than mere instructions to apply the judicial exception using generic computer components. Mere instructions to apply a judicial exception using generic computer components cannot provide an inventive concept.
The additional element (4) and (5) simply append well-understood, routine, and conventional activity previously known to the industry, specified at a high level of generality, to the judicial exception is not indicative of an inventive concept. MPEP § 2106.05(d)(II) expressly states that the courts have recognized the computer functions of receiving or transmitting data over a network, e.g., using the Internet to gather data and presenting offers and gathering statistics as well‐understood, routine, and conventional computer functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as an insignificant extra-solution activity. Thus, a person of ordinary skill in the art would readily comprehend that it is well-understood, routine, and conventional in the computing art to displaying results of the analysis and receiving a selected at least one feature of the features for training a machine learning model. Therefore, the limitation remains insignificant extra-solution activity even upon reconsideration and does not amount to significantly more.
Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the additional elements as a combination adds nothing that is not already present when looking at the additional elements taken individually. Even when considered in combination, the additional elements represent mere instructions to apply a judicial exception using generic computer components, insignificant extra-solution activities, and therefore do not provide an inventive concept. The claim is not patent eligible.
Claims 20 is rejected under 35 U.S.C. 101 as directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more for at least the reasons stated above.
Claim 20 recites the limitation:
(a) if the metrics determined from the expert dataset are within the corresponding defined trust interval, testing the machine learning model by determining at least one metric for each sample of the validation data set and/or the test dataset.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
These claims are dependent on Claim 19, but do not add any feature or subject matter that would solve the judicial exception deficiencies of Claim 19.
Claims 20 recites further mental steps which can be practically performed in the human mind alone using observation, evaluation, judgment, and opinion or with the aid of pen and paper and thus, fails to make the claim any less abstract (see MPEP § 2106.04(a)(2)(III)).
Claim 20 recites further additional elements that amount to no more than mere instructions to apply the judicial exception using generic computer components. Mere instructions to apply a judicial exception using generic computer components cannot provide an inventive concept. See MPEP 2106.05(f).
Therefore, Claims 20 does not add any steps or additional elements, when considered both individually and as a combination, that would convert Claim 19 into patent-eligible subject matter.
Claims 19-20 are therefore not drawn to patent-eligible subject matter as they are directed to an abstract idea without significant more.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 7-10, 11-13 and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over US 20220405613 (hereinafter “Han”) in view of US 20200273570 (hereinafter “SUBRAMANIAN”).
As per Claim 1, Han discloses:
A method for providing a testing and validation platform for machine learning lifecycle testing and validation, the method comprising:
analyzing […] a received dataset by conducting at least one statistical test of the dataset, the dataset comprising data representative of samples each comprising features and a corresponding target (Paragraph [0012], “According to another embodiment of the present disclosure, an approach is provided in which a method, system, and program product select a training dataset that includes multiple features and a target. For each of the features, the method, system, and program product select one of the features and perform a statistical test on the selected feature to determine whether the selected feature is statistically important to the target [analyzing […] a received dataset by conducting at least one statistical test of the dataset, the dataset comprising data representative of samples each comprising features and a corresponding target] (emphasis added).”;
receiving […] a selected at least one feature of the features for training a machine learning model (Paragraph [0060], “As discussed herein, system 300 uses both training dataset 305 and testing dataset 320 to select a set of features to build a machine learning model [receiving […] a selected at least one feature of the features for training a machine learning model] (emphasis added).”;
validating […] subsets of the dataset, the subsets comprising an expert dataset, a training dataset, and a validation dataset and/or a test dataset, wherein each subset comprises data representative of distinct samples (Paragraph [0004], “Adversarial validation combines the training data with the testing data and creates a pseudo target, which is “0” if the sample is from the training dataset and “1” if the sample is from the testing dataset [validating […] subsets of the dataset, the subsets comprising an expert dataset, a training dataset, and a validation dataset and/or a test dataset, wherein each subset comprises data representative of distinct samples] (emphasis added).”; and
testing […] and after the machine learning model is trained, the machine learning model, by using the expert dataset to determine at least one metric for each sample of the expert dataset and comparing each metric of the at least one metric to a corresponding defined trust interval (Paragraph [0049], “Then, for each feature Xi in differential feature set B 340, system 300 performs lead feature correlation statistic computations 355 on both the training entries and the testing entries to determine the correlation or association statistic between Xi and Xj and its confidence interval in the training dataset (emphasis added).”; Paragraph [0050], “If the testing dataset correlation is out of the range of the training dataset correlation, then system 300 determines that the relationship between Xi and Xj in the testing dataset is different from the relationship between Xi and Xj in the training dataset [testing […] and after the machine learning model is trained, the machine learning model, by using the expert dataset to determine at least one metric for each sample of the expert dataset and comparing each metric of the at least one metric to a corresponding defined trust interval] (emphasis added).”.
Han does not disclose:
[…] at the testing and validation platform […]
displaying, by the testing and validation platform, results of the analysis;
[…] by the testing and validation platform […]
However, SUBRAMANIAN discloses:
[…] at the testing and validation platform […] (Paragraph [0026], “In some implementations, the predictive analysis platform may perform multiple iterations of training of the machine learning model, depending on an outcome of testing of the machine learning model (e.g., by submitting different portions of the data as the training set of data, the validation set of data, and the test set of data [[…] at the testing and validation platform […]] (emphasis added).”;
displaying, by the testing and validation platform, results of the analysis (Paragraph [0026], “In some implementations, the predictive analysis platform may perform multiple iterations of training of the machine learning model, depending on an outcome of testing of the machine learning model (e.g., by submitting different portions of the data as the training set of data, the validation set of data, and the test set of data (emphasis added).”; Paragraph [0061], “Turning to FIG. 2C, and as shown by reference number 215, the predictive analysis platform may provide a UI for display that includes information that identifies a result of an analysis, a prediction that the predictive analysis platform generated, a score that the predictive analysis platform generated, and/or the like [displaying, by the testing and validation platform, results of the analysis] (emphasis added).”;
[…] by the testing and validation platform […] (Paragraph [0026], “In some implementations, the predictive analysis platform may perform multiple iterations of training of the machine learning model, depending on an outcome of testing of the machine learning model (e.g., by submitting different portions of the data as the training set of data, the validation set of data, and the test set of data [[…] by the testing and validation platform […]] (emphasis added).”;
Han is within the same field of endeavor as the claimed invention regardingan approach is provided in which a method, system, and program product perform a distribution test on a plurality of datasets corresponding to a plurality of predictive features. SUBRAMANIAN is within the same field of endeavor as the claimed invention regarding a predictive analysis platform that is capable of processing data, from multiple separate and isolated systems. Thus, Han and SUBRAMANIAN are analogous art to the claimed invention.
Therefore, it would have been obvious to one of ordinary skill in the art before effective filing date of the claimed invention to incorporate the teaching of SUBRAMANIAN into the teaching of Han to include “[…] at the testing and validation platform […] displaying, by the testing and validation platform, results of the analysis; […] by the testing and validation platform […]” The modification would be obvious because one of the ordinary skills in the art would be motivated to improve a security and/or privacy of data that is accessible by the predictive analysis platform (SUBRAMANIAN, paragraph [0011]).
As per Claim 2, the rejection of Claim 1 is incorporated; and the combination of Han and SUBRAMANIAN discloses “by the testing and validation platform”, and Han further discloses:
if the metrics determined from the expert dataset are within the corresponding defined trust interval, testing […] the machine learning model by determining at least one metric for each sample of the validation data set and/or the test dataset (Paragraph [0004], “Adversarial validation combines the training data with the testing data and creates a pseudo target, which is “0” if the sample is from the training dataset and “1” if the sample is from the testing dataset (emphasis added).”; Paragraph [0049], “Then, for each feature Xi in differential feature set B 340, system 300 performs lead feature correlation statistic computations 355 on both the training entries and the testing entries to determine the correlation or association statistic between Xi and Xj and its confidence interval in the training dataset (emphasis added).”; Paragraph [0050], “If the testing dataset correlation is out of the range of the training dataset correlation, then system 300 determines that the relationship between Xi and Xj in the testing dataset is different from the relationship between Xi and Xj in the training dataset [if the metrics determined from the expert dataset are within the corresponding defined trust interval, testing […] the machine learning model by determining at least one metric for each sample of the validation data set and/or the test dataset] (emphasis added).”.
As per Claim 3, the rejection of Claim 2 is incorporated; and Han further discloses:
comparing the metrics determined from the expert dataset with the metrics determined from the validation data set and/or the test dataset (Paragraph [0050], “If the testing dataset correlation is out of the range of the training dataset correlation, then system 300 determines that the relationship between Xi and Xj in the testing dataset is different from the relationship between Xi and Xj in the training dataset [comparing the metrics determined from the expert dataset with the metrics determined from the validation data set and/or the test dataset] (emphasis added).”.
As per Claim 7, the rejection of Claim 3 is incorporated; and Han further discloses:
[…] by determining metrics from the expert dataset, determining metrics from the validation data set and/or the test dataset, and comparing the metrics from the expert dataset with the metrics from the validation data set and/or the test dataset (Paragraph [0049], “Then, for each feature Xi in differential feature set B 340, system 300 performs lead feature correlation statistic computations 355 on both the training entries and the testing entries to determine the correlation or association statistic between Xi and Xj and its confidence interval in the training dataset (emphasis added).”; Paragraph [0050], “If the testing dataset correlation is out of the range of the training dataset correlation, then system 300 determines that the relationship between Xi and Xj in the testing dataset is different from the relationship between Xi and Xj in the training dataset [[…] by determining metrics from the expert dataset, determining metrics from the validation data set and/or the test dataset, and comparing the metrics from the expert dataset with the metrics from the validation data set and/or the test dataset] (emphasis added).”.
But Han does not explicitly disclose:
iteratively testing the machine learning model […]
However, SUBRAMANIAN discloses:
iteratively testing the machine learning model […] (Paragraph [0026], “In some implementations, the predictive analysis platform may perform multiple iterations of training of the machine learning model, depending on an outcome of testing of the machine learning model (e.g., by submitting different portions of the data as the training set of data, the validation set of data, and the test set of data [iteratively testing the machine learning model […]] (emphasis added).”.
SUBRAMANIAN is within the same field of endeavor as the claimed invention regarding a predictive analysis platform that is capable of processing data, from multiple separate and isolated systems. Thus, SUBRAMANIAN is an analogous art to the claimed invention.
Therefore, it would have been obvious to one of ordinary skill in the art before effective filing date of the claimed invention to incorporate the teaching of SUBRAMANIAN into the teaching of Han to include “iteratively testing the machine learning model […]” The modification would be obvious because one of the ordinary skills in the art would be motivated to produce more accurate machine learning model by iteratively training machine learning model (SUBRAMANIAN, paragraph [0028]).
As per Claim 8, the rejection of Claim 7 is incorporated; and Han does not explicitly disclose:
results from each test of each machine model iteration are saved for comparison.
However, SUBRAMANIAN discloses:
results from each test of each machine model iteration are saved for comparison (Paragraph [0026], “In some implementations, the predictive analysis platform may perform multiple iterations of training of the machine learning model, depending on an outcome of testing of the machine learning model (e.g., by submitting different portions of the data as the training set of data, the validation set of data, and the test set of data (emphasis added).”; Paragraph [0037], “Additionally, or alternatively, and as another example, the historical data component may perform a comparison of a signature of processed data associated with an anonymized identifier to multiple signatures of other data stored in the data structure, and may identify the historical data based on a match of signatures. Additionally, or alternatively, and as another example, the historical data component may use a machine learning model to identify the historical data (e.g., by identifying historical data that has a similar signature to a signature of processed data associated with an anonymized identifier) [results from each test of each machine model iteration are saved for comparison] (emphasis added).”.
SUBRAMANIAN is within the same field of endeavor as the claimed invention regarding a predictive analysis platform that is capable of processing data, from multiple separate and isolated systems. Thus, SUBRAMANIAN is an analogous art to the claimed invention.
Therefore, it would have been obvious to one of ordinary skill in the art before effective filing date of the claimed invention to incorporate the teaching of SUBRAMANIAN into the teaching of Han to include “results from each test of each machine model iteration are saved for comparison” The modification would be obvious because one of the ordinary skills in the art would be motivated to produce more accurate machine learning model by iteratively training machine learning model (SUBRAMANIAN, paragraph [0028]).
As per Claim 9, the rejection of Claim 1 is incorporated; and Han further discloses:
wherein the at least one statistical test determines correlations between the features of the dataset […] the determined correlations. (Paragraph [0049], “Then, for each feature Xi in differential feature set B 340, system 300 performs lead feature correlation statistic computations 355 on both the training entries and the testing entries to determine the correlation or association statistic between Xi and Xj and its confidence interval in the training dataset [wherein the at least one statistical test determines correlations between the features of the dataset and the displayed results include the determined correlations] (emphasis added).”.
But Han does not explicitly disclose:
[…] the displayed results include […].
However, SUBRAMANIAN discloses:
[…] the displayed results include […] (Paragraph [0061], “Turning to FIG. 2C, and as shown by reference number 215, the predictive analysis platform may provide a UI for display that includes information that identifies a result of an analysis, a prediction that the predictive analysis platform generated, a score that the predictive analysis platform generated, and/or the like [[…] the displayed results include […]] (emphasis added).”.
SUBRAMANIAN is within the same field of endeavor as the claimed invention regarding a predictive analysis platform that is capable of processing data, from multiple separate and isolated systems. Thus, SUBRAMANIAN is an analogous art to the claimed invention.
Therefore, it would have been obvious to one of ordinary skill in the art before effective filing date of the claimed invention to incorporate the teaching of SUBRAMANIAN into the teaching of Han to include “[…] the displayed results include […]” The modification would be obvious because one of the ordinary skills in the art would be motivated to allow displaying of statistical test results.
As per Claim 10, the rejection of Claim 1 is incorporated; and Han further discloses:
wherein the at least one statistical test identifies a degree of influence each of the features have on the at least one target and […] the features and the corresponding identified degrees of influence (Paragraph [0051], “Consistent feature set C 365 includes features that have statistical consistency between training dataset 305 and testing dataset 320 and provide predictive influence to a specified target [wherein the at least one statistical test identifies a degree of influence each of the features have on the at least one target and […] the features and the corresponding identified degrees of influence] (emphasis added).”.
But Han does not explicitly disclose:
[…] the displayed results include […].
However, SUBRAMANIAN discloses:
[…] the displayed results include […] (Paragraph [0061], “Turning to FIG. 2C, and as shown by reference number 215, the predictive analysis platform may provide a UI for display that includes information that identifies a result of an analysis, a prediction that the predictive analysis platform generated, a score that the predictive analysis platform generated, and/or the like [[…] the displayed results include […]] (emphasis added).”.
SUBRAMANIAN is within the same field of endeavor as the claimed invention regarding a predictive analysis platform that is capable of processing data, from multiple separate and isolated systems. Thus, SUBRAMANIAN is an analogous art to the claimed invention.
Therefore, it would have been obvious to one of ordinary skill in the art before effective filing date of the claimed invention to incorporate the teaching of SUBRAMANIAN into the teaching of Han to include “[…] the displayed results include […]” The modification would be obvious because one of the ordinary skills in the art would be motivated to enable displaying of statistical test results.
Claims 11-13, 17 and 18 are system claims corresponding to the method claims hereinabove (Claims 1-3, 7 and 10, respectively). Therefore, Claims 11-13, 17 and 18 are rejected for the same reasons set forth in the rejections of Claims 1-3, 7 and 10.
Claims 19 and 20 are non-transitory computer-readable storage medium claims corresponding to the method claims hereinabove (Claims 1 and 2, respectively). Therefore, Claims 19 and 20 are rejected for the same reasons set forth in the rejection of Claims 1 and 2.
Claims 4-6 and 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Han in view of SUBRAMANIAN as applied to claims 1 and 11 above, and further in view of US 20230197287 (hereinafter “Samadani”).
As per Claim 4, the rejection of Claim 3 is incorporated; and Han further discloses
[…] determined from the expert data […] determined from the validation data set and/or the test dataset, and comparing […] determined from the expert dataset with […] (Paragraph [0049], “Then, for each feature Xi in differential feature set B 340, system 300 performs lead feature correlation statistic computations 355 on both the training entries and the testing entries to determine the correlation or association statistic between Xi and Xj and its confidence interval in the training dataset (emphasis added).”; Paragraph [0050], “If the testing dataset correlation is out of the range of the training dataset correlation, then system 300 determines that the relationship between Xi and Xj in the testing dataset is different from the relationship between Xi and Xj in the training dataset [[…] determined from the expert data […] determined from the validation data set and/or the test dataset, and comparing […] determined from the expert dataset with […]] (emphasis added).”.
but the combination of Han and SUBRAMANIAN does not explicitly disclose:
defining clusters of the metrics […] determine cluster boundaries, defining clusters of the metrics […] the clusters of the metrics […] the cluster boundaries.
However, Samadani discloses:
defining clusters of the metrics […] determine cluster boundaries, defining clusters of the metrics […] the clusters of the metrics […] the cluster boundaries (Paragraph [0073], “When the NH is within a threshold distance to the closest existing cluster of the clusters of the IHG, the process may continue to act 225. However, if it is determined that the NH is not within a threshold distance (e.g., d is greater than or equal to the threshold distance) to the closest existing cluster of the clusters of the IHG, the process may continue to act 228 (emphasis added).”; Paragraph [0081], “Alternatively, the hierarchy level and the number of clusters can be determined using goodness of clustering metrics such as silhouette coefficients and/or elbow method on the sum of squared errors (the square of the Euclidean distance of the point to its cluster head or cluster centroid) [defining clusters of the metrics […] determine cluster boundaries, defining clusters of the metrics […] the clusters of the metrics […] the cluster boundaries] (emphasis added).”.
Samadani is within the same field of endeavor as the claimed invention regarding a system that determining whether another hospital is within a threshold distance to a closest one of the plurality of clusters. Thus, Samadani is an analogous art to the claimed invention.
Therefore, it would have been obvious to one of ordinary skill in the art before effective filing date of the claimed invention to incorporate the teaching of Samadani into the combined teachings of Han and SUBRAMANIAN to include “defining clusters of the metrics […] determine cluster boundaries, defining clusters of the metrics […] the clusters of the metrics […] the cluster boundaries” The modification would be obvious because one of the ordinary skills in the art would be motivated to improve the accuracy and performance of the measurement by allowing usage of clusters of metrics (Samadani, paragraph [0005]).
As per Claim 5, the rejection of Claim 4 is incorporated; and the combination of Han and SUBRAMANIAN discloses “determined from the validation data set and/or the test dataset” but the combination of Han and SUBRAMANIAN does not explicitly disclose:
determining that the machine learning model is safe if the clusters of the metrics […] are inside the cluster boundaries.
However, Samadani discloses:
determining that the machine learning model is safe if the clusters of the metrics […] are inside the cluster boundaries (Paragraph [0073], “When the NH is within a threshold distance to the closest existing cluster of the clusters of the IHG, the process may continue to act 225. However, if it is determined that the NH is not within a threshold distance (e.g., d is greater than or equal to the threshold distance) to the closest existing cluster of the clusters of the IHG, the process may continue to act 228 (emphasis added).”; Paragraph [0081], “Alternatively, the hierarchy level and the number of clusters can be determined using goodness of clustering metrics such as silhouette coefficients and/or elbow method on the sum of squared errors (the square of the Euclidean distance of the point to its cluster head or cluster centroid (emphasis added).”; Paragraph [0086], “Further, the resulting prediction uncertainty can be used by embodiments of the system to qualify a valid prediction as the prediction with the least variation across different models [defining clusters of the metrics […] determine cluster boundaries, defining clusters of the metrics […] the clusters of the metrics […] the cluster boundaries] (emphasis added).”.
Samadani is within the same field of endeavor as the claimed invention regarding a system that determining whether another hospital is within a threshold distance to a closest one of the plurality of clusters. Thus, Samadani is an analogous art to the claimed invention.
Therefore, it would have been obvious to one of ordinary skill in the art before effective filing date of the claimed invention to incorporate the teaching of Samadani into the combined teachings of Han and SUBRAMANIAN to include “determining that the machine learning model is safe if the clusters of the metrics […] are inside the cluster boundaries” The modification would be obvious because one of the ordinary skills in the art would be motivated to improve the accuracy and performance of the measurement by allowing usage of clusters of metrics (Samadani, paragraph [0005]).
As per Claim 6, the rejection of Claim 4 is incorporated; and the combination of Han and Samadani discloses “that are inside the cluster boundaries” and Han further discloses:
[…] metrics determined from the validation data set and/or the test dataset […] (Paragraph [0049], “Then, for each feature Xi in differential feature set B 340, system 300 performs lead feature correlation statistic computations 355 on both the training entries and the testing entries to determine the correlation or association statistic between Xi and Xj and its confidence interval in the training dataset (emphasis added).”; Paragraph [0050], “If the testing dataset correlation is out of the range of the training dataset correlation, then system 300 determines that the relationship between Xi and Xj in the testing dataset is different from the relationship between Xi and Xj in the training dataset [[…] metrics determined from the validation data set and/or the test dataset […]] (emphasis added).”.
But the combination of Han and Samadani does not explicitly disclose:
determining a score for the machine learning model based on the percentage of […]
However, SUBRAMANIAN discloses:
determining a score for the machine learning model based on the percentage of […] (Paragraph [0032], “As another example, the predictive analysis platform may determine, using a linear regression technique, that a threshold percentage of values of data elements, in a set of values of data elements, do not indicate future combinations of future care, whether a claim should be approved, and/or the like, and may determine that those values of data elements are to receive relatively low association scores (emphasis added).”; Paragraph [0050], “For example, the predictive analysis platform may use a machine learning model to perform an analysis, and the machine learning model may output a score in association with outputting a result of an analysis [determining a score for the machine learning model based on the percentage of […]] (emphasis added).”.
SUBRAMANIAN is within the same field of endeavor as the claimed invention regarding a predictive analysis platform that is capable of processing data, from multiple separate and isolated systems. Thus, SUBRAMANIAN is an analogous art to the claimed invention.
Therefore, it would have been obvious to one of ordinary skill in the art before effective filing date of the claimed invention to incorporate the teaching of SUBRAMANIAN into the combined teaching of Han and Samadani to include “determining a score for the machine learning model based on the percentage of […]” The modification would be obvious because one of the ordinary skills in the art would be motivated to improve a security and/or privacy of data that is accessible by the predictive analysis platform by outputting a score from the machine learning model (SUBRAMANIAN, paragraph [0011]).
Claims 14-16 are system claims corresponding to the method claims hereinabove (Claims 4-6, respectively). Therefore, Claims 14-16 are rejected for the same reasons set forth in the rejections of Claims 14-16.
Conclusion
The prior art made of record and not relied upon is considered pertinent to the Applicant’s disclosure. They are as follows:
US 2022/0083445 (hereinafter “Nyati”) discloses computerized systems that perform real-time and/or online monitoring, tracking, and/or measuring of machine learning model performance.
US 2022/0066906 (hereinafter “Kumar”) discloses systems and techniques enable prediction of a state of an application at a future time, with high levels of accuracy and specificity.
US 2020/0341830 (hereinafter “Bangad”) discloses an application monitoring device that includes a memory operable to store an application and a fault detection engine implemented by a processor.
US 2020/0242000 (hereinafter “Khosrowpour”) discloses techniques are disclosed for determining the run-time performance of an application executing on a computing system with low impact on the performance of the computing system.
9. Any inquiry concerning this communication or earlier communications from the Examiner should be directed to Yanbin Li whose telephone number is 571-272-0906. The Examiner can normally be reached on Monday through Friday from 8:30 AM to 4:30 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, the Applicant is encouraged to use the USPTO Automated Interview Request (AIR) at https://www.uspto.gov/interviewpractice.
If attempts to reach the Examiner by telephone are unsuccessful, the Examiner’s supervisor, Wei Mui, can be reached at 571-272-3708. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for more information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO customer service representative, call 800-786-9199 (in USA or Canada) or 571-272-1000.
/Y.L./Examiner, Art Unit 2191
/WEI Y MUI/Supervisory Patent Examiner, Art Unit 2191