DETAILED ACTION
Status of Claims
This Office action is responsive to communications filed on 2024-11-22. Claim(s) 1-20 is/are pending and are examined herein.
Claim(s) 1-20 is/are objected to.
Claim(s) 4-6, 8, 12-13, and 18-20 is/are rejected under 35 USC 112(b).
Claim(s) 1-20 is/are rejected under 35 USC 101.
Claim(s) 1-3 and 9-17 is/are rejected under 35 USC 102.
Claim(s) 4-8 and 18-20 is/are rejected under 35 USC 103.
Notice of Pre-AIA or AIA Status
The present application, filed on or after 2013-03-16, is being examined under the first inventor to file provisions of the AIA .
Priority
The present application claims priority from European Patent Application No. 23211700.2, filed 2023-11-23. Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The attached information disclosure statement(s) (IDS), submitted on 2024-11-22, 2026-01-30, and 2026-06-08, is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the attached information disclosure statement(s) is/are being considered by the examiner.
Examiner’s Remarks
MPEP 2111.04(II) indicates that the “broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only that those steps that must be performed and does not include steps that are not required to be performed because the condition(s) are not met”. Claims 12-13 recite such conditional limitations. Specifically:
[Claim 12] in the case that a candidate model has a higher performance metric than that determined for a currently-selected model, the method comprises using the candidate model to replace the currently-selected model in response to a difference between the performance metrics for the two models exceeding a predetermined threshold.
[Claim 13] selecting the second machine learning model in response to the analysis of the model activity data indicating a data quality problem concerning the known problematic signal.
In both cases, the conditions in these limitations are not positively recited by the claims: claim 12 does not explicitly recite the performance metric of the candidate model being higher than that of the currently-selected model and the difference between the performance metrics exceeding a threshold, and claim 13 does not explicitly recite the analysis indicating a data quality problem. The following is a list of contingent limitations occurring in method claims in the instant application. Thus, in keeping with the interpretation of contingent limitations in method claims prescribed by MPEP 2111.04, subsection II, the limitations cited above are not part of the broadest reasonable interpretation of method claims in which they occur. The applicant is advised to amend the claims to positively recite the conditions if they wish for claim scope to include the ensuing steps.
Claim Objections
Claim(s) 1-20 is/are objected to because of the following informalities:
Claims 1 and 15 recite based on the analysis of the model activity data [emphasis added] but the underlined phrase lacks antecedent basis. The examiner suggests “based on the analyzing of the obtained model activity data” for proper antecedent basis. Dependent claims 2-14 and 16-20 inherit the rejection.
Claims 2 and 16 recite maintaining the obtained model activity data and/or the model management data [emphasis added] but the underlined phrase renders claim scope unclear. The examiner suggests “maintaining the obtained model activity data
Similarly, claims 3 and 17 recite with the obtained model activity data and/or the model management data [emphasis added] but the underlined phrase renders claim scope unclear. The examiner suggests “with the obtained model activity data
Claims 4, 7, 14 and 18 recite one of the models in the distributed setup [emphasis added] but the underlined phrase lacks antecedent basis. The examiner suggests “one of the machine learning models in the distributed setup” for proper antecedent basis. Dependent claims 5-6, 8, and 19-20 inherit the objection.
Claims 5 and 19 recite outputting the at least one known similar issue [emphasis added] but the underlined phrase lacks antecedent basis. The examiner suggests “outputting the at least one known
Claim 7 recites based on the model activity data [emphasis added] but this should be “based on the obtained model activity data” for consistent nomenclature.
Claim 14 recites to inform a human about the current state or future state of the industrial process [emphasis added] but the underlined phrase lacks antecedent basis. The examiner suggests “to inform a human about a current state or a future state of the industrial process” for proper antecedent basis.
Appropriate correction is required.
Claim Rejections - 35 USC 112(b)
The following is a quotation of 35 USC 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 USC 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 4-6, 8, 12-13, and 18-20 is/are rejected under 35 USC 112(b) or 35 USC 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 USC 112, the applicant), regards as the invention.
Claims 4 and 18 recite wherein the reported issue is an unknown issue [emphasis added] but the meaning of this limitation is unclear for at least the following reasons. First, the word “unknown” is subjective language: something that is “unknown” to one person may be “known” to another, and the claim provides no clarification regarding whom the issue is to be “unknown” to. MPEP 2173.05(b)(IV) indicates that, in the presence of subjective claim terminology, “[s]ome objective standard must be provided in order to allow the public to determine the scope of the claim. A claim term that requires the exercise of subjective judgment without restriction may render the claim indefinite”. In the present instance, the specification merely repeats the language used by the claim without providing any objective criteria regarding what it means for an issue to be “unknown”. Second, it is not clear what it means for an issue that has been “reported” to nonetheless be “unknown”, since the fact that it has been “reported” would appear to suggest that it is necessarily known, both to the entity which reported the issue and to the entity to whom the issue was reported. These issues together regarding the “unknown issue” of the claim render the claim indefinite. Dependent claims 5-6, 8, and 19-20 inherit the rejection. As best understood by the examiner in view of the subsequent limitations recited by the claim (e.g., “generating a label for the unknown issue”), the “unknown issue” of the claim is interpreted herein as referring to an issue that is unlabeled before model activity data is analyzed but which may be labeled by the analysis steps of the claims.
Claims 5 and 19 further recite at least one known issue which is similar to the unknown issue [emphasis added] but this is indefinite because it is both approximative and subjective language. It is indefinite as approximative language in view of MPEP 2173.05(b)(III)(C). It is subjective language because two issues that one person regards as being “similar” may not be regarded as being “similar” by another person. MPEP 2173.05(b)(IV) indicates that, in the presence of subjective claim terminology, “[s]ome objective standard must be provided in order to allow the public to determine the scope of the claim. A claim term that requires the exercise of subjective judgment without restriction may render the claim indefinite” and, in the present instance, the specification provides no clear objective criterion by which issues are deemed to be similar.
Claim 8 recites the reported issue or the predicted issue relates to one or more of model quality, data quality, and operational quality [emphasis added] but the underlined phrase lacks antecedent basis since neither the claim itself nor parent claim 4 introduce a “predicted issue”. The examiner suggests “the reported issue
Claim 12 recites wherein, in the case that a candidate model has a higher performance metric than that determined for a currently-selected model, the method comprises using the candidate model to replace the currently-selected model in response to a difference between the performance metrics for the two models exceeding a predetermined threshold [emphasis added] but the underlined phrases lack antecedent. Alternative language is advised. The examiner suggests:
“whereina candidate model to replace a currently-selected model in response to a difference between a performance metric for the candidate model and a performance metric for a currently-selected model exceeding a predetermined non-negative threshold”
The examiner notes that the difference between the two performance metrics exceeding a “non-negative” threshold also ensures that the performance metric of the candidate model is higher than that of the currently-selected model, so the suggestion as made above appears not to change intended claim scope. For the purpose of compact prosecution, the claim is interpreted as encompassing at least the above interpretation. The examiner notes that the suggestion made here remains a conditional limitation, as in the original (cf. examiner’s remarks).
Claim 13 recites The method of claim 10, … wherein selecting the machine learning model based on environmental conditions comprises selecting the second machine learning model in response to the analysis of the model activity data indicating a data quality problem concerning the known problematic signal. [emphasis added] but the underlined phrases lacks antecedent basis, and neither the claim itself nor parent claim 10 recites a step of “selecting”. As best understood by the examiner, this claim is likely intended to be dependent on claim 11, which does recite a step of selecting a machine learning model. The examiner correspondingly suggests:
“The method of claim 11, … wherein the selecting of the one of the machine learning models based on environmental conditions comprises selecting the second machine learning model in response to the analyzing of the model activity data indicating a data quality problem concerning the known problematic signal.”
For the purpose of compact prosecution, the claim is interpreted as encompassing at least the above interpretation. The examiner notes that the suggestion made here remains a conditional limitation, as in the original (cf. examiner’s remarks).
Claim Rejections - 35 USC 101 - Statutory Categories
35 USC 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 15-20 is/are rejected under 35 USC 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter for the following reasons.
Claim 15 is directed to A computer-readable medium comprising… [emphasis added]. However, the specification does not include an explicit disavowal of transitory signals, and in fact specifically indicates that transitory signals are included: “a propagated signal may be included within the scope of computer-readable storage media” [specification, 0060]. MPEP 2106.03 indicates that “examples of claims that are not directed to any of the statutory categories” include those that are directed to “[t]ransitory forms of signal transmission (often referred to as ‘signals per se’), such as a propagating electrical or electromagnetic signal or carrier wave”. Consequently, the scope of claim 15 includes non-statutory subject matter. Dependent claims 16-20 inherit the rejection. The examiner suggests “A non-transitory computer-readable medium comprising…” to avoid this issue.
Claim Rejections - 35 USC 101 - Abstract Idea
35 USC 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-20 is/are rejected under 35 USC 101 because the claimed invention(s) is/are directed to abstract ideas without significantly more.
Claim 1
Step 1. The claim and its dependents 2-14 fall under the statutory category of methods. An analysis of step 2 for each of these claims follows.
Step 2A Prong 1. The claim recites the following abstract ideas:
A [computer-implemented] method for monitoring machine learning models (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
analyzing the obtained model activity data; (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
managing the activity of the machine learning models in the distributed setup. (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
[A] computer-implemented [method] (This generic computing components for performing an abstract idea. See MPEP 2106.05(f)(2).)
in a distributed setup, (This recites a general link between an abstract idea and a particular field of use or technological environment. See MPEP 2106.05(h).)
the method comprising: obtaining model activity data relating to activity of the machine learning models in the distributed setup; (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
and based on the analysis of the model activity data, outputting model management data for (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
[A] computer-implemented [method] (This generic computing components for performing an abstract idea. See MPEP 2106.05(f)(2).)
in a distributed setup, (This recites a general link between an abstract idea and a particular field of use or technological environment. See MPEP 2106.05(h).)
the method comprising: obtaining model activity data relating to activity of the machine learning models in the distributed setup; (This insignificant extra-solution activity is well-understood, routine, conventional as it is mere data transfer. See MPEP 2106.05(d)(II), “Receiving or transmitting data over a network” and/or “Storing and retrieving information in memory”.)
and based on the analysis of the model activity data, outputting model management data for (The insignificant extra-solution activity is well-understood, routine, conventional as it is merely presenting output. See MPEP 2106.05(d)(II), “Presenting offers”.)
Claim 2
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
[The method of claim 1, further comprising] maintaining the obtained model activity data and/or the model management data in one or more knowledge databases. (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
[The method of claim 1, further comprising] maintaining the obtained model activity data and/or the model management data in one or more knowledge databases. (This insignificant extra-solution activity is well-understood, routine, conventional as it is mere data storage. See MPEP 2106.05(d)(II), “Electronic recordkeeping” and/or “Storing and retrieving information in memory”.)
Claim 3
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
[The method of claim 1, further comprising] updating a cache of one or more nodes of the distributed setup with the obtained model activity data and/or the model management data. (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
[The method of claim 1, further comprising] updating a cache of one or more nodes of the distributed setup with the obtained model activity data and/or the model management data. (This insignificant extra-solution activity is well-understood, routine, conventional as it is mere data storage. See MPEP 2106.05(d)(II), “Electronic recordkeeping” and/or “Storing and retrieving information in memory”.)
Claim 4
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[and wherein analyzing the model activity data comprises] generating a label for the unknown issue, (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
[The method of claim 1, wherein] the obtained model activity data relates to an issue reported by one of the models in the distributed setup, wherein the reported issue is an unknown issue, (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
[the method comprising] outputting the label as at least part of the model management data. (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
[The method of claim 1, wherein] the obtained model activity data relates to an issue reported by one of the models in the distributed setup, wherein the reported issue is an unknown issue, (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
[the method comprising] outputting the label as at least part of the model management data. (The insignificant extra-solution activity is well-understood, routine, conventional as it is merely presenting output. See MPEP 2106.05(d)(II), “Presenting offers”.)
Claim 5
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[The method of claim 4, wherein analyzing the model activity data further comprises] identifying at least one known issue which is similar to the unknown issue, (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
[the method comprising] outputting data related to the at least one known similar issue as at least part of the model management data. (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
[the method comprising] outputting data related to the at least one known similar issue as at least part of the model management data. (The insignificant extra-solution activity is well-understood, routine, conventional as it is merely presenting output. See MPEP 2106.05(d)(II), “Presenting offers”.)
Claim 6
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[The method of claim 4, wherein analyzing the model activity data further comprises] predicting a cause-and-effect knowledge graph for the unknown issue, (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
[the method comprising] outputting the cause-and-effect knowledge graph as at least part of the model management data. (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
[the method comprising] outputting the cause-and-effect knowledge graph as at least part of the model management data. (The insignificant extra-solution activity is well-understood, routine, conventional as it is merely presenting output. See MPEP 2106.05(d)(II), “Presenting offers”.)
Claim 7
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[wherein analyzing the model activity data comprises] predicting a further issue that can arise in the distributed setup based on the model activity data, (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
[The method of claim 1, wherein] the obtained model activity data relates to an issue reported by one of the models in the distributed setup, (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
[the method comprising] outputting the predicted further issue as at least part of the model management data. (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
[The method of claim 1, wherein] the obtained model activity data relates to an issue reported by one of the models in the distributed setup, (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
[the method comprising] outputting the predicted further issue as at least part of the model management data. (The insignificant extra-solution activity is well-understood, routine, conventional as it is merely presenting output. See MPEP 2106.05(d)(II), “Presenting offers”.)
Claim 8
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
[The method of claim 4, wherein] the reported issue or the predicted issue relates to one or more of model quality, data quality, and operational quality. (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
[The method of claim 4, wherein] the reported issue or the predicted issue relates to one or more of model quality, data quality, and operational quality. (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
Claim 9
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
[The method of claim 1, further comprising] receiving user feedback on the model management data (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
and storing the user feedback in a knowledge database. (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
[The method of claim 1, further comprising] receiving user feedback on the model management data (This insignificant extra-solution activity is well-understood, routine, conventional as it is mere data transfer. See MPEP 2106.05(d)(II), “Receiving or transmitting data over a network” and/or “Storing and retrieving information in memory”.)
and storing the user feedback in a knowledge database. (This insignificant extra-solution activity is well-understood, routine, conventional as it is mere data storage. See MPEP 2106.05(d)(II), “Electronic recordkeeping” and/or “Storing and retrieving information in memory”.)
Claim 10
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[The method of claim 1, wherein analyzing the model activity data comprises] analyzing performance of the machine learning models in the distributed setup. (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
Claim 11
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[The method of claim 10, wherein outputting the model management data comprises] selecting one of the machine learning models for use based on environmental conditions. (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
Claim 12
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[The method of claim 10, further comprising] determining model performance over a particular time period using at least one performance metric, (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
wherein, in the case that a candidate model has a higher performance metric than that determined for a currently-selected model, the method comprises using the candidate model to replace the currently-selected model in response to a difference between the performance metrics for the two models exceeding a predetermined threshold. (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
Claim 13
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[wherein selecting the machine learning model based on environmental conditions comprises] selecting the second machine learning model in response to the analysis of the model activity data indicating a data quality problem concerning the known problematic signal. (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
[The method of claim 10, wherein] the distributed setup comprises at least a first machine learning model trained with a known problematic signal and at least a second machine learning model trained without the known problematic signal, (This recites a general link between an abstract idea and a particular field of use or technological environment. See MPEP 2106.05(h).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
[The method of claim 10, wherein] the distributed setup comprises at least a first machine learning model trained with a known problematic signal and at least a second machine learning model trained without the known problematic signal, (This recites a general link between an abstract idea and a particular field of use or technological environment. See MPEP 2106.05(h).)
Claim 14
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[The method of claim 1, further comprising] utilizing a prediction output from at least one of the models in the distributed setup to control an industrial process or to inform a human about the current state or future state of the industrial process. (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
Claim 15
Step 1. The claim and its dependents 16-20 fall under the statutory category of machines. An analysis of step 2 for each of these claims follows.
Step 2A Prong 1. The claim recites the following abstract ideas:
monitor machine learning models (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
analyzing the obtained model activity data; (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
managing the activity of the machine learning models in the distributed setup. (This recites a mental process that can be performed in the human mind or by a human using pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
A computer-readable medium comprising instructions stored on tangible media that, when executed by a computing system, cause the computing system to: (This generic computing components for performing an abstract idea. See MPEP 2106.05(f)(2).)
in a distributed setup, (This recites a general link between an abstract idea and a particular field of use or technological environment. See MPEP 2106.05(h).)
by: obtaining model activity data relating to activity of the machine learning models in the distributed setup; (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
and based on the analysis of the model activity data, outputting model management data for (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
A computer-readable medium comprising instructions stored on tangible media that, when executed by a computing system, cause the computing system to: (This generic computing components for performing an abstract idea. See MPEP 2106.05(f)(2).)
in a distributed setup, (This recites a general link between an abstract idea and a particular field of use or technological environment. See MPEP 2106.05(h).)
by: obtaining model activity data relating to activity of the machine learning models in the distributed setup; (This insignificant extra-solution activity is well-understood, routine, conventional as it is mere data transfer. See MPEP 2106.05(d)(II), “Receiving or transmitting data over a network” and/or “Storing and retrieving information in memory”.)
and based on the analysis of the model activity data, outputting model management data for (The insignificant extra-solution activity is well-understood, routine, conventional as it is merely presenting output. See MPEP 2106.05(d)(II), “Presenting offers”.)
Claims 16-20 inherit limitations from claim 15 and recite additional limitations which are substantially similar to those recited by claims 2-6, respectively, so they are rejected by the same rationale.
Claim Rejections - 35 USC 102
The following is a quotation of the appropriate paragraphs of 35 USC 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 USC 102(b)(2)(C) for any potential 35 USC 102(a)(2) prior art against the later invention.
Claim(s) 1-3 and 9-17 is/are rejected under 35 USC 102(a)(1) as being anticipated by Thomas FAULHABER et al. (US20190156247A1, published 2019-05-23; hereafter, “Faulhaber”).
Claim 1
Faulhaber discloses:
A computer-implemented method for monitoring machine learning models ([Faulhaber, 0016, 0029, and figure 1]: Faulhaber discloses “methods, apparatus, systems, and non-transitory computer-readable storage media” in which “the performance of machine learning (ML) models can be dynamically evaluated” [Faulhaber, 0016]. These ML models, labeled 118A-118N in [Faulhaber, figure 1 and 0029], map to the “machine learning models” of the claim.) in a distributed setup, the method comprising: ([Faulhaber, 0074, 0117, and figure 8]: Faulhaber describes an “illustrative operating environment” which “includes end user devices 802, a model hosting system 840, a training data store 860, a training metrics data store 865, a container data store 870, a training model data store 875, and a model prediction data store 880” [Faulhaber, 0074 and figure 8]. While this operating environment is already a “distributive setup” as recited by the claim, the examiner notes that Faulhaber also indicates that “the model training system 820 and/or the model hosting system 820 could also operate within a computing environment having a fewer or greater number of devices than are illustrated in Fig. 8” [Faulhaber, 0117].)
obtaining model activity data relating to activity of the machine learning models in the distributed setup; ([Faulhaber, 0034]: Faulhaber discloses “provid[ing] data 136 to the analytics engine 122” where the “data 136 may include, for example, the input data 134…, the individual inference results 142 generated by the ML models 118, etc” [Faulhaber, 0034]. The data provided to the analytics engine maps to the “model activity data” of the claim.)
analyzing the obtained model activity data; ([Faulhaber, 0034]: Faulhaber discloses that the “analytics engine 122 can determine, using such data 136, the quality of the inferences of the ML model(s) 118” [Faulhaber, 0034]. This analysis maps to the “analyzing” step of the claim.)
and based on the analysis of the model activity data, outputting model management data for managing the activity of the machine learning models in the distributed setup. ([Faulhaber, 0034, 0037]: Faulhaber discloses that, “[w]ith such an analysis, the analytics engine 122 can perform any number of operations, including but not limited to updating… how the model selector 1120 selects ML models…, updating… how the result generator 114 generates results…, updating a model training system to cause particular models 118A-118N… to be trained or re-trained, logging such analysis result data in a logging system 128, reporting analysis result data back to client(s) 102, etc” [Faulhaber, 0034; see also, 0037]. Any of these operations map to the “model management data for managing the activity of the machine learning models” of the claim.)
Claim 2
Faulhaber discloses the elements of the parent claim(s). It also discloses:
[The method of claim 1, further comprising] maintaining the obtained model activity data and/or the model management data in one or more knowledge databases. ([Faulhaber, 0034, 0074, 0120-0121, figure 8]: As noted above, Faulhaber discloses a number of data stores [Faulhaber, 0074 and figure 8]. This includes, for example, a “training data store 860 [that] stores training data and/or evaluation data” [Faulhaber, 0120] and a “training metrics data store 865 [that] stores model metrics” [Faulhaber, 0121]; these exemplify the “obtained model activity data” as mapped above. Faulhaber also discloses “logging [the] analysis result data in a logging system 128” [Faulhaber, 0034]; this analysis data is the “model management data” as mapped above. In other words, the data stores of figure 8 and the logging system 128 map to the “one or more knowledge databases” of the claim.)
Claim 3
Faulhaber discloses the elements of the parent claim(s). It also discloses:
[The method of claim 1, further comprising] updating a cache of one or more nodes of the distributed setup with the obtained model activity data and/or the model management data. ([Faulhaber, 0136]: Faulhaber discloses an embodiment where a network “locally cache[s] at least some data… [and] communicate[s] with virtualized data store service 1010 via one or more communications channels to upload new or modified data from a local cache” [Faulhaber, 0136]. In other words, one of the local caches of Faulhaber maps to the “cache” of the claim.)
Claim 9
Faulhaber discloses the elements of the parent claim(s). It also discloses:
[The method of claim 1, further comprising] receiving user feedback on the model management data and storing the user feedback in a knowledge database. ([Faulhaber, 0039, 0074, figure 8]: Faulhaber discloses that the analytic engine can obtain feedback about its results via “a prompt to the user asking whether the results were good” [Faulhaber, 0039]. The response to this prompt maps to the “user feedback” of the claim. Moreover, as noted under the parent claim, Faulhaber describes a number of data stores for storing the data described in the disclosed system [Faulhaber, 0074 and figure 8]. The data store that stores the feedback results maps to the “knowledge database” of the claim.)
Claim 10
Faulhaber discloses the elements of the parent claim(s). It also discloses:
[The method of claim 1, wherein analyzing the model activity data comprises] analyzing performance of the machine learning models in the distributed setup. ([Faulhaber, 0034]: As noted under the parent claim, Faulhaber discloses that the “analytics engine 122 can determine, using such data 136, the quality of the inferences of the ML model(s) 118” [Faulhaber, 0034]. A model’s quality maps to the “performance” of the claim.)
Claim 11
Faulhaber discloses the elements of the parent claim(s). It also discloses:
[The method of claim 10, wherein outputting the model management data comprises] selecting one of the machine learning models for use based on environmental conditions. ([Faulhaber, 0034, 0037]: Faulhaber discloses the results of the analysis can be used to “switch over some or all traffic to a ‘new’ model (e.g., if its performance meets or exceeds some threshold, such as having an accuracy value that is greater than the ‘old’ model's corresponding accuracy value)” [Faulhaber, 0037]. The “new” model of Faulhaber maps to the “one of the machine learning models” recited by the claim.)
Claim 12
Faulhaber discloses the elements of the parent claim(s). It also discloses:
[The method of claim 10, further comprising] determining model performance over a particular time period using at least one performance metric, ([Faulhaber, 0096]: Faulhaber discloses that a “ML model evaluator 828 periodically generates model metrics” [Faulhaber, 0096]. A period of time after which the evaluator generates metrics maps to the “particular time period” of the claim, and the metrics map to the “at least one performance metric” of the claim.)
wherein, in the case that a candidate model has a higher performance metric than that determined for a currently-selected model, the method comprises using the candidate model to replace the currently-selected model in response to a difference between the performance metrics for the two models exceeding a predetermined threshold. ([Faulhaber, 0037]: The examiner notes that this recites a conditional limitation, since the claim does not positively recite the candidate model having higher performance than the currently-selected model or the difference between the performance metrics actually exceeding a predetermined threshold. Nonetheless, Faulhaber discloses “switch[ing] over some or all traffic to a ‘new’ model (e.g., if its performance meets or exceeds some threshold, such as having an accuracy value that is greater than the ‘old’ model's corresponding accuracy value)” [Faulhaber, 0037]. In other words, the “old” model of Faulhaber maps to the “currently-selected model” of the claim, and the “new” model of Faulhaber to the “candidate model” of the claim. Faulhaber gives the example of performing a switch when the new model has greater performance than the old model, i.e., “in the case that [the] candidate model has a higher performance metric than that determined for [the] currently-selected model” as recited by the claim. Moreover, taking the “predetermined threshold” of the claim to be 0, the example given in Faulhaber does in fact have the “difference between the performance metrics for the two models exceeding [the] predetermined threshold” as recited by the claim.)
Claim 13
Faulhaber discloses the elements of the parent claim(s). It also discloses:
[The method of claim 10, wherein] the distributed setup comprises at least a first machine learning model trained with a known problematic signal and at least a second machine learning model trained without the known problematic signal, wherein selecting the machine learning model based on environmental conditions comprises selecting the second machine learning model in response to the analysis of the model activity data indicating a data quality problem concerning the known problematic signal. ([Faulhaber, 0022, 0034, 0037]: As noted above, the results of the analytics engine can be used to determine “how the model selector 1120 selects ML models (e.g., to push more traffic to ‘better’ performing models, to steer traffic away from worse performing models, etc.)” [Faulhaber, 0034] and/or to “switch over some or all traffic to a ‘new’ model (e.g., if its performance meets or exceeds some threshold, such as having an accuracy value that is greater than the ‘old’ model's corresponding accuracy value)” [Faulhaber, 0037]. Faulhaber also indicates that the choice of model “may depend on dynamic factors, such as spiky traffic and/or data distribution drifts” [Faulhaber, 0022]. Dynamic factors such as data distribution drift map to the “known problematic signal” and the “data quality problem” of the claim. A model with worse performance maps to the “first machine learning model” of the claim, and a model with better performance to the “second machine learning model trained” of the claim. Steering traffic away from a worse-performing model towards a better-performing one maps to the step of “selecting the second model in response to the analysis of the model activity” as recited by the claim.)
Claim 14
Faulhaber discloses the elements of the parent claim(s). It also discloses:
[The method of claim 1, further comprising] utilizing a prediction output from at least one of the models in the distributed setup to control an industrial process or to inform a human about the current state or future state of the industrial process. ([Faulhaber, 0029]: Faulhaber discloses that “[m]any different types of ML models (or combinations of models working together as a processing pipeline) may be hosted and/or trained” [Faulhaber, 0029] (giving numerous examples throughout the specification, including language analysis [Faulhaber, 0023], natural language translation [Faulhaber, 0029], sentiment analysis [Faulhaber, 0029], natural language embedding [Faulhaber, 0029], image classification [Faulhaber, 0033], predicting statistical attributes [Faulhaber, 0033], etc). The result of any of the models maps to the “prediction output” of the claim since any of these types of ML models fall under the broadest reasonable interpretation of being “to control an industrial process or to inform a human user about the current state or future state of the industrial process” as recited by the claim. The applicant is invited to consult Janusz as cited below (e.g., “operational data may be… industrial data” [Janusz, 0006]).)
Claim 15
Faulhaber discloses:
A computer-readable medium comprising instructions stored on tangible media that, when executed by a computing system, cause the computing system to: ([Faulhaber, 0069]: Faulhaber discloses that the system disclosed therein is “performed under the control of one or more computer systems configured with executable instructions and are implemented as code… The code is stored on a computer-readable storage medium” [Faulhaber, 0069].)
monitor machine learning models ([Faulhaber, 0016, 0029, and figure 1]: Faulhaber discloses “methods, apparatus, systems, and non-transitory computer-readable storage media” in which “the performance of machine learning (ML) models can be dynamically evaluated” [Faulhaber, 0016]. These ML models, labeled 118A-118N in [Faulhaber, figure 1 and 0029], map to the “machine learning models” of the claim.) in a distributed setup by: ([Faulhaber, 0074, 0117, and figure 8]: Faulhaber describes an “illustrative operating environment” which “includes end user devices 802, a model hosting system 840, a training data store 860, a training metrics data store 865, a container data store 870, a training model data store 875, and a model prediction data store 880” [Faulhaber, 0074 and figure 8]. While this operating environment is already a “distributive setup” as recited by the claim, the examiner notes that Faulhaber also indicates that “the model training system 820 and/or the model hosting system 820 could also operate within a computing environment having a fewer or greater number of devices than are illustrated in Fig. 8” [Faulhaber, 0117].)
obtaining model activity data relating to activity of the machine learning models in the distributed setup; ([Faulhaber, 0034]: Faulhaber discloses “provid[ing] data 136 to the analytics engine 122” where the “data 136 may include, for example, the input data 134…, the individual inference results 142 generated by the ML models 118, etc” [Faulhaber, 0034]. The data provided to the analytics engine maps to the “model activity data” of the claim.)
analyzing the obtained model activity data; ([Faulhaber, 0034]: Faulhaber discloses that the “analytics engine 122 can determine, using such data 136, the quality of the inferences of the ML model(s) 118” [Faulhaber, 0034]. This analysis maps to the “analyzing” step of the claim.)
and based on the analysis of the model activity data, outputting model management data for managing the activity of the machine learning models in the distributed setup. ([Faulhaber, 0034, 0037]: Faulhaber discloses that, “[w]ith such an analysis, the analytics engine 122 can perform any number of operations, including but not limited to updating… how the model selector 1120 selects ML models…, updating… how the result generator 114 generates results…, updating a model training system to cause particular models 118A-118N… to be trained or re-trained, logging such analysis result data in a logging system 128, reporting analysis result data back to client(s) 102, etc” [Faulhaber, 0034; see also, 0037]. Any of these operations map to the “model management data for managing the activity of the machine learning models” of the claim.)
Claims 16-17 inherit limitations from claim 15 and recite additional limitations which are substantially similar to those recited by claim 2-3, respectively, so they are rejected by the same rationale.
Claim Rejections - 35 USC 103
The following is a quotation of 35 USC 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4-5, 7-8, and 18-19 is/are rejected under 35 USC 103 as being unpatentable over Faulhaber, further in view of Andrzej JANUSZ et al. (US20250005433A1, filed 2023-06-30; hereafter, “Janusz”).
Claim 4
Faulhaber discloses the elements of the parent claim(s). It also discloses:
[The method of claim 1, wherein] the obtained model activity data relates to an issue reported by one of the models in the distributed setup, ([Faulhaber, 0034]: The “obtained model activity data” as mapped above “may include, for example, the input data 134…, the individual inference results 142 generated by the ML models 118, etc” and is used by the analytics engine to analyze model performance [Faulhaber, 0034]. The results of the analytics engine map to the “issue reported by one of the [machine learning] models” of the claim, and the data used by the analytics engine falls under the broadest reasonable interpretation of “relat[ing] to [the] issue” as recited by the claim.)
Faulhaber might not distinctly disclose:
wherein the reported issue is an unknown issue, and wherein analyzing the model activity data comprises generating a label for the unknown issue, the method comprising outputting the label as at least part of the model management data.
Janusz is in the field of machine learning. It Janusz discloses a system for “explain[ing] a cause of a mistake in a machine learning model” [Janusz, 0006] which first determines whether a mistake occurred and then provides a cause for the mistake [Janusz, 0007]. The determination that a mistake occurs corresponds to the results of the analytics engine of Faulhaber, i.e., to the “issue” of the claim as mapped above. Moreover, Faulhaber in view of Janusz discloses:
wherein the reported issue is an unknown issue, ([Janusz, 0007, 0009]: As noted above, Janusz first determines that a mistake occurred, and then uses a diagnostic model to provides a cause for the mistake [Janusz, 0007]. The determination that a mistake occurred (before a cause has been assigned to the mistake) maps to the “unknown issue” of the claim.) and wherein analyzing the model activity data comprises generating a label for the unknown issue, the method comprising outputting the label as at least part of the model management data. ([Janusz, 0009-0011]: Janusz gives numerous examples of causes that might be provided for a particular mistake [Janusz, 0009-0011]. For example, a mistake might result “because a labeling of a historical training data set on which the evaluated model was formed was erroneous” or “because an external condition changed which caused the predictive data produced by the evaluated model to no longer conform to predictive trends”, etc [Janusz, 0009]. Any of these causes maps to the “label for the unknown issue” of the claim. In the combination, the diagnostics model of Janusz is part of the analytics engine of Faulhaber, so that the cause provided by the diagnostics model (i.e., the “label” of the claim) is part of the “model management data” as mapped under the parent claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the machine learning monitoring system of Faulhaber with the diagnostics system of Janusz because it allows for “explain[ing] a cause of the mistake” [Janusz, 0006], thereby resulting in a more transparent and useable system.
Claim 5
Faulhaber in view of Janusz discloses the elements of the parent claim(s). It also discloses:
[The method of claim 4, wherein analyzing the model activity data further comprises] identifying at least one known issue which is similar to the unknown issue, the method comprising outputting the at least one known similar issue as at least part of the model management data. ([Janusz, 0014]: Janusz discloses “generate a set of historical neighborhoods comprising a set of historical instances that were processed in a similar way to the current instance on which mistakes of the diagnosed machine learning model may be observable” [Janusz, 0014]. The “current instance on which mistakes of the diagnosed machine learning model may be observable” correspond to the “unknown issue” as mapped above, so the “historical instances that were processed in a similar way to the current instance” map to the “at least one known issue” of the claim (since they are “similar to the unknown issue”, as required by the claim).)
The same motivation to combine applies.
Claim 8
Faulhaber in view of Janusz discloses the elements of the parent claim(s). It also discloses:
[The method of claim 4, wherein] the reported issue or the predicted issue relates to one or more of model quality, data quality, and operational quality. ([Faulhaber, 0034]: As noted above, Faulhaber discloses that the “analytics engine 122 can determine, using such data 136, the quality of the inferences of the ML model(s) 118” [Faulhaber, 0034]. In other words, the “reported issue” as mapped above falls under the broadest reasonable interpretation of “relat[ing] to one or more of model quality, data quality, and operational quality” as recited by the claim. The applicant is also invited to consult [Janusz, 0009-0011].)
The same motivation to combine applies.
Claim 7
Faulhaber discloses the elements of the parent claim(s). It also discloses:
[The method of claim 1, wherein] the obtained model activity data relates to an issue reported by one of the models in the distributed setup, ([Faulhaber, 0034]: The “obtained model activity data” as mapped above “may include, for example, the input data 134…, the individual inference results 142 generated by the ML models 118, etc” [Faulhaber, 0034]. Such data falls under the broadest reasonable interpretation of “relat[ing] to an issue” as recited by the claim.)
Faulhaber might not distinctly disclose:
wherein analyzing the model activity data comprises predicting a further issue that can arise in the distributed setup based on the model activity data, the method comprising outputting the predicted further issue as at least part of the model management data.
Janusz is in the field of machine learning. It Janusz discloses a system for “explain[ing] a cause of a mistake in a machine learning model” [Janusz, 0006] which first determines whether a mistake occurred and then provides a cause for the mistake [Janusz, 0007]. The determination that a mistake occurs corresponds to the results of the analytics engine of Faulhaber, i.e., to the “issue” of the claim as mapped above. Moreover, Faulhaber in view of Janusz discloses:
wherein analyzing the model activity data comprises predicting a further issue that can arise in the distributed setup based on the model activity data, the method comprising outputting the predicted further issue as at least part of the model management data. ([Janusz, 0009-0011]: Janusz gives numerous examples of causes that might be provided for a particular mistake [Janusz, 0009-0011]. This includes numerous examples that are indicative of “a further issue that can arise” as recited by the claim. For example, a mistake might be due to “concept drift in a relationship between an input data and the predictive data caused because a property of a target variable has changed over time” [Janusz, 0010], and the presence of concept drift would indicate that the same type of mistake will continue to arise. Similarly, a mistake might be “because the evaluated model may be underfitted” [Janusz, 0010], and a model being underfitted will cause it to produce further mistakes. In the combination, the diagnostics model of Janusz is part of the analytics engine of Faulhaber, so that the cause provided by the diagnostics model (i.e., the “label” of the claim) is part of the “model management data” as mapped under the parent claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the machine learning monitoring system of Faulhaber with the diagnostics system of Janusz because it allows for “explain[ing] a cause of the mistake” [Janusz, 0006], thereby resulting in a more transparent and useable system.
Claims 18-19 inherit limitations from claim 15 and recite additional limitations which are substantially similar to those recited by claim 4-5, respectively, so they are rejected by the same rationale.
Claim(s) 6 and 20 is/are rejected under 35 USC 103 as being unpatentable over Faulhaber in view of Janusz, further in view of Jean FENG et al. (Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare, published 2022-05-31; hereafter, “Feng”).
Claim 6
Faulhaber in view of Janusz discloses the elements of the parent claim(s). While it discloses identifying causes for mistakes [Janusz, 0006] as well as generating visual reports that “include an interactive plot to help to explore diagnoses for individual instances and analyze their statistics for specific groups” [Janusz, 0059], it may be argued that it does not distinctly disclose a cause-and-effect knowledge graph. In other words, Faulhaber in view of Janusz might not distinctly disclose:
[The method of claim 4, wherein analyzing the model activity data further comprises] predicting a cause-and-effect knowledge graph for the unknown issue, the method comprising outputting the cause-and-effect knowledge graph as at least part of the model management data.
Feng is in the field of machine learning. It discloses a system for “continual monitoring and updating of AI algorithms” [Feng, title]. Moreover, Faulhaber in view of Janusz and Feng discloses:
[The method of claim 4, wherein analyzing the model activity data further comprises] predicting a cause-and-effect knowledge graph for the unknown issue, the method comprising outputting the cause-and-effect knowledge graph as at least part of the model management data. ([Feng, page 2 section titled “Cause-and-effect diagrams” and figure 2]: Feng discloses that drops in system performance can be explained using a “cause-and-effect diagram—also known as the fishbone diagram or Ishikawa diagram” [Feng, page 2 section titled “Cause-and-effect diagrams]. See [Feng, figure 2] for an example. These cause-and-effect diagrams map to the “cause-and-effect knowledge graph” of the claim. In the combination, such diagrams are part of the visual report of Janusz, i.e., they are determined by the analytics engine of Faulhaber in view of Janusz so that they are part of the “model management data” of the claim as mapped under the parent claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the machine learning monitoring system of Faulhaber in view of Janusz with cause-and-effect diagrams as disclosed in Feng because they “can help unlayer the potential causes” [Feng, page 2 section titled “Cause-and-effect diagrams”], thereby resulting in a more transparent and useable system.
Claim 20 inherits limitations from claim 15 and recites additional limitations which are substantially similar to those recited by claim 6, so it is rejected by the same rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Aditya DHANEKULA et al. (A Comparative Analysis of Monitoring and Observability Tools for Machine Learning and Data Science Pipelines, published 2022-09-19) describes numerous machine learning monitoring and observability tools, including aspects such as: detecting anomalies and errors, checking model performance, checking data quality, performing root cause analysis [Dhanekula, figure 1].
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Shishir AGRAWAL whose telephone number is +1 703-756-1183. The examiner can normally be reached Monday through Thursday, 08:30-14:30 Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey SHMATOV can be reached on +1 571-270-3428. The fax phone number for the organization where this application or proceeding is assigned is +1 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at +1 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call +1 800-786-9199 (IN USA OR CANADA) or +1 571-272-1000.
/S.A./Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights