Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
2. This Office Action is sent in response to Applicant’s Communication received on 07/29/2026 for application number 17/792,583.
Response to Amendments
3. The Amendment filed 07/29/2026 has been entered. Claims 1, 2, and 20 have been amended. Claims 1-21 remain pending in the application.
Response to Arguments
Applicant argues that Chatterjee in view of Merrill fails to disclose the amended independent claim 1. However, the argument is moot since this is a newly presented limitation, thus changing the scope of the claim. However, a new combination of references is applied.
Claim Rejections – 35 USC § 103
4. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5. Claims 1-3 and 17-21 are rejected under 35 U.S.C. 103 as being unpatentable over Chatterjee et al. (U.S. Patent Application Pub. No. US 20210049512 A1) in view of Gupta et al. (U.S. Patent Application Pub. No. US 20190147369 A1).
Claim 1: Chatterjee teaches an information processing apparatus, comprising:
circuitry configured to (i.e. processor; para. [0072])
control (i.e. A trained classifier 132 may be obtained using the training data 110 and the selected algorithm; para. [0026]) a predetermined learning model to perform learning (i.e. a client of a machine learning service may submit a request to generate a classification model for a specified data set or data source (a source from which various observation records may be collected by the service and used for training the requested model); para. [0018, 0019, 0035]),
control a conversion model to perform learning (i.e. Such transformations, which may include for example binarization of some or all categorical attribute values and/or binning of some or all numeric attribute values, may be performed to simplify the task of mining rules corresponding to the classifier's predictions; para. [0019, 0065]), the conversion model converting an output of the predetermined learning model into a rule group (i.e. A number of different explainer algorithms (which may also be referred to as rule mining algorithms) may be available in library 125 in the depicted embodiment, from which a particular algorithm may be chosen by explainer selector 160 to generate attribute-predicate based rules in the depicted embodiment … A set of explanatory rules or assertions may be generated by the selected rule miner from the intermediate data set. The rules may be expressed in an easy-to-understand format in which the conditions for a given rule are formulated in terms of predicates on attribute ranges or values, and the implication part of the given rule indicates the class or output variable value produced by the classifier for an observation record whose attributes match or satisfy the predicates; para. [0028, 0066]) output in a format that can be interpreted by a user (i.e. At the end of the rule generation and ranking phase, a ranked explanatory ruleset 165 may be available at explainer 162. In some embodiments in which the rules are originally expressed in terms of transformed attributes rather than the original attributes of the data, the explainer 162 may also include one or more reverse feature transformer components 166. The reverse feature transformers 166 may reformulate the rules in terms of the original input data attributes, which may make the rules easier to understand for clients; para. [0029, 0066]) using a user interface (i.e. In their programmatic interactions with the MLS via interfaces 261, clients 264 may indicate a number of preferences or requirements which may be taken into account when training classifiers and/or explaining classifier predictions in various embodiments. For example, clients may indicate a classification algorithm, a rule mining algorithm, sizes of training sets and/or test sets, relative weights that are to be assigned to various attributes with respect to classification, and so on. Some clients 264 may simply indicate a source of the observation records and leave the modeling and explanation-related decisions to the MLS; other clients, who are more conversant with the statistics involved or who are experts in the subject matter or domain for which the stream records are collected, may provide more detailed guidance or preferences with regard to the classification or explanation stages; para. [0037, 0039]),
acquire evaluation information obtained by evaluating the rule group in accordance with a predetermined standard (i.e. in which multiple explanatory rules are generated, they may then be ranked relative to one another using any combination of a variety of metrics and techniques in different embodiments. For example, rules may be ranked based on their explanatory accuracy or confidence level (e.g., how often their implication between the attribute predicates and the target class holds true, within the training data and/or a test data set comprising observations which were not used for training the classifier), their support or coverage (e.g., what fraction of observations meet the predicates indicates in a given rule), and so on. In various embodiments, the ranking process may involve the use of test or evaluation data sets—e.g., the predictions for a group of observation records which were not used for training may be obtained from the classifier, and the explanatory rules may be ranked based on how well they are able to explain these post-training predictions (e.g., predictions on test data). In some embodiments, the ranking procedure may utilize at least some of the training data and the corresponding classifier predictions; para. [0020, 0051-0054]); and
adjust learning processing of the predetermined learning model based on the evaluation information (i.e. the explainer 162 may provide an indication to the client 180 that no satisfactory explanatory rule was found. The explainer may keep track of the number of cases in which an explanatory rule could not be found in some embodiments. If the number or fraction of such unsatisfactory responses exceeds a threshold, additional rules may be mined and/or the classifier may be retrained with additional records in one such embodiment; para. [0022, 0032]),
wherein the evaluation information indicates at least one of a rate of correct solutions or a degree of accuracy with respect to each output rule of the rule group (i.e. rules may be ranked based on their explanatory accuracy or confidence level (e.g., how often their implication between the attribute predicates and the target class holds true, within the training data and/or a test data set comprising observations which were not used for training the classifier), their support or coverage (e.g., what fraction of observations meet the predicates indicates in a given rule), and so on; para. [0020, 0028, 0052, 0054]).
Chatterjee does not explicitly teach a predetermined standard input with respect to the rule group output using the user interface, control display of the evaluation information on a screen of the user interface.
However, Gupta teaches acquire evaluation information obtained by evaluating the rule group (i.e. the term “fitness function” refers to an objective function for evaluating a solution produced by a genetic algorithm in terms of one or more factors, such as precision and coverage of the solution. As used herein, “precision” refers to a correctness of a rule or set of rules, e.g., whether the rule or set correctly predict the observed outputs. The term “coverage” refers to an amount of the entire body of input data to which a rule or set of rules apply … The precision measure of a rule Ri is indicative of correctness of the rule over an entirety of the training user data 208; para. [0022, 0050, 0051]) in accordance with a predetermined standard input with respect to the rule group output using the user interface (i.e. The condition combination module 218 may select a subset of rules in relation to each of the observed outputs based on different selection techniques. The selection technique used to select these subsets may be based, in part, on user input indicating a preference for one of different optimization bases. Alternately or in addition, the condition combination module 218 may use a particular one of the selection techniques based on configuration settings. By way of example, the different selection techniques may select rules to optimize coverage of the selected rules, optimize the above-noted F1-score of the selected rules, optimize a second-level genetic; para. [0068, 0069, 0077]), control display of the evaluation information on a screen of the user interface (i.e. FIG. 3 depicts an example user interface 300 that is configured to present determined rules for a black-box machine-learning model. The illustrated example 300 includes rule-presentation interface 302, which is depicted as a displayable dialog box; para. [0077-0078]), and adjust learning processing of the predetermined learning model based on the evaluation information (i.e. These rules allow a developer of such a model generally to determine from the rules why the model is producing a certain output when given a certain input. In scenarios where the output produced by the model is not the desired behavior, the developer can adjust data used to train the model (or add training data corresponding to the desired behavior), and retrain the model so that it behaves in the desired manner; para. [0019]), wherein the evaluation information indicates at least one of a rate of correct solutions or a degree of accuracy with respect to each output rule of the rule group (i.e. the condition combination module 218 is configured to compute a precision measure and coverage measure for each generated rule. The precision measure of a rule Ri is indicative of correctness of the rule over an entirety of the training user data 208; para. [0050, 0051, 0061]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Chatterjee to include the feature of Gupta. One would have been motivated to make this modification because it enables users to assess the quality of generated rules, identify undesirable model behavior, and improve the underlying classifier through retraining, thereby improving model transparency, debugging, and model performance.
Claim 2: Chatterjee and Gupta teach the information processing apparatus according to claim 1. Chatterjee further teaches wherein the predetermined standard includes a standard defined by the user (i.e. In their programmatic interactions with the MLS via interfaces 261, clients 264 may indicate a number of preferences or requirements which may be taken into account when training classifiers and/or explaining classifier predictions in various embodiments. For example, clients may indicate a classification algorithm, a rule mining algorithm, sizes of training sets and/or test sets, relative weights that are to be assigned to various attributes with respect to classification, and so on; para. [0039, 0066]).
Gupta further teaches wherein the predetermined standard includes a standard defined by the user using the user interface (i.e. The selection technique used to select these subsets may be based, in part, on user input indicating a preference for one of different optimization bases. Alternately or in addition, the condition combination module 218 may use a particular one of the selection techniques based on configuration settings. By way of example, the different selection techniques may select rules to optimize coverage of the selected rules, optimize the above-noted F1-score of the selected rules, optimize a second-level genetic algorithm with the selected rules, and so forth; para. [0068, 0094]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Chatterjee to include the feature of Gupta. One would have been motivated to make this modification because it enables users to assess the quality of generated rules, identify undesirable model behavior, and improve the underlying classifier through retraining, thereby improving model transparency, debugging, and model performance.
Claim 3: Chatterjee and Gupta teach the information processing apparatus according to claim 1. Chatterjee further teaches wherein the learning model includes a prediction model for predicting a target item (i.e. a client of a machine learning service may submit a request to generate a classification model for a specified data set or data source (a source from which various observation records may be collected by the service and used for training the requested model). Each observation record may contain one or more input variables and at least one output or “target” variable (the variable for which the model is make predictions); para. [0018, 0019]).
Claim 17: Chatterjee and Gupta teach the information processing apparatus according to claim 1. Chatterjee further teaches wherein the circuitry controls the conversion model to perform learning (i.e. A number of different explainer algorithms (which may also be referred to as rule mining algorithms) may be available in library 125 in the depicted embodiment, from which a particular algorithm may be chosen by explainer selector 160 to generate attribute-predicate based rules in the depicted embodiment; para. [0028]), the conversion model conforming to the predetermined standard (i.e. in which multiple explanatory rules are generated, they may then be ranked relative to one another using any combination of a variety of metrics and techniques in different embodiments. For example, rules may be ranked based on their explanatory accuracy or confidence level (e.g., how often their implication between the attribute predicates and the target class holds true, within the training data and/or a test data set comprising observations which were not used for training the classifier), their support or coverage (e.g., what fraction of observations meet the predicates indicates in a given rule), and so on. In various embodiments, the ranking process may involve the use of test or evaluation data sets—e.g., the predictions for a group of observation records which were not used for training may be obtained from the classifier, and the explanatory rules may be ranked based on how well they are able to explain these post-training predictions (e.g., predictions on test data). In some embodiments, the ranking procedure may utilize at least some of the training data and the corresponding classifier predictions; para. [0020, 0051-0054]).
Claim 18: Chatterjee and Gupta teach the information processing apparatus according to claim 1. Chatterjee further teaches wherein the conversion model includes a learning model using at least one algorithm of a decision tree or a rule fit (i.e. a CART (Classification and Regression Tree) algorithm, an ID3 (Iterative Dichotomizer 3) algorithm, a C4.5 algorithm or a C5.0 algorithm; para. [0025, 0034]).
Claim 19 is similar in scope to Claim 1 and is rejected under a similar rationale. Chatterjee further teaches an information processing method, executed by a computer system (i.e. computer system; para. [0071]).
Claim 20 is similar in scope to Claim 1 and is rejected under a similar rationale. Chatterjee further teaches a non-transitory computer-readable storage medium having embodied thereon a program (i.e. non-transitory computer-accessible storage medium; para. [0076]), which when executed by a computer system causes the computer system to execute a method (i.e. computer system; para. [0071]).
Claim 21: Chatterjee and Gupta teach the information processing apparatus according to claim 1. Chatterjee further teaches wherein the circuitry is further configured to control output (i.e. the explainer may provide a representation of the highest-ranking rule to the client; para. [0021, 0067]) of the evaluation information (i.e. rules may be ranked based on their explanatory accuracy or confidence level (e.g., how often their implication between the attribute predicates and the target class holds true, within the training data and/or a test data set comprising observations which were not used for training the classifier), their support or coverage (e.g., what fraction of observations meet the predicates indicates in a given rule), and so on; para. [0020]) using the user interface (i.e. The interfaces may include, for example, one or more web-based consoles or web pages, application programming interfaces (APIs), command-line tools, graphical user interfaces (GUIs) or the like. Using interfaces 261, clients 264 may, for example, submit a request to train a classification model using records from an input data source 230, or a request to explain a prediction of a classification mode; para. [0037]).
Gupta further teaches wherein the circuitry is further configured to control output of the evaluation information using the user interface (i.e. the second-ranked rule 306 is depicted with ranking 310 and fitness score 312. The ranking 310 is indicative of the rule's position in the list of sorted determined black-box rules b 220 e.g., that are sorted as described above … rankings, fitness scores, and properties may be presented in connection with respective rules in accordance with one or more implementations; para. [0077-0082]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Chatterjee to include the feature of Gupta. One would have been motivated to make this modification because it enables users to assess the quality of generated rules, identify undesirable model behavior, and improve the underlying classifier through retraining, thereby improving model transparency, debugging, and model performance.
6. Claims 4, 11, and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Chatterjee in view of Gupta, and further in view of Kuwajima et al. (U.S. Patent Pub. No. US 11625561 B2).
Claim 4: Chatterjee and Gupta teach the information processing apparatus according to claim 3. Chatterjee further teaches wherein the rule group includes at least one output rule describing an output of the prediction model (i.e. the rule mining algorithm may be selected by the service (e.g., based on knowledge base entries), while in at least one embodiment the client may suggest or propose a rule mining algorithm if desired. Some number of explanatory rules or assertions may be generated for the predictions already made with respect to the training data set by the classifier. A given explanatory rule or assertion may be considered as a combination of one or more attribute predicates (e.g., ranges or exact matches of input attribute values) and implied target class values. For example, one rule may be expressed as the logical equivalent of “if (input attribute A1 is in range R1) and (input attribute A2 has the value V1), then the target class is predicted to be TC1”. In this rule, the constraints on A1 and A2 respectively represent two attribute predicates, and the rule indicates an explanatory relationship between the predicates and the prediction regarding the target class; para. [0019, 0028, 0050]).
Chatterjee does not explicitly teach to generate at least one of an explanatory sentence or a chart relating to each of the output rules.
However, Kuwajima teaches wherein the circuitry is further configured to generate at least one of an explanatory sentence or a chart relating to each of the output rules (i.e. fig. 5, the explanation information generation unit 15 reads out the concept data corresponding to the identification number of the ground feature from the annotation DB 17, and generates a sentence that explains the basis of the inference result using the read-out concept data. For example, when the concept data “sky”, “grass field”, and “small animal” are read out from the ground features of the identification numbers “#1”, “#4”, and “#9”, the explanation sentence is generated such as “this image is X because the image includes “sky”, “grass field” and “small animal.”; col. 7, lines 5-30).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Chatterjee and Gupta to include the feature of Kuwajima. One would have been motivated to make this modification because by displaying the explanation information generated based on the concept data corresponding to the ground feature together with the class as the inference result, the user can easily grasp the ground that leads to the inference result.
Claim 11: Chatterjee, Gupta, and Kuwajima teach the information processing apparatus according to claim 4. Chatterjee further teaches wherein the evaluation information includes information relating to a violation rule that is the output rule failing to satisfy the predetermined standard (i.e. A number of different explainer algorithms (which may also be referred to as rule mining algorithms) may be available in library 125 in the depicted embodiment, from which a particular algorithm may be chosen by explainer selector 160 to generate attribute-predicate based rules in the depicted embodiment. Each rule of the explainer 162 may indicate some set of conditions or predicates based on attribute values in the transformed or untransformed versions of the training set observation records, and an indication of the target variable value predicted by the classifier if the set of conditions is met. A number of rules with different levels of specificity may be generated. The rules may be ranked relative to one another, using various metrics such as support (e.g., the fraction of the training data which meets the attribute criteria of the rule), confidence or accuracy (e.g., the fraction of the rule's predictions which match the classifier's predictions, etc.), precision, recall, etc. in different embodiments and for different types of classification problems. For example, different ranking metrics may be used for binary classification than for multi-class classification; para. [0020, 0028]), and wherein the circuitry adjusts at least one of learning data of the prediction model, or a learning parameter of the prediction model with reference to a data range specified by the violation rule (i.e. If no matching rules are found (as also detected in operations corresponding to element 919), a message indicating that an explanation is currently unavailable may be provided in response to the request regarding Pred-j (element 925). In at least some embodiments, metrics regarding the kind of responses (e.g., a satisfactory explanation versus a “no explanation is available” message) provided to explanation requests may be updated. If the metrics regarding unsatisfactory responses reach a threshold, in such embodiments this may trigger the re-generation of additional rules (e.g., using a larger input set for the rule miner, including at least some observations for which no explanations were available) and/or re-training of the classifier; para. [0022, 0068]).
Claim 13: Chatterjee, Gupta, and Kuwajima teach the information processing apparatus according to claim 11. Chatterjee further teaches wherein the circuitry adjusts the learning parameter, and wherein the learning parameter includes at least one of a parameter for adjusting the output of the prediction model relating to the learning data or a parameter for adjusting a loss function of the prediction model (i.e. The weights may be adjusted over time as additional information becomes available, and an activation function may be applied to the inputs at a given node to convert the weighted inputs to the outputs. Each internal or hidden node may represent or summarize some combination of properties of one or more input layer nodes; para. [0055]).
7. Claims 5-6 and 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over Chatterjee in view of Gupta, Kuwajima, and further in view of Dalli et al. (U.S. Patent Application Pub. No. US 20210256377 A1).
Claim 5: Chatterjee, Gupta, and Kuwajima teach the information processing apparatus according to claim 4. Chatterjee does not explicitly teach wherein the circuitry is further configured to generate a check item for causing the user to check whether or not each of the output rules satisfies the predetermined standard.
However, Dalli teaches wherein the circuitry is further configured to generate a check item for causing the user to check whether or not each of the output rules satisfies the predetermined standard (i.e. fig. 11, A control and quality check 2030 may be applied such that the level of importance attributed to the final result is allocated in a manner which is fair and which does not cause unnecessary bias. The result of the control and quality check may be validated through the control node 2040, which determines if an exception should be triggered, and which then may seek human verification 2050 based on a result of said determination. In an exemplary embodiment, the control and quality check 2030 may entail applying one or more rules or conditions, and determining the validity of the feature attribution generated from the XAI/XNN model based on the satisfaction (or lack thereof) of said rules or conditions. (It may be contemplated for the one or more rules or conditions to be externally derived rules or conditions, such as rules or conditions which have been derived from legal conditions, institutional policy, and so forth, but further rules and conditions may also be contemplated, such as rules and conditions derived from interpretation of human verification 2050.); para. [0176]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Chatterjee, Gupta, and Kuwajima to include the feature of Dalli. One would have been motivated to make this modification because it ensures that rule-based outputs adhere to industry guidelines or ethical requirements.
Claim 6: Chatterjee, Gupta, Kuwajima, and Dalli teach the information processing apparatus according to claim 5. Chatterjee does not explicitly teach wherein the circuitry is further configured to read, as the evaluation information, a check result of the check item by the user.
However, Dalli further teaches wherein the circuitry is further configured to read, as the evaluation information, a check result of the check item by the user (i.e. fig. 11, the workflow may proceed normally and generate analysis of the partitions which had been triggered as shown in step 2060, along with visualization of the summarized feature attributions 2070. The final analysis and visualizations are then sent to a user interface or other output via the output node 2080; para. [0177]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Chatterjee, Gupta, and Kuwajima to include the feature of Dalli. One would have been motivated to make this modification because it ensures that rule-based outputs adhere to industry guidelines or ethical requirements.
Claim 8: Chatterjee, Gupta, and Kuwajima teach the information processing apparatus according to claim 4. Chatterjee does not explicitly teach store a database relating to the predetermined standard, wherein the circuitry is further configured to determine whether or not each output rule satisfies the predetermined standard based on the database.
However, Dalli teaches a non-transitory computer-readable storage medium to store a database relating to the predetermined standard (i.e. It may be further contemplated that a process definition may contain an interaction with a human, either in an interactive (synchronous) or non-interactive (asynchronous) manner. Such processes are referred to as Human in the Loop (HIL) processes. In an exemplary embodiment, an HIL process may be utilized to approve or reject a recommended decision being proposed by an explainable system after offering a combination of Answer, Explanation and/or Justification to the human user. In a further exemplary embodiment, a rule variable, or workflow variable or other suitable data storage may be utilized to keep track of a sequence of interactions with a human user, allowing interactive dialogue and interactive sessions to be implemented; para. [0091, 0161]), wherein the circuitry is further configured to determine whether or not each output rule satisfies the predetermined standard based on the database (i.e. fig. 11, A control and quality check 2030 may be applied such that the level of importance attributed to the final result is allocated in a manner which is fair and which does not cause unnecessary bias. The result of the control and quality check may be validated through the control node 2040, which determines if an exception should be triggered, and which then may seek human verification 2050 based on a result of said determination. In an exemplary embodiment, the control and quality check 2030 may entail applying one or more rules or conditions, and determining the validity of the feature attribution generated from the XAI/XNN model based on the satisfaction (or lack thereof) of said rules or conditions. (It may be contemplated for the one or more rules or conditions to be externally derived rules or conditions, such as rules or conditions which have been derived from legal conditions, institutional policy, and so forth, but further rules and conditions may also be contemplated, such as rules and conditions derived from interpretation of human verification 2050.); para. [0176, 0177]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Chatterjee, Gupta, and Kuwajima to include the feature of Dalli. One would have been motivated to make this modification because it ensures that rule-based outputs adhere to industry guidelines or ethical requirements.
Claim 9: Chatterjee, Gupta, Kuwajima, and Dalli teach the information processing apparatus according to claim 8. Chatterjee does not explicitly teach wherein the circuitry is further configured to generate a check item for each output rule determined as failing to satisfy the predetermined standard, the check item causing the user to check whether or not the output rule satisfies the predetermined standard.
However, Dalli further teaches wherein the circuitry is further configured to generate a check item for each output rule determined as failing to satisfy the predetermined standard, the check item causing the user to check whether or not the output rule satisfies the predetermined standard (i.e. fig. 11, A control and quality check 2030 may be applied such that the level of importance attributed to the final result is allocated in a manner which is fair and which does not cause unnecessary bias. The result of the control and quality check may be validated through the control node 2040, which determines if an exception should be triggered, and which then may seek human verification 2050 based on a result of said determination. In an exemplary embodiment, the control and quality check 2030 may entail applying one or more rules or conditions, and determining the validity of the feature attribution generated from the XAI/XNN model based on the satisfaction (or lack thereof) of said rules or conditions. (It may be contemplated for the one or more rules or conditions to be externally derived rules or conditions, such as rules or conditions which have been derived from legal conditions, institutional policy, and so forth, but further rules and conditions may also be contemplated, such as rules and conditions derived from interpretation of human verification 2050.); para. [0176, 0177]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Chatterjee, Gupta, and Kuwajima to include the feature of Dalli. One would have been motivated to make this modification because it ensures that rule-based outputs adhere to industry guidelines or ethical requirements.
Claim 10: Chatterjee, Gupta, Kuwajima, and Dalli teach the information processing apparatus according to claim 8. Chatterjee further teaches wherein the circuitry generates, as the evaluation information, information relating to the output rule determined as failing to satisfy the predetermined standard (i.e. with respect to rule R1, four out of the ten records Rec0-Rec9 meet the predicate conditions (that the country be UK and the year of birth be earlier than 1985), so the support metric is computed as 4/10 or 40%. Among the four records which satisfy the predicates of rule R1, three match the implication of rule R1, so the confidence is set to ¾ or 75%. Similarly, a support metric value of 3/10 or 30% is calculated for rule R2, and a confidence level of 100% is computed for R2. The support and confidence levels may be combined to derive an overall ranking metric for the rules in some embodiments—e.g., a formula in which the support level is simply added to the confidence level to obtain the overall ranking metric, or in which a weighted sum of the two metrics is used as the overall ranking metric, may be used in different embodiments; para. [0020, 0021, 0051-0054]).
However, Dalli further teaches wherein the circuitry generates, as the evaluation information, information relating to the output rule determined as failing to satisfy the predetermined standard (i.e. fig. 11, A control and quality check 2030 may be applied such that the level of importance attributed to the final result is allocated in a manner which is fair and which does not cause unnecessary bias. The result of the control and quality check may be validated through the control node 2040, which determines if an exception should be triggered, and which then may seek human verification 2050 based on a result of said determination. In an exemplary embodiment, the control and quality check 2030 may entail applying one or more rules or conditions, and determining the validity of the feature attribution generated from the XAI/XNN model based on the satisfaction (or lack thereof) of said rules or conditions. (It may be contemplated for the one or more rules or conditions to be externally derived rules or conditions, such as rules or conditions which have been derived from legal conditions, institutional policy, and so forth, but further rules and conditions may also be contemplated, such as rules and conditions derived from interpretation of human verification 2050.); para. [0176, 0177]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Chatterjee, Gupta, and Kuwajima to include the feature of Dalli. One would have been motivated to make this modification because it ensures that rule-based outputs adhere to industry guidelines or ethical requirements.
8. Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Chatterjee in view of Kuwajima, Gupta, Dalli, and further in view of Kaduwela et al. (U.S. Patent Application Pub. No. US 20170212732 A1).
Claim 7: Chatterjee, Gupta, and Kuwajima teach the information processing apparatus according to claim 4. Chatterjee does not explicitly teach wherein generate a check item of a data item specified by the user among a plurality of data items.
However, Dalli teaches wherein the circuitry is further configured to generate a check item of a data item among a plurality of data items included in learning data of the prediction model (i.e. fig. 11, A control and quality check 2030 may be applied such that the level of importance attributed to the final result is allocated in a manner which is fair and which does not cause unnecessary bias. The result of the control and quality check may be validated through the control node 2040, which determines if an exception should be triggered, and which then may seek human verification 2050 based on a result of said determination. In an exemplary embodiment, the control and quality check 2030 may entail applying one or more rules or conditions, and determining the validity of the feature attribution generated from the XAI/XNN model based on the satisfaction (or lack thereof) of said rules or conditions. (It may be contemplated for the one or more rules or conditions to be externally derived rules or conditions, such as rules or conditions which have been derived from legal conditions, institutional policy, and so forth, but further rules and conditions may also be contemplated, such as rules and conditions derived from interpretation of human verification 2050.); para. [0176]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Chatterjee, Gupta, and Kuwajima to include the feature of Dalli. One would have been motivated to make this modification because it ensures that rule-based outputs adhere to industry guidelines or ethical requirements.
However, Kaduwela teaches wherein the circuitry is further configured to generate the check item of a data item specified by the user among a plurality of data items (i.e. The method 700 begins at step 710 as the user interacts with a definitions/checks module 702 via the client device 202. The definitions/checks module 702 is a GUI that contains all of the tools a user requires to provide the needed input to create check data quality. At step 710, the user creates logical definition items before creating checks at step 712 and applying them to the logical definition items created in step 710. At step 714, the user chooses the type of check to perform (i.e., to apply to the definition item): either a standard operation, as shown in step 716, or a certain condition to be checked and an associated action to be taken if the condition is not met (i.e., in the form of an if, then statement) as shown at step 718. At step 720, the user creates a check list with one or more checks and, for check lists containing at least two checks, the order in which the checks should be performed; para. [0096]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Chatterjee, Gupta, Kuwajima, and Dalli to include the feature of Kaduwela. One would have been motivated to make this modification because users can focus on specific items that are critical to their application.
9. Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Chatterjee in view of Kuwajima, Gupta, and further in view of Rothstein et al. (U.S. Patent Application Pub. No. US 20230017097 A1).
Claim 12: Chatterjee, Gupta, and Kuwajima teach the information processing apparatus according to claim 11. Chatterjee does not explicitly teach wherein the circuitry is further configured to perform at least one of processing to reduce a number of pieces of the learning data that causes the violation rule failing to satisfy the predetermined standard, in the learning data included in the data range specified by the violation rule, or processing to add dummy data as the learning data to the data range specified by the violation rule, the dummy data being adjusted to satisfy the predetermined standard.
However, Rothstein teaches wherein the circuitry is further configured to perform at least one of processing to reduce a number of pieces of the learning data that causes the violation rule failing to satisfy the predetermined standard, in the learning data included in the data range specified by the violation rule, or processing to add dummy data as the learning data to the data range specified by the violation rule, the dummy data being adjusted to satisfy the predetermined standard (i.e. fig. 2, Following the parallel one or more steps of outlier filters described above, at a step 250 the outlier pairs would be removed from the training data before training of a ML model. Training would then be performed at a step 260. An ML model, such as a neural network described below with respect to FIG. 4, would be trained from the remaining training data. The resulting ML model, in production, would be used to predict parameters of new patterns from new scatterometric data; para. [0056]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Chatterjee, Gupta, and Kuwajima to include the feature of Rothstein. One would have been motivated to make this modification because it effectively filtering out problematic training instances.
10. Claims 14 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Chatterjee in view of Kuwajima, Gupta, and further in view of Tyagi et al. (U.S. Patent Application Pub. No. US 20170142048 A1).
Claim 14: Chatterjee, Gupta, and Kuwajima teach the information processing apparatus according to claim 11. Chatterjee further teaches the prediction model includes a classification model using a classification relating to the target item as a predicted value (i.e. a client of a machine learning service may submit a request to generate a classification model for a specified data set or data source (a source from which various observation records may be collected by the service and used for training the requested model). Each observation record may contain one or more input variables and at least one output or “target” variable (the variable for which the model is make predictions); para. [0018, 0019]).
Chatterjee does not explicitly teach the circuitry adjusts learning processing of the prediction model such that the predicted value of the prediction model in the data range specified by the violation rule substantially matches the predicted value of the prediction model in a data range specified by the output rule that satisfies the predetermined standard.
However, Tyagi teaches the circuitry adjusts learning processing of the prediction model such that the predicted value of the prediction model in the data range specified by the violation rule substantially matches the predicted value of the prediction model in a data range specified by the output rule that satisfies the predetermined standard (i.e. The expected value can be compared with the actual value, and adjustments can be made to the initial weights used to calculate the actual value such that the actual value matches the expected value, or approaches the expected value; para. [0056-0062]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Chatterjee, Gupta, and Kuwajima to include the feature of Tyagi. One would have been motivated to make this modification because it enhances adaptive learning efficiency.
Claim 15: Chatterjee, Gupta, and Kuwajima teach the information processing apparatus according to claim 11. Chatterjee further teaches wherein the prediction model includes a regression model using a value of the target item as a predicted value (i.e. regression algorithm; para. [0025, 0034, 0042]).
Chatterjee does not explicitly teach wherein the circuitry adjusts learning processing of the prediction model such that a distribution of the predicted value of the prediction model in the data range specified by the violation rule substantially matches a distribution of the predicted value of the prediction model in a data range specified by the output rule that satisfies the predetermined standard.
However, Tyagi teaches wherein the circuitry adjusts learning processing of the prediction model such that a distribution of the predicted value of the prediction model in the data range specified by the violation rule substantially matches a distribution of the predicted value of the prediction model in a data range specified by the output rule that satisfies the predetermined standard (i.e. The expected value can be compared with the actual value, and adjustments can be made to the initial weights used to calculate the actual value such that the actual value matches the expected value, or approaches the expected value; para. [0056-0062]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Chatterjee, Gupta, and Kuwajima to include the feature of Tyagi. One would have been motivated to make this modification because it enhances adaptive learning efficiency.
11. Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Chatterjee in view of Gupta, and further in view of Talagala et al. (U.S. Patent Application Pub. No. US 20190108417 A1).
Claim 16: Chatterjee and Gupta teach the information processing apparatus according to claim 3. Chatterjee does not explicitly teach wherein to present a plurality of adjustment methods relating to an output of the prediction model in a selectable manner in the user interface, and the circuitry adjusts learning processing of the prediction model based on a method selected by the user among the plurality of adjustment methods using the user interface.
However, Talagala teaches wherein the circuitry is further configured to present a plurality of adjustment methods relating to an output of the prediction model in a selectable manner in the user interface, and the circuitry adjusts learning processing of the prediction model based on a method selected by the user among the plurality of adjustment methods using the user interface (i.e. the administration module 404 is configured to present an interface for monitoring, controlling, modifying, setting, configuring, and/or the like one or more parameters of the pipelines 202, 204, 206a-c of the logical machine learning layer 200, 225, 250. For instance, the graphical interface may allow a user to configure various settings of the training pipeline 204 such as selecting and/or modifying the training data that is used to train the machine learning model, setting the weights or other characteristics of the machine learning model, selecting a particular machine learning model for the inference pipeline 206a-c to use, binding machine learning pipelines 202, 204, 206a-c to analytics engines, and/or the like. Similarly, the administration module 404 may receive input from a user that specifies which pipelines 202, 204, 206a-c should be logically grouped for analyzing the objective, different policies that define how the logical machine learning layer 200, 225, 250 operates, and/or the like; para. [0087]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Chatterjee and Gupta to include the feature of Talagala. One would have been motivated to make this modification because it helps users understand how different methods influence model’s predictions.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Mermoud et al. (Pub. No. US 20180278486 A1), a device in a network receives data regarding a plurality of predefined health status rules that evaluate one or more observed conditions of the network. The device, using the data regarding the plurality of health status rules for the network, trains a machine learning-based classifier to generate predictions regarding outputs of the health status rules. The device adjusts the machine learning-based classifier based on feedback associated with the generated predictions. The device provides an indication of one or more of the predictions regarding the outputs of the health status rules to a user interface.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAN TRAN whose telephone number is (303)297-4266. The examiner can normally be reached on Monday - Thursday - 8:00 am - 5:00 pm MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached on 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TAN H TRAN/Primary Examiner, Art Unit 2141
