DETAILED ACTION
Response to Amendment
The amendment filed on 6 February 2026 has been entered.
Claims 1-20 are pending.
Claims 2-4 are cancelled.
Claims 1, 5-20 are amended.
Claims 21-22 are new.
Claims 1, 5-22 will be pending.
Applicant’s amendments to the Claims have overcome each and every rejection under 35 USC 101 previously set forth in the Non-Final Office Action mailed 6 November 2025.
Allowable Subject Matter
Claim 1 is allowed.
The following is an Examiner’s statement of reasons for allowance.
The claim in the application is deemed to be directed to a nonobvious improvement over the prior art of record. The prior art does not anticipate, or render obvious as a whole, the claim limitations of Claim 1 as disclosed in Applicant’s claims.
Presented with the additional limitations of Claim 1, the prior art fails to make a prima facie case
of obviousness for limitations
“generating a set of impact values for the first level of features, comprising a first impact value indicating a first contribution of the first server to the target server usage difference and a second impact value indicating a second contribution of the second server to the target server usage”, “based on the first impact value being greater than the second impact value, generating a first subsequent set of impact values, for the first subset of features indicating contributions of the first subset of servers to the target server usage and a second subsequent set of impact values for the second subset of features indicating contributions of the second subset of servers to the target server usage”, “selecting a new target feature from the first subset of features or the second subset of features based on the new target feature having a highest impact value indicating that a machine learning model used to generate predicted server usage for a server of the first subset of servers or the second subset of servers is to be retrained”
because the machine learning models and methods configured for identifying performance degradation based on comparison of actual and predicted results shown to be found in the prior arts, made of record (see Non-Final Office Action, mailed 6 November 2025) cannot be reconciled with the above limitations and an obviousness determination would require impermissible hindsight.
In summary, the references made of record, fail to disclose the required claimed technical
features recited by the Claim 1 limitations as a whole. The claim, definite, and enabled by the Specification is allowed.
Response to Arguments
Applicant’s remarks, regarding the rejections of claims under 35 USC 103, have been fully considered.
Applicant respectfully submits that independent Claims 5 and 13, which have been amended based on the amendments to Claim 1, are also patent eligible and patentable for at least some of the same reasons as Claim 1, while the dependent claims are allowable at least based on their dependency from an allowable claim as well as for the additional features recited therein. New Claims 21 and 22, which depend from independent Claim 5, are therefore also allowable via their dependency from Claim 5 as well as for the additional features recited therein.
Applicant’s arguments have been considered, but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 5-7, 9, 11-15, 17, 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (U.S. Pre-Grant Publication No. 20210097425, hereinafter ‘Yang'), in view of Grossman (U.S. Patent No. 12165017, hereinafter 'Grossman').
Regarding claim 5 and analogous claim 13, Yang teaches A method comprising ([0082] FIG. 5 shows a computer system 500 in accordance with the disclosed embodiments. Computer system 500 includes a processor 502, memory 504, storage 506, and/or other components found in electronic computing devices. Processor 502 may support parallel processing and/or multi-threaded operation with other processors in computer system 500. Computer system 500 may also include input/output (I/O) devices such as a keyboard 508, a mouse 510, and a display 512.; [0087] The methods and processes described in the detailed description section can be embodied as code and/or data, which can be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and/or data stored on the computer-readable storage medium, the computer system performs the methods and processes embodied as data structures and code and stored within the computer-readable storage medium.):
storing, in a database, a feature hierarchy representing a plurality of features respectively associated with a plurality of machine learning models that are organized into a plurality of levels in accordance with feature dependencies ([0069] Second, data used by the system may be stored, defined, and/or transmitted using a number of techniques. For example, the system may be configured to retrieve a feature hierarchy feature hierarchy 108, insight templates 208, and/or feature-insight mappings 228 from storing, in a database different types of repositories, including relational databases, graph databases, data warehouses, filesystems, and/or flat files. The system may also obtain and/or transmit representing a plurality of features that are organized into a plurality of levels in accordance with feature dependencies scores 214, feature importance metrics 216, feature metadata 218, feature hierarchy 108, insight templates 208, feature-insight mappings 228, ranking 224, and/or narrative insights 238 in a number of formats, including database records, property lists, Extensible Markup language (XML) documents, JavaScript Object Notation (JSON) objects, and/or other types of structured data.; [0015] The disclosed embodiments provide a method, apparatus, and system for respectively associated with a plurality of machine learning models processing output from machine learning models. The output includes scores representing predictions, estimates, and/or inferences by the machine learning models of propensities, preferences, behavior, categories, and/or other attributes of users, companies, jobs, items, and/or other entities. The output also, or instead, includes a list of the most important features for a given score produced by a machine learning model and/or feature importance metrics representing the effects of features inputted into the machine learning model on the score.);
Yang fails to teach inputting past result data for the plurality of features into the plurality of machine learning models to obtain a plurality of predicted results respectively associated with the plurality of machine learning models; storing, in the database, in association with the plurality of features, the plurality of predicted results and a plurality of observed results; based on a target feature of the plurality of features and a subset of features associated with the target feature, executing performance degradation instructions to compute a set of impact values for the subset to derive a contribution of each feature of the subset of features to a difference, in the target feature, between the plurality of predicted results and the plurality of observed results; and based on the contribution of each feature of the subset of features indicating that a new target feature has a highest impact value, retraining a machine learning model associated with the new target feature using a new training dataset, for the new target feature, comprising new predicted values from the machine learning model and new observed values associated with the new target feature.
Grossman teaches inputting past result data for the plurality of features into the plurality of machine learning models to obtain a plurality of predicted results respectively associated with the plurality of machine learning models ([Col. 4, Lines 32-47] As part of training the machine learning model, the computing device generates (and assigns) a plurality of weighting coefficients (i.e., feature importances) corresponding to the plurality of input variables of the training data. There is a 1:1 correspondence between the weighting coefficients and the input variables. For example, regression analysis may be used to select an optimal set of weighting coefficients that in combination most accurately predict the known/correct outcome in the training data (sometimes referred to as “ground truth”) inputting past result data for the plurality of features into the plurality of machine learning models based on the plurality of input variables of the training data. Each weighting coefficient is a measure of how influential (e.g., significant, impactful, degree of correlation, etc.) the computing device determined that a single respective input variable of the training data should be on the to obtain a plurality of predicted results respectively associated with the plurality of machine learning models output prediction that the machine learning model engine generates.);
storing, in the database, in association with the plurality of features, the plurality of predicted results and a plurality of observed results ([Col. 4, Lines 32-36] As part of training the machine learning model, the computing device generates (and assigns) a plurality of weighting coefficients in association with the plurality of features (i.e., feature importances) corresponding to the plurality of input variables of the training data.; [Col. 4, Line 64-Col. 5, Line 4] After training a machine learning model engine, the computing device stores the results (e.g., the weighting coefficients) generated during the Training Phase in a database (e.g., data storage system 112 in FIG. 1 ), thereby allowing a machine learning model analyzer (e.g., machine learning model analyzer 104 in FIG. 1 ) to retrieve (e.g., access, receive, etc.) the storing, in the database stored the plurality of predicted results and a plurality of observed results results at a later time, such as when assessing data drift in the machine learning model.; [Col. 6, Lines 21-29] As will be appreciated, therefore, data that is considered scoring data as of “today” may be used as training data at some point in the future (i.e., after the actual outcomes are known or can be inferred with reasonable accuracy). Additionally, once the actual outcomes are known, that information itself may be used to determine whether to retrain the model (i.e., because it may be determined that the model is no longer predicting actual outcomes with a sufficient degree of accuracy).);
based on a target feature of the plurality of features and a subset of features associated with the target feature, executing performance degradation instructions to compute a set of impact values for the subset to derive a contribution of each feature of the subset of features to a difference, in the target feature between the plurality of predicted results and the plurality of observed results ([Col. 3, Lines 47-63] For example, a first input variable of a plurality of input variables may have a higher drift metric than a second input variable. If the first input variable happens to have a negligible impact (also referred to herein as, “feature importance”) on the output prediction, then it may not be necessary to re-train the model. (The based on a target feature of the plurality of features and a subset of features associated with the target feature feature importance of a particular input variable is sometimes mathematically represented by a weighting coefficient for the input variable, with different input variables of the model having different weighting coefficients. The term “weighting coefficient” is a specific example of the more general term “feature importance.” For simplicity, in the following discussion, the term “weighting coefficient” will be used, although it will be understood that other types of feature importances may be used as well. In this vein, it may also be noted that the terms “input variable” and “feature” are used interchangeably herein.; [Col. 4, Lines 32-47] As part of training the machine learning model, the computing device executing performance degradation instructions to compute a set of impact values for the subset to derive a contribution of each feature of the subset of features to a difference generates (and assigns) a plurality of weighting coefficients (i.e., feature importances) corresponding to the plurality of input variables of the training data. There is a 1:1 correspondence between the weighting coefficients and the input variables. For example, regression analysis may be used to in the target feature between the plurality of predicted results and the plurality of observed results select an optimal set of weighting coefficients that in combination most accurately predict the known/correct outcome in the training data (sometimes referred to as “ground truth”) based on the plurality of input variables of the training data. Each weighting coefficient is a measure of how influential (e.g., significant, impactful, degree of correlation, etc.) the computing device determined that a single respective input variable of the training data should be on the output prediction that the machine learning model engine generates.); and
based on the contribution of each feature of the subset of features indicating that a new target feature has a highest impact value ([Col. 4, Lines 47-63] An input variable having a higher feature importance value (weighting coefficient) has a greater impact on the predictions made by the machine learning model engine than an input variable having a lower feature importance value. Typically, but not necessarily, the weighting coefficients sum to a value of ‘1.’ For example, the computing device may train a machine learning model engine with a set of training data associated with a plurality of input variables. Upon analyzing (e.g., performing a regression analysis on) the input variables of the training data and the output predictions that the machine learning model engine generates based on different sets of weighting coefficients, the computing device may generate (e.g., select) an based on the contribution of each feature of the subset of features indicating that a new target feature has a highest impact value optimal set of weighting coefficients (i.e., one weighting coefficient for each of the input variables) that in combination most accurately predict the known/correct outcome in the training data.),
retraining a machine learning model associated with the new target feature using a new training dataset, for the new target feature, comprising new predicted values from the machine learning model and new observed values associated with the new target feature ([Col. 6, Lines 21-29] As will be appreciated, therefore, data that is considered scoring data as of “today” may be using a new training dataset used as training data at some point in the future (i.e., for the new target feature, comprising new predicted values from the machine learning model and new observed values associated with the new target feature after the actual outcomes are known or can be inferred with reasonable accuracy). Additionally, once the actual outcomes are known, that information itself may be used to determine retraining a machine learning model associated with the new target feature whether to retrain the model (i.e., because it may be determined that the model is no longer predicting actual outcomes with a sufficient degree of accuracy).).
Yang and Grossman are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Yang, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Grossman to Yang before the effective filing date of the claimed invention in order to account for the differences in feature importances of the plurality of input variables (cf. Grossman, [Col. 4, Lines 7-15] According to embodiments herein, an overall drift metric for a machine learning model across a plurality of input variables is calculated. Specifically, a weighted average drift (WAD) score/metric is calculated in a way that accounts for the differences in feature importances of the plurality of input variables. (Herein, the terms “WAD score” and “WAD metric” are used interchangeably.) The WAD score may then be used to determine whether it is necessary to re-train the machine learning model.).
Regarding claim 6 and analogous claim 14, Yang, as modified by Grossman, teaches The method of claim 5 and The one or more non-transitory, computer-readable media of claim 13, respectively.
Grossman teaches displaying, via an I/O interface or an I/O device, an alert indicating the machine learning model to be retrained ([Col. 7, Lines 24-32] Thus, in some embodiments, the disclosure herein improves the operation of the computer system 100 shown in FIG. 1 in that the machine learning model analyzer 104 is now able to transmit an alert signal indicating to the client devices 102 that the machine learning model executing on the machine learning model servers 106 should be retrained, or at least, that the outputs generated by the machine learning model servers 106 should be discounted or disregarded.).
Yang and Grossman are combinable for the same rationale as set forth above with respect to claim 5.
Regarding claim 7 and analogous claim 15, Yang, as modified by Grossman, teaches The method of claim 5 and The one or more non-transitory, computer-readable media of claim 13, respectively.
Grossman teaches wherein retraining the machine learning model comprises: displaying, via an I/O interface or an I/O device, an alert comprising the new target feature; and in connection with the alert being displayed, receiving a command to retrain the machine learning model using the new training dataset ([Col. 23, Lines 11-32] In some embodiments, the alert signal may trigger the machine learning model server 106 to retrain the machine learning model engine 108, using a second (e.g., newer) set of training data that is different than the training data previously used to train the machine learning model engine 108. In some arrangements, the alert may cause the machine learning model server 106 to deny requests from a computing device (e.g., a client device 102) for an output prediction that would otherwise be generated by the machine learning model engine 108. In this manner, the alert may cause the machine learning model engine 108 to be removed from a production environment (i.e., so it is no longer “deployed,” provides responses devoid of predictions, etc.). In some arrangements, the WAD generation circuit 320 may send an alert to a computing device (e.g., a client device 102) to trigger the computing device to display the WAD score on a screen associated with the computing device. In other arrangements, the WAD generation circuit 320 may present the weighted average drift score on a display (e.g., computer screen 103) associated with the machine learning model analyzer 104. Various other examples of such alerts have previously been provided.; [Col. 9, Lines 56-64] The machine learning model analyzer 104 is an electronic computing device associated with an organization that is configured to retrieve model data (e.g., weighting coefficients) from a data storage system 112 and generate a weighted average drift (WAD) score based on the model data. The machine learning model analyzer 104, in some arrangements, may be configured to send an alert to a client device 102 causing the client device 102 to display information associated with the alert on a computer screen.).
Yang and Grossman are combinable for the same rationale as set forth above with respect to claim 5.
Regarding claim 9 and analogous claim 17, Yang, as modified by Grossman, teaches The method of claim 5 and The one or more non-transitory, computer-readable media of claim 13, respectively.
Yang teaches further comprising determining, for each subsequent level of the feature hierarchy based on subsequent sets of impact values, a subsequent target feature at each subsequent level of the feature hierarchy having a subsequent highest impact value ([0021] Next, some or all determining, for each subsequent level of the feature hierarchy parent features in the feature hierarchy are ranked based on subsequent sets of impact values based on feature importance metrics for child features of the parent features. For example, an overall score for each first-level parent feature in the feature hierarchy is a subsequent target feature at each subsequent level of the feature hierarchy having a subsequent highest impact value calculated as the highest feature importance metric associated with a child feature of the first-level parent feature, and the first-level parent features are ranked by descending overall score. When two or more first-level parent features in the ranking are grouped under a second-level parent feature in the feature hierarchy, the first-level parent feature with the highest overall score is retained in the ranking, and all other first-level parent features under the same second-level parent feature are removed from the ranking to reduce the redundancy of information in the narrative insights.).
Yang and Grossman are combinable for the same rationale as set forth above with respect to claim 5.
Regarding claim 11 and analogous claim 19, Yang, as modified by Grossman, teaches The method of claim 9 and The one or more non-transitory, computer-readable media of claim 17, respectively.
Yang teaches further comprising: identifying one or more dependencies between one or more features of a first level of features of the plurality of levels and the target feature to calculate a relationship between each feature of the first level of features and the target feature ([0040] For example, analysis apparatus 202 generates a mapping of one or more attributes in feature metadata 218 (e.g., feature name, value, ID, location, description transformation, etc.) for a model feature to a first-level parent feature (i.e., a “super feature”) of the feature and/or a second-level parent feature (e.g., an “ultra feature”) above the first-level parent feature in feature hierarchy 108. Analysis apparatus 202 optionally updates the mapping to include a name or identifier of an insight template for one or both parent-level features and/or a feature position (i.e., placeholder) in the insight template. As a result, analysis apparatus 202 establishes associations among model features 212, feature hierarchy elements 220, and/or insight templates 208 to improve and/or streamline subsequent processing using model features 212, feature hierarchy 108, and insight templates 208.),
identifying one or more subsequent dependencies between subsequent features of each subsequent set each respective subsequent target feature to calculate, for each subsequent level of the feature hierarchy, a subsequent relationship between each feature of each subsequent set and each respective subsequent target feature ([0040] For example, analysis apparatus 202 generates a mapping of one or more attributes in feature metadata 218 (e.g., feature name, value, ID, location, description transformation, etc.) for a model feature to a first-level parent feature (i.e., a “super feature”) of the feature and/or a second-level parent feature (e.g., an “ultra feature”) above the first-level parent feature in feature hierarchy 108. Analysis apparatus 202 optionally updates the mapping to include a name or identifier of an insight template for one or both parent-level features and/or a feature position (i.e., placeholder) in the insight template. As a result, analysis apparatus 202 establishes associations among model features 212, feature hierarchy elements 220, and/or insight templates 208 to improve and/or streamline subsequent processing using model features 212, feature hierarchy 108, and insight templates 208.); and
generating, based on the relationship and each subsequent relationship, the feature hierarchy ([0042] Analysis apparatus 202 also includes functionality to calculate metrics, statistics, and/or other derived values from model features 212 during or after mapping of model features 212 to feature hierarchy elements 220 in feature hierarchy 108. These calculations are performed when one or more model features 212 are mapped to groupings of features in feature hierarchy 108 and/or insight templates 208 that include these derived values.).
Yang and Grossman are combinable for the same rationale as set forth above with respect to claim 5.
Regarding claim 12 and analogous claim 20, Yang, as modified by Grossman, teaches The method of claim 11 and The one or more non-transitory, computer-readable media of claim 19, respectively.
Yang teaches wherein generating the feature hierarchy comprises generating, based on each relationship and each subsequent relationship, a structure comprising a number of levels, a particular subset of the plurality of features on each level, the one or more dependencies, and the one or more subsequent dependencies ([0033] As mentioned above, feature hierarchy 108 groups features that are available as input into machine learning models (e.g., features used with the machine learning models by machine learning platform 206) under parent features in one or more feature hierarchy levels 222. In some embodiments, each parent feature in feature hierarchy 108 represents a common definition, category, and/or concept for features grouped under the parent feature. As a result, feature hierarchy 108 is used to organize large numbers of disparate features into semantically related groupings.; [0034] For example, the feature hierarchy includes groupings of features under two levels of parent features. The first level of the feature hierarchy includes “super features” that are parents of features inputted into the machine learning model.; [0035] Continuing with the above example, the feature hierarchy also includes groupings of first-level super features under a second, higher level of “ultra features.” Each ultra feature in the second level represents a commonality, redundancy, or overlap in definition for first-level super features grouped under the ultra feature (e.g., an ultra feature of “searches” includes child super features of “average searches per user” and “total searches by users”).).
Yang and Grossman are combinable for the same rationale as set forth above with respect to claim 5.
Claims 8, 10, 16, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Yang, Grossman, and further in view of Lee et al. (U.S. Pre-Grant Publication No. 20230401512, hereinafter 'Lee').
Regarding claim 8 and analogous claim 16, Yang, as modified by Grossman, teaches The method of claim 5 and The one or more non-transitory, computer-readable media of claim 13, respectively.
Yang, as modified by Grossman, fails to teach further comprising: accessing an indicator of the machine learning model associated with the new target feature; and accessing, using the indicator, the new training dataset for retraining the machine learning model.
Lee teaches further comprising: accessing an indicator of the machine learning model associated with the new target feature; and accessing, using the indicator, the new training dataset for retraining the machine learning model ([0020] The calculations can be employed to determine stability between a feature and an output, e.g., a stability of a feature-target relationship, over time. A feature-target relationship stability can be referred to as a temporal stability measurement (TSM). TMC 120 can generate one or more TSMs, e.g., a group of VPMs can correspond to a group of TSMs. It is noted that in this example, the count of VPM in the VPMs can be the as or different than the count of TSM in the TSMs, e.g., there can be more VPMs that TSMs, there can be more TSMs than VPMs, or there can be an equal count of VPMs and TSMs. Generally, a TSM can indicate how significant an impact a VPM is expected to have on a model over time, e.g., a TSM will typically include a accessing an indicator of the machine learning model associated with the new target feature variety of indicators for feature impact across time for an input model.; [0039] An output from TMC 420, e.g., TSM 422, fixed point component 424 calculations, over time component 426 calculations, model data 402, VPM 412, etc., can be consumed by temporal feature stability detection component (TFSDC) 460. TFSDC 460 can monitor a model in production to detect when temporal generalization is degrading, failing, etc. Conventional systems and tools do not measure stability of a relationship between a feature and a target, e.g., feature-target stability, over time. However, monitoring a system, e.g., a model in a production environment, can set an alert, flag, notification, trigger a response, etc., to a feature-target relationship with a sufficiently degraded stability, e.g., according to a TG degradation rule, filter, ranking, algorithm, etc.; [0030] System 200 can comprise DASH 240 that can present result data 204, etc. DASH 240 can be the same as, or similar to, DASH 140, e.g., a dashboard enabling user interactions with system 200 that can include presenting a user with results from FAC 230, TMC 220, fixed point component 224 calculations, over time component 226 calculations, MSC 210, VPM 212, model data 202, etc., can enable selection of permutation schema, can facilitate automated updating of models in development and/or production environments, can facilitate designation of a subsequent analysis, etc., among many dashboard-type interactions with a system described herein, e.g., system 100, 200, . . . , etc.; [0040] In embodiments, both VPMs and TSMs can be employed to measure the stability of a model in a production environment, e.g., TFSDC 460 can monitor stability to allow system 400 to alert for feature-target instability over a period of time. TFSDC 460 analysis of one or more TSM 422 can be contrasted with conventional techniques such as LIME plots, etc., to analyze fixed-point production. As such, TFSDC 460 can analyze TSMs over a set of production data across time. These values can then be accessing, using the indicator, the new training dataset for retraining the machine learning model used in heuristic, statistical, or machine learning algorithms to alert about certain features, e.g., features that need to be repaired/modified, have degraded and should be pruned, can be countered with a newly added feature, etc. Furthermore, data collected from system 400 can be used within a FAC, for example similar to or the same as FAC 130, 230, 330, etc., to further target certain features by mutating the production model, analysis of the permutations, and selection of a preferred updated model that can be implemented to replace a current production model suffering from temporal generalization effects.).
Yang, Grossman, and Lee are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Yang and Grossman, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Lee to Yang before the effective filing date of the claimed invention in order to mitigate temporal generalization losses, such as by identifying, removing, modifying, transforming, etc., features, explanatory variables, models, etc., that can have an unstable relationship with a target model outcome over time (cf. Lee, [0015] The disclosed subject matter discloses a technology intended to mitigate temporal generalization losses, such as by identifying, removing, modifying, transforming, etc., features, explanatory variables, models, etc., that can have an unstable relationship with a target model outcome over time. As an example, a model performance can be monitored to detect TG, which can cause initiation of a process transforming the model, variables of the model, etc., into a more stable representation, e.g., into an updated model, etc., that can have improved temporal stability and correspondingly mitigate TG effects.).
Regarding claim 10 and analogous claim 18, Yang, as modified by Grossman, teaches The method of claim 9 and The one or more non-transitory, computer-readable media of claim 17, respectively.
Yang teaches further comprising: determining, for each subsequent level of the feature hierarchy based on the subsequent sets of impact values, whether each subsequent level of the feature hierarchy comprises additional target features, wherein each additional target feature has an impact value within a certain threshold of the highest impact value at each subsequent level determining that one or more subsequent levels comprise one or more additional target features ([0045] Next, analysis apparatus 202 generates a determining, for each subsequent level of the feature hierarchy based on the subsequent sets of impact values ranking 224 of some or all feature hierarchy elements 220 based on mappings of model features 212 to feature hierarchy elements 220, one or more feature hierarchy levels 222 in feature hierarchy 108, and feature importance metrics 216 for model features 212. In one or more embodiments, analysis apparatus 202 generates ranking 224 based on overall scores 226 associated with parent features in one or more feature hierarchy levels 222. In turn, overall scores 226 are based on feature importance metrics 216 of model features 212 mapped to child features grouped under the parent features.; [0046] For example, analysis apparatus 202 calculates an overall score for each first-level parent feature in feature hierarchy 108 as the highest feature importance metric associated with a child feature of the first-level parent feature and/or another aggregation (e.g., sum, average, median, etc.) of whether each subsequent level of the feature hierarchy comprises additional target features feature importance metrics 216 for child features grouped under the first-level parent feature. Analysis apparatus 202 then generates ranking 224 by ordering the first-level parent features by descending overall scores 226. When two or more first-level parent features in ranking 224 are grouped under a second-level parent feature in feature hierarchy 108, analysis apparatus 202 keeps only the first-level parent feature with the highest overall score in ranking 224 to reduce the redundancy of information in narrative insights 238 generated from first-level parent features in ranking 224.; [0047] In addition, wherein each additional target feature has an impact value within a certain threshold of the highest impact value at each subsequent level determining that one or more subsequent levels comprise one or more additional target features elements in ranking 224 are ordered so that features or concepts with greater impact on the output are ranked higher than features or concepts with less impact on the output.); and
Yang, as modified by Grossman, fails to teach retraining one or more additional machine learning models used to generate predicted values for the one or more additional target features using a new training dataset.
Lee teaches retraining one or more additional machine learning models used to generate predicted values for the one or more additional target features using a new training dataset ([0033] Accordingly, feature scoring component 322 can enable identification of features, for example, according to a ranking by feature impact, which can be expected to most change a models temporal generalization via a modification to the feature, removal of the feature from the model, weighting of the feature value, addition of a new counter feature to the model, etc.; [0040] In embodiments, both VPMs and TSMs can be employed to measure the stability of a model in a production environment, e.g., TFSDC 460 can monitor stability to allow system 400 to alert for feature-target instability over a period of time. TFSDC 460 analysis of one or more TSM 422 can be contrasted with conventional techniques such as LIME plots, etc., to analyze fixed-point production. As such, TFSDC 460 can analyze TSMs over a set of production data across time. These values can then be used in heuristic, statistical, or machine learning algorithms to alert about certain features, e.g., features that need to be repaired/modified, have degraded and should be pruned, can be countered with a newly added feature, etc. Furthermore, retraining one or more additional machine learning models used to generate predicted values for the one or more additional target features using a new training dataset data collected from system 400 can be used within a FAC, for example similar to or the same as FAC 130, 230, 330, etc., to further target certain features by mutating the production model, analysis of the permutations, and selection of a preferred updated model that can be implemented to replace a current production model suffering from temporal generalization effects.).
Yang, Grossman, and Lee are combinable for the same rationale as set forth above with respect to claim 8.
Claims 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Yang, Grossman, and further in view of Ghosh et al. (NPL: "LEARNING BASED HIERARCHICAL LOAD BALANCER", hereinafter 'Ghosh').
Regarding claim 21, Yang, as modified by Grossman, teaches The method of claim 5.
Yang, as modified by Grossman, fails to teach wherein the feature hierarchy represents a hierarchy of servers linked together to form a computing network, storing the feature hierarchy comprises: generating the feature hierarchy to represent a hierarchy of servers linked together to form a computing network including the target feature representing a target server usage of a target server of the hierarchy of servers by: determining a first level of features comprising a first feature and a second feature respectively representing first server usage of a first server and second server usage of a second server, the target server referring queries to at least one of the first server or the second server, and determining a second level of features comprising a first subset of features and a second subset of features respectively representing first subset server usages of a first subset of servers and second subset server usages of a second subset of servers, the first server referring the queries to at least one of the first subset of servers and the second server refers the queries to at least one of the second subset of servers.
Ghosh teaches wherein the feature hierarchy represents a hierarchy of servers linked together to form a computing network, storing the feature hierarchy comprise ([3.2.2 Asynchronous reduction of statistics, pg. 13] The feature hierarchy represents a hierarchy of servers linked together to form a computing network, storing the feature hierarchy nodes are organized in a hierarchical structure as shown in Figure 5.5. Raw statistics are collected at the iteration boundary of each node. These statistics are then aggregated and transformed into features on each of these nodes.):
generating the feature hierarchy to represent a hierarchy of servers linked together to form a computing network including the target feature representing a target server usage of a target server of the hierarchy of servers by ([4.1.1 General methodology, pg. 15] To evaluate load, we need to collect traces and performance metrics from the application. It needs to provide generating the feature hierarchy to represent a hierarchy of servers linked together to form a computing network including the target feature representing a target server usage of a target server of the hierarchy of servers by load estimation on each processor and the accuracy of this estimation based on collected traces and metrics directly affects the performance of the corresponding load balancer. As all subsequent decisions are based on this estimation. The collected metrics are used to determine two kinds of trends namely computation based trends and communication based trends. For the computation metrics we collect processor loads, object loads, utilization of each processor, background loads, etc. We can do this by using instrumentation methods to measure task execution overheads.):
determining a first level of features comprising a first feature and a second feature respectively representing first server usage of a first server and second server usage of a second server, the target server referring queries to at least one of the first server or the second server ([3.2.1 Statistics collection, pg. 13] The load balancer framework collects raw statistics on each individual chare using the Charm++ runtime framework. These determining a first level of features comprising a first feature and a second feature respectively representing first server usage of a first server and second server usage of a second server raw statistics need to be converted into meaningful features as mentioned in table 3.1. At each iteration boundary, an individual chare collects raw statistics, transforms them into meaningful features and keeps aggregating them. When the decision is made to perform a load balancing operation, these the target server referring queries to at least one of the first server or the second server aggregated statistics are used to make a load balancing decision using the trained learning model. These counters are then reset for the subsequent iterations.), and
determining a second level of features comprising a first subset of features and a second subset of features respectively representing first subset server usages of a first subset of servers and second subset server usages of a second subset of servers, the first server referring the queries to at least one of the first subset of servers and the second server refers the queries to at least one of the second subset of servers ([3.2.2 Asynchronous reduction of statistics, pg. 13-14] The nodes are organized in a hierarchical structure as shown in Figure 5.5. Raw statistics are collected at the iteration boundary of each node. These statistics are then aggregated and transformed into features on each of these nodes. We have kept the levels of the tree fixed for this work. So for a four level tree where the top level, say level 0, is the root node and any load balancing decision made here is made for the entire system. The bottom level (level 3) represents each individual chare. Since, the bottom level has only chare, there is no load balancing call that is made at this level. determining a second level of features comprising a first subset of features and a second subset of features respectively representing first subset server usages of a first subset of servers and second subset server usages of a second subset of servers Level 1 and level 2 each has a group of chares numbered next to each other. When a the first server referring the queries to at least one of the first subset of servers and the second server refers the queries to at least one of the second subset of servers load balancing call is made at node level which is not the lowest level, statistics from all the nodes that is a child of it are merged recursively. Since, these statistics are already aggregated, these can be merged by a singular reduce operation. The main motivation to aggregate these statistics and reduce them on each level is to avoid increase in data sizes as we travel up the tree. To gather statistical information from child nodes, we use Charm++ framework and pass these information as messages to parent nodes. The parent node is also responsible for an reducing these statistics and aggregating them such that they accurately represent all the child nodes or processors that are part of the group the parent node controls.).
Yang, Grossman, and Ghosh are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Yang and Grossman, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Ghosh to Yang before the effective filing date of the claimed invention in order to scale the load balancer better across multiple nodes (cf. Ghosh, [Abstract, pg. ii] Here, we propose a load balancing algorithm that uses runtime system collected statistics to represent the state of an application and then uses learning models to predict a good choice of load balancer to use based on the collected statistics. We also propose the load balancer within a hierarchical framework that allows us to scale the load balancer better across multiple nodes.).
Regarding claim 22, Yang, as modified by Grossman, teaches The method of claim 5.
Yang, as modified by Grossman, fails to teach wherein each machine learning model of the plurality of machine learning models output a predicted server usage of a cloud-based server of a plurality of cloud-based servers forming a computing network, and wherein the plurality of predicted results comprise a plurality of predicted server usage values for the plurality of cloud-based servers, respectively, and wherein the plurality of observed results comprise a plurality of observed server usage values measured by the plurality of cloud-based servers, respectively.
Ghosh teaches wherein each machine learning model of the plurality of machine learning models output a predicted server usage of a cloud-based server of a plurality of cloud-based servers forming a computing network ([2.3 RELATED WORK, pg. 7] Load balancers in general have been an extensively studies problem. It has applications in high-performance parallel applications, distributed cloud clusters, operating systems, dis tributed databases, web servers, etc. In each of these fields, the higher-level idea is the same but the implementation of the load balancer varies depending on the use-case, the nature of the tasks and their loads that we balance among the nodes.; [1.2 SUMMARY, pg. 3] The proposed load balancer needed development of tracing and metrics collection frame work, which continuously monitors the system configurations and the state of an application over time. Subsequently, this collected wherein each machine learning model of the plurality of machine learning models output a predicted server usage of a cloud-based server of a plurality of cloud-based servers forming a computing network information is used to predict the behaviour of the system and application using supervised machine learning algorithms. This predicted be haviour helps decide the load balancer that needs to be used at a particular instant of time for a given subset of compute nodes to achieve best performance in the subsequent runs of the application.), and
wherein the plurality of predicted results comprise a plurality of predicted server usage values for the plurality of cloud-based servers, respectively, and wherein the plurality of observed results comprise a plurality of observed server usage values measured by the plurality of cloud-based servers, respectively ([Learning used for load balancers in operating systems:, pg. 9] Specific to the problem of using learning models for load balancers, the paper by Chen et. al.[28] have used machine learning to understand the state of an operating system and propose load balancing decisions in the operating system. Learning methods have been used to predict load balancing[29] in cloud servers using the system characteristics of the servers. The paper by Agarwal et. al.[30] also surveys other existing learning approaches for load balancing.; [eXtreme Gradient Boosting(XGBoost), pg. 26] Xgboost[44] uses the idea of wherein the plurality of predicted results comprise a plurality of predicted server usage values for the plurality of cloud-based servers, respectively, and wherein the plurality of observed results comprise a plurality of observed server usage values measured by the plurality of cloud-based servers gradient tree boosting. These use classification and regression trees(CART) as their weak learners. These are similar to decision trees, except that each leaf node is not attached with a score, which decides its weight in the final decision. The algorithm solves the deficiencies of the previous model with respect to over-fitting and and the training time using statistical techniques like regularization. Though, this comes at the cost of increased hyper-paramaters which need to be carefully tuned to get the best performance. We have used validation sets to tune these hyper-parameters.).
Yang, Grossman, and Ghosh are combinable for the same rationale as set forth above with respect to claim 21.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAGGIE MAIDO whose telephone number is (703) 756-1953. The examiner can normally be reached M-Th: 6am - 4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached on (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MM/Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights