DETAILED ACTION
This final action is in response to the amendment and remarks filed on 08/04/2026 for application 18/325,844.
Claims 1-5, 11-15, and 17-19 have been amended. Claims 10 and 20 are cancelled. Claims 21 and 22 are newly added claims.
Claims 1-9, 11-19, and 21-22 thereby remain pending in the application. Claims 1, 11, and 19 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-22 are rejected under 35 U.S.C. 103 as being unpatentable over Harris et al. (Pub. No. US 20180330258 A1, “Autonomous Learning Platform for Novel Feature Discovery”, filed 05/09/2017), hereinafter Harris, in view of Brumbaugh et al. (“Bighead: A Framework-Agnostic, End-to-End Machine Learning Platform”, available 2019), hereinafter Brumbaugh, and Chatzimparmpas et al. (“FeatureEnVi: Visual Analytics for Feature Engineering Using Stepwise Selection and Semi-Automatic Extraction Approaches”, available April 2022), hereinafter Chatzimparmpas, and Tepper et al. (Pub. No. US 20200327365 A1, “Large-Scale Similarity Search with On-Chip Cache”, published 10/15/2020), hereinafter Tepper.
Regarding claim 1, Harris teaches A system (“Various embodiments are directed to performing autonomous learning for updating input features used for an artificial intelligence model. For example, a method can comprise receiving updated data of an information space that includes a graph of nodes having a defined topology, the updated data including historical data of requests to the artificial intelligence model and output results associated with the requests, wherein different categories of input data corresponds to different input nodes of the graph…In addition, the method may also include grouping input nodes to identify candidate input features to the artificial intelligence model, wherein the input node(s) of a candidate input feature are correlated to each other, and wherein the candidate input features have strength values above a threshold. Furthermore, the method may comprise training an enlarged artificial intelligence model using the historical data for the candidate input features, the training providing a ranking of candidate input features relative to a measure of effect on an output value of the enlarged artificial intelligence model, and selecting the top N candidate input features to be used in the artificial intelligence model.” [Harris ¶ 0005]; “Other embodiments are directed to systems, devices, and computer readable media associated with methods described herein” [Harris ¶ 0007]) comprising:
a non-transitory memory ([Harris ¶ 0007] as detailed above); and
one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations ([Harris ¶ 0007] as detailed above) comprising:
generating a plurality of variable clusters and a cluster representative vector for each of the plurality of variable clusters based on a plurality of variable vectors for a plurality of variables using a machine learning (ML) clustering technique; (“At step 430, the newly detected features may be smoothed (binned) to create a reduced feature set. The nodes of the graph may be grouped or formed into clusters, in which nodes that are close to each other in topology form a cluster and may have similar properties such as similar signal-to-noise ratio. Clusters with sufficiently high connection to a given output node may be identified as a possible input feature” [Harris ¶ 0066]; “Features may be manually selected from the graph by one skilled in the art based on knowledge about the information space. This may done based on common sense rules and/or using any data mining technique, such as cluster analysis. For example, k-means clustering in combination with pre-defined rules (e.g. node Y: ‘male’ should not be of the same feature as node A: ‘female’) may be used to group nearby nodes into a single feature” [Harris ¶ 0059]; K-means clustering expressly calculates centroids (i.e., representative vectors) for each cluster)
storing cluster data comprising the plurality of variable clusters and the cluster representative vector for each of the plurality of variable clusters ([Harris ¶ 0066] as detailed in claim 1 above; The clusters (and associated features) are stored for input to the downstream model), and the system perform[ing] a variable engineering for an ML model; (“In some implementations, an information space can be analyzed to determine suitable features for training the AI model… An artificial intelligence model may make predictions about the information space using a learning algorithm that evaluates the position and connectivity of the nodes in a graph and recognizes patterns. The patterns may be reinforced by an indication that an outcome predicted using the AI model was correct or incorrect, which may further have an effect on the AI model. Accurate predictions and the recognized patterns that lead to those predictions can be remembered so that more accurate predictions may be made in the future” [Harris ¶ 0031-0032]; “Rather than score individual nodes for predictiveness, the AI model may score a collection of nodes, known as features. This may accelerate the learning process as well as require less memory for the AI to make accurate predictions. The features shown in FIG. 3 are the nodes enclosed by dashed lines (e.g., nodes Y,W,X and A,X,B). The features may be used to train an AI model, where a learning algorithm is used to train the AI and score the predictiveness of the features. For example, first feature 301: ‘male, San Francisco, Shops at music store’, may be used as a feature for predicting a recommendation to be sent to a user device… Over time, the AI model may become better at predicting the best recommendations for each device in the transaction network. For example, the AI model may determine that second feature 302: ‘female, San Francisco, shops at thrift shop’ is highly predictive of a recommendation for ‘hiking’ being accepted” [Harris ¶ 0058-0060])
responsive to performing the variable engineering, accessing an ML model configuration associated with the ML model (“In some implementations, an information space can be analyzed to determine suitable features for training the AI model. The information space may be represented as a topological graph of nodes connected by edges, which can have assigned weight, distance, or cost” [Harris ¶ 0031]; “FIG. 3 shows a depiction of selecting input features from a topological graph. In graph 300, each node (A through G, S through Z) may represent a specific set (e.g., one piece or a collection) of information determined from collected data….The information of graph 300 may be used as training data for an artificial intelligence model” [Harris ¶ 0055, 0057]; A topological graph (i.e., configuration) is accessed to obtain features which are used as training data for an AI model (i.e., associated with an ML model));
determining a first variable that enables a similarity detection based on the ML model configuration, wherein the first variable corresponds to a new variable used by the ML model during a performance of the variable engineering (“The information space may include a graph of nodes having a defined topology, where most of the nodes correspond to different inputs and some of the nodes correspond to outputs. An input may be any known or pre-defined characteristic of the information space, and an output may be any predicted outcome that depends on one or more inputs. Examples of an input may include transaction data, user profile data, text in a conversation, etc. Examples of outputs may include a targeted recommendation, a probability of fraud, a response to initiate a purchase, etc.” [Harris ¶ 0062]; “At step 430, the newly detected features may be smoothed (binned) to create a reduced feature set. The nodes of the graph may be grouped or formed into clusters, in which nodes that are close to each other in topology form a cluster and may have similar properties such as similar signal-to-noise ratio. Clusters with sufficiently high connection to a given output node may be identified as a possible input feature. A predetermined amount of input features may be added to a current set of input features to develop a new feature set or current version of the model. For example, 50 new input features can be added to a current set of 200 input features for the current version of the model These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; Wherein current nodes of the topology graph (i.e., model configuration) represent features, an output node may be identified for determining clusters of nodes with sufficiently high connection (i.e. similarity), and features of these nodes may be newly selected as input features (i.e., new variables), in conjunction with the existing current set of input features (i.e., including first variable) for training the model (i.e., determined to be predictive/suitable features during variable engineering process))
determining, in real-time during the performance of the variable engineering and using the cluster data, a first cluster of variables of the plurality of variable clusters matching the first variable based on a vector similarity for a first variable vector representing the first variable and the cluster representative vector representing the first cluster of variables (“Clusters with sufficiently high connection to a given output node may be identified as a possible input feature. A predetermined amount of input features may be added to a current set of input features to develop a new feature set or current version of the model. For example, 50 new input features can be added to a current set of 200 input features for the current version of the model. The 50 new features can be identified based on the connection strength to the selected output nodes by, for example, selecting the top 50 features based on a shortest distance from an output node. These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; “An edge may be associated with a numerical value, known as a weight, that may be assigned to the pairwise connection between the two nodes. The edge weight may be identified as a strength of connectivity between two nodes and/or may be related to a cost or distance” [Harris ¶ 0048]; “ An edge weight may also be used to express a cost or a distance required to move from one state or node to the next” [Harris ¶ 0024]; For a selected output node corresponding to the current set of input features (incl. first variable), a selection of clusters (incl. first cluster) each representing new input features can be identified (i.e., matched) based on relative connection strength (i.e., similarity), wherein connection strength may be further measured through edge weights representing any form of distance between nodes (e.g., Euclidean distance between vectors)); and
presenting variable similarity data for at least one of the first cluster of
variables or each variable in the first cluster of variables (“These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; “At step 440, the new feature set may be used to retrain the model using the new features. The training of the enlarged artificial intelligence model may be performed using the historical data for candidate input features, and the training may provide a ranking of the candidate input features relative to a measure of their effect on an output value of the enlarged artificial intelligence model… In one embodiment, the features in the enlarged feature set may be ranked according to strength of correlation to a predicted outcome, and the feature set may then be capped to its original size (e.g. ranking 250 features and then selecting only the top 200 features as the new feature set)” [Harris ¶ 0067]; Wherein the selected clusters (incl. first cluster) form the candidate features that are input for training, the features are further presented in a ranking based on the strength of their correlation (i.e., variable similarity) to a predicted outcome), wherein the variable similarity data is presented with an option to replace the first variable with a second variable from the first cluster of variables in place of creating the first variable as the new variable for the ML model ([Harris ¶ 0066, 0067] as detailed above; Wherein previous input features and new features may form the candidate features that are input to the downstream model, the features set may be further capped (e.g., top 50 features) wherein some previous input features are replaced by new input features (including, e.g., those drawn from a first cluster) in the next training batch based on measured importance).
However, Harris does not expressly teach a client application accessible by the system perform[ing] the variable engineering for [the] ML model, wherein the ML model is deployable with a computing service of the system, or wherein variable similarity data is presented in a user interface associated with performing the variable engineering for the ML model.
In the same field of endeavor, Brumbaugh teaches a machine learning platform including a feature engineering framework (“Here, we introduce Bighead, a framework-agnostic, end-to-end platform for machine learning. It offers a seamless user experience requiring only minimal efforts that span feature set management, prototyping, training, batch (offline) inference, realtime (online) inference, evaluation, and model lifecycle management. In contrast to existing platforms, it is designed to be highly versatile and extensible, and supports all major machine learning frameworks, rather than focusing on one particular framework” [Brumbaugh Abstract]; see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, incl. Feature Generation and Zipline component [Brumbaugh page 553]; “The lifecycle of a machine learning model typically starts with collecting datasets comprised of features for the model to train and perform inference on, followed by constant iterations on processing the features to extract relevant information, called feature engineering… Zipline is a framework built to tackle these challenges in feature management for machine learning projects. It provides a general-purpose feature processing, aggregation, and joining framework that guarantees point-in-time correctness for the feature sets” [Brumbaugh page 554 Zipline: Feature Management]) wherein a client application accessible by the system perform[s] the variable engineering for [the] ML model, and the ML model is deployable with a computing service of the system (see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, incl. Feature Generation and Zipline component [Brumbaugh page 553]; “To address the challenges in the domain of machine learning outlined in the previous sections, Bighead is designed to be: Seamless: Offer a streamlined user experience from prototyping to production, across different frameworks. Enable users to easily write, iterate on, and deploy readable, robust, and reproducible machine learning models. Provide integration with existing data infrastructure and make data access and storage straightforward” [Brumbaugh page 553 Design Goals and Architecture]; “Users only need to define features once, and Zipline makes them available in both inference and training environments, guaranteeing a single, consistent source of data for model consumption… Finally, Zipline exposes an API for users to define a feature set. This API includes 1) metadata, such as owner and documentation, 2) source of the feature set, including a schema and relevant properties, and 3) a list of features and how to aggregate each of them.” [Brumbaugh page 554-555 Zipline: Feature Management]; Zipline, the feature engineering component (i.e. client application), exists as a user-facing component within the deployable Bighead service) and wherein variable similarity data is presented in a user interface associated with performing a variable engineering for the ML model (“Built with the design goals in mind, Bighead consists of multiple components (Figure 1):… Bighead UI: A user interface that provides easy operations and visualizations for models (Section IX) [Brumbaugh page 553 Architecture]; “More specifically, users can carry over highly customizable metadata from training to the Bighead UI, such as evaluation metrics and visualizations. For example, users can explore the model performance in the UI to determine which models should be deployed, via feature importance plots, ROC curves, and precision-recall curves” [Brumbaugh page 557 Bighead UI]; see Fig. 6 – “Screenshots of UI showing an example Model, its Versions and Artifacts, and details of an Artifact”, including Feature importance (top 20) plot [Brumbaugh page 558])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated a system wherein a client application accessible by the system perform[s] the variable engineering for [the] ML model, and the ML model is deployable with a computing service of the system, and wherein variable similarity data is presented in a user interface associated with performing a variable engineering for the ML model as taught by Brumbaugh into Harris because they are both directed towards systems that utilize feature engineering frameworks for preprocessing input data to a machine learning model. Given that the Bighead platform is already explained to be highly versatile and extensible [Brumbaugh Abstract], a person of ordinary skill would be able to incorporate the teachings of Brumbaugh to thereby extend the autonomous learning platform of Harris into an operational, deployable end-to-end machine learning platform. Doing so would enable real-time deployment of the resulting platform in a consistent and reliable manner, thereby addressing scalability challenges and improving the user experience [Brumbaugh Abstract].
However, the combination does not expressly teach responsive to a selection of the option, replacing the first variable with the second variable in the ML model configuration.
In the same field of endeavor, Chatzimparmpas teaches a feature engineering visual analytics system (“Despite that, while several visual analytics tools exist to monitor and control the different stages of the ML life cycle (especially those related to data and algorithms), feature engineering support remains inadequate. In this paper, we present FeatureEnVi, a visual analytics system specifically designed to assist with the feature engineering process. Our proposed system helps users to choose the most important feature, to transform the original features into powerful alternatives, and to experiment with different feature generation combinations” [Chatzimparmpas Abstract]) that responsive to a selection of the option, replac[es] the first variable with the second variable in the ML model configuration (see Fig. 1 – “Selecting important features, transforming them, and generating new features with FeatureEnVi” [Chatzimparmpas page 1774]; “Also, instead of appending features progressively (called forward selection) or considering all features and then discarding some (known as backward elimination), we choose a stepwise selection approach. Therefore, we start with all features, but we can add or remove any number of features at different stages.” [Chatzimparmpas page 1774 Introduction]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated teach responsive to a selection of the option, replacing the first variable with the second variable in the ML model configuration as taught by Chatzimparmpas into Harris and Brumbaugh because they are all directed towards feature engineering systems, with Brumbaugh and Chatzimparmpas specifically being directed towards visual analytics. Incorporating the teachings of Chatzimparmpas would thereby allow for increased feature engineering support functions in the provided user interfaces [Chatzimparmpas Abstract].
However, the combination does not expressly teach storing cluster data in a device-side cache to be accessed in real-time during the performance of the variable engineering.
In the same field of endeavor, Tepper teaches a similarity-based feature detection procedure via clustering (“In process 100, input data 102 (e.g., an image of a couch) is provided. The input data 102 may be decomposed 104 into a high dimensional feature vector 106. The high dimensional feature vector 106 may be a representation of the input data 102. The high dimensional feature vector 106 may be modified from a second data format to a first data format that enables similarity searching… Thus, the modified vector 108 may be a representation of the high dimensional feature vector 106. Furthermore, as described below, the modified vector 108 may be compared against clusters in a database. For example, a database may be partitioned into clusters to reduce search speeds and times. The clusters may be in the form of the first data format to enable such similarity searches” [Tepper ¶ 0018-0020]) that stores cluster data in a device-side cache to be accessed in real-time during the performance of the variable engineering (“A clustering method that clusters the database and, simultaneously, computes cluster representatives (e.g., centroids) in Hamming space may be executed…The generation of clusters may be executed offline. The process 100 may then select a subset of clusters based on the modified vector 108, 110…The selector may be an SoC that includes one or more of a GPU, CPU, FPGA and/or other devices that have on-chip memories (e.g., caches) to store data and execute the aspects as described herein.” [Tepper ¶ 0021-0022]; “Example 1 includes a computing system including a data storage that is to store data associated with a plurality of clusters that are each in a first data format, a system-on-chip coupled to the data storage and that is to include a cache, a memory including a set of executable program instructions, which when executed by the system-on-chip, cause the computing system to store a plurality of classes that represent the plurality of clusters in the cache, wherein each of the classes is to represent a group of the plurality of clusters, modify input data from a second data format to the first data format, and conduct a similarity search based on the input data in the first data format to assign the input data to at least one class of the plurality of classes” [Tepper ¶ 0090]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated storing cluster data in a device-side cache to be accessed in real-time during the performance of the variable engineering as taught by Tepper into the combination because both Harris and Tepper are directed towards similarity-based feature detection procedures via clustering. Incorporating the teachings of Tepper would reduce cost of memory access and latency during query processing for subsequent similarity determinations (“The fast on-chip selector may exhibit a reduced memory footprint and allow for leveraging the benefits of a SAM, and fitting the selector data (e.g., the data the on-chip selector utilizes during similarity searching) in on-chip memory to reduce costly memory and data storage accesses. Thus, the similarity search process 100 may result in orders of magnitude faster searching based on an efficient and focused identification of partitions of a data store to explore for matches, and a focused Stochastic Associative Memory analysis (e.g., avoiding costly memory and data storage accesses) within the partitions” [Tepper ¶ 0016]).
Regarding claim 2, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 1, and Harris further teaches wherein the plurality of variable clusters includes the first cluster of variables and at least one second cluster of variables ([Harris ¶ 0066] as detailed in claim 1 above; “Features may be manually selected from the graph by one skilled in the art based on knowledge about the information space. This may done based on common sense rules and/or using any data mining technique, such as cluster analysis. For example, k-means clustering in combination with pre-defined rules (e.g. node Y: ‘male’ should not be of the same feature as node A: ‘female’) may be used to group nearby nodes into a single feature” [Harris ¶ 0059]). Tepper further teaches wherein the similarity detection is performed using one or more of the plurality of variable clusters stored in the device-side cache ([Tepper ¶ 0021-0022, 0090] as detailed in claim 1 above).
Regarding claim 3, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 2, and Harris further teaches wherein the ML clustering technique utilizes at least one of a k-nearest neighbors algorithm, a k-means algorithm, a means-shift algorithm, or a Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm with a vector similarity scoring between at least one of each of the plurality of variable vectors or nearest neighbors of the plurality of variable vectors ([Harris ¶ 0059] as detailed above; K-means clustering expressly may be used to create clusters)
Regarding claim 4, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 2, and Harris further teaches wherein the cluster representative vector for each of the plurality of variable clusters comprises one of an average of the plurality of variable vectors in each of the plurality of variable clusters or a representative one of the plurality of variable vectors in the each of the plurality of variable clusters ([Harris ¶ 0059] as detailed in claim 2 above; K-means clustering expressly calculates centroids (i.e., representative vector averages) for each cluster)
Regarding claim 5, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 2, and Harris further teaches wherein the determining the first cluster of variables matching the first variable is performed in real-time when generating variables and based on an input of at least one variable definition or variable description data for the new variable, ([Harris ¶ 0066] as detailed in claim 1 above; “The information space may include a graph of nodes having a defined topology, where most of the nodes correspond to different inputs and some of the nodes correspond to outputs. An input may be any known or pre-defined characteristic of the information space, and an output may be any predicted outcome that depends on one or more inputs. Examples of an input may include transaction data, user profile data, text in a conversation, etc. Examples of outputs may include a targeted recommendation, a probability of fraud, a response to initiate a purchase, etc.” [Harris ¶ 0062]). Tepper further teaches wherein the generating of the plurality of variable clusters is precomputed prior to the accessing the ML model configuration ([Tepper ¶ 0021-0022] as detailed in claim 1 above).
Regarding claim 6, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 1, and Harris further teaches displaying the variable similarity data in a list comprising at least one variable from the at least one of the first cluster of variables or each variable in the first cluster of variables, wherein the list includes variable usage data for the at least one variable ([Harris ¶ 0066, 0067] as detailed in claim 1 above; Wherein the selected clusters form the candidate features that are input for training, the features are further presented in a ranking based on the strength of their correlation (i.e., variable similarity) to a predicted outcome (i.e., their level of “usage” for generating said outcome)). Brumbaugh further teaches displaying variable similarity data in the user interface, wherein the list enables selection and viewing of the at least one variable ([Brumbaugh page 557 Bighead UI] as detailed in claim 1 above; see Fig. 6 – “Screenshots of UI showing an example Model, its Versions and Artifacts, and details of an Artifact”, including Feature importance (top 20) plot [Brumbaugh page 558]).
Regarding claim 7, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 1, and Harris further teaches wherein, prior to the determining the first cluster of variables, the operations further comprise:
generating the first variable vector based on at least one variable parameter for the first variable; (“An artificial intelligence model may learn from a set of collected data that is expressed as a graph characterized by nodes whose connection to one another are expressed as edges. Nodes within the graph of the information space may correspond to specific pieces of information collected from data. For example, for a given information space representing data collected in a transaction network, any given piece of information extracted from a transaction may be expressed as a node (e.g. account number, device ID, merchant category code, transaction type, etc.)” [Harris ¶ 0073]; Any collected parameters may be thereby expressed as nodes within the information space (i.e., variable vectors)) and
calculating the vector similarity using a vector similarity technique ([Harris ¶ 0066] as detailed in claim 1 above).
Regarding claim 8, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teach the limitations of parent claim 1, and Brumbaugh further teaches wherein the first variable and the plurality of variables each comprise a JavaScript Object Notation (JSON) container having variable logic comprising at least one of a source table, a transformation logic, a windowing parameter, or a filter definition (“Finally, Zipline exposes an API for users to define a feature set. This API includes 1) metadata, such as owner and documentation, 2) source of the feature set, including a schema and relevant properties, and 3) a list of features and how to aggregate each of them. The feature set definitions are stored in a repository so that they can be easily discovered, inspected, and shared. Below is an example of a feature set definition. Zipline is aware of the timestamp of each raw event (ts), the primary key (listing), the aggregation (sum), and time windows (7d, 14d, etc.) for each feature” [Brumbaugh page 555 Zipline: Feature Management]; See example feature set definition which is in JSON format and includes source –
PNG
media_image1.png
578
726
media_image1.png
Greyscale
[Brumbaugh page 555]).
Regarding claim 9, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teach the limitations of parent claim 1, and Harris further teaches wherein each of the plurality of variables is associated with a measurable datum usable by the ML model to generate an ML model output. ([Harris ¶ 0062] as detailed in claim 1 above; Input features may include measurable types of data (e.g., transaction data)). Brumbaugh further teaches wherein the operations further comprise receiving a selection of at least one variable parameter defining variable logic for the first variable during the performance of the variable engineering of the ML model, and wherein the presenting the variable similarity data occurs during the performance of the variable engineering based on the selection of the at least one variable parameter (see Fig. 6 – “Screenshots of UI showing an example Model, its Versions and Artifacts, and details of an Artifact”, including Feature importance (top 20) plot [Brumbaugh page 558] as detailed in claim 1 above; Selection of a UI screen parameter may enable a user to view training details including feature importance (i.e., variable similarity data)).
Regarding claim 21, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 1, and Harris further teaches training the ML model using the second variable in place of the first variable based on the replacing (“These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; As explained above, wherein previous input features and new features may form the candidate features that are input to the downstream model, the features set may be further capped (e.g., top 50 features) wherein some previous input features are replaced by new input features (including, e.g., those drawn from a first cluster) in the next training batch based on measured importance).
Regarding claim 11, Harris teaches A method (“Various embodiments are directed to performing autonomous learning for updating input features used for an artificial intelligence model. For example, a method can comprise receiving updated data of an information space that includes a graph of nodes having a defined topology, the updated data including historical data of requests to the artificial intelligence model and output results associated with the requests, wherein different categories of input data corresponds to different input nodes of the graph…In addition, the method may also include grouping input nodes to identify candidate input features to the artificial intelligence model, wherein the input node(s) of a candidate input feature are correlated to each other, and wherein the candidate input features have strength values above a threshold. Furthermore, the method may comprise training an enlarged artificial intelligence model using the historical data for the candidate input features, the training providing a ranking of candidate input features relative to a measure of effect on an output value of the enlarged artificial intelligence model, and selecting the top N candidate input features to be used in the artificial intelligence model.” [Harris ¶ 0005]; “Other embodiments are directed to systems, devices, and computer readable media associated with methods described herein” [Harris ¶ 0007]) comprising:
receiving a feature parameter for a feature usable by the ML model for an ML model output task; (“Rather than score individual nodes for predictiveness, the AI model may score a collection of nodes, known as features…Features may be manually selected from the graph by one skilled in the art based on knowledge about the information space” [Harris ¶ 0058-0059]; “The information space may include a graph of nodes having a defined topology, where most of the nodes correspond to different inputs and some of the nodes correspond to outputs. An input may be any known or pre-defined characteristic of the information space, and an output may be any predicted outcome that depends on one or more inputs. Examples of an input may include transaction data, user profile data, text in a conversation, etc. Examples of outputs may include a targeted recommendation, a probability of fraud, a response to initiate a purchase, etc.” [Harris ¶ 0062]) wherein the feature corresponds to a new feature to be created during a performance of a feature engineering for the ML model; (At step 430, the newly detected features may be smoothed (binned) to create a reduced feature set. The nodes of the graph may be grouped or formed into clusters, in which nodes that are close to each other in topology form a cluster and may have similar properties such as similar signal-to-noise ratio. Clusters with sufficiently high connection to a given output node may be identified as a possible input feature. A predetermined amount of input features may be added to a current set of input features to develop a new feature set or current version of the model. For example, 50 new input features can be added to a current set of 200 input features for the current version of the model These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066])
generating a feature vector for the feature based at least on the feature parameter; (“An artificial intelligence model may learn from a set of collected data that is expressed as a graph characterized by nodes whose connection to one another are expressed as edges. Nodes within the graph of the information space may correspond to specific pieces of information collected from data. For example, for a given information space representing data collected in a transaction network, any given piece of information extracted from a transaction may be expressed as a node (e.g. account number, device ID, merchant category code, transaction type, etc.)” [Harris ¶ 0073]; Any collected parameters may be thereby expressed as nodes within the information space (i.e., variable vectors)))
accessing a plurality of feature scores for a plurality of features preexisting with the ML model training system, wherein the plurality of feature scores are associated with a plurality of feature vectors for the plurality of features; (“Rather than score individual nodes for predictiveness, the AI model may score a collection of nodes, known as features…Features may be manually selected from the graph by one skilled in the art based on knowledge about the information space” [Harris ¶ 0058-0059]; “The information space may include a graph of nodes having a defined topology, where most of the nodes correspond to different inputs and some of the nodes correspond to outputs. An input may be any known or pre-defined characteristic of the information space, and an output may be any predicted outcome that depends on one or more inputs. Examples of an input may include transaction data, user profile data, text in a conversation, etc. Examples of outputs may include a targeted recommendation, a probability of fraud, a response to initiate a purchase, etc.” [Harris ¶ 0062]; “At step 430, the newly detected features may be smoothed (binned) to create a reduced feature set. The nodes of the graph may be grouped or formed into clusters, in which nodes that are close to each other in topology form a cluster and may have similar properties such as similar signal-to-noise ratio. Clusters with sufficiently high connection to a given output node may be identified as a possible input feature” [Harris ¶ 0066]; Wherein nodes of the topology graph (i.e., model configuration) represent features, an output node (i.e., first variable) may be identified for determining clusters with sufficiently high connection (i.e. scores) that comprise a group of nodes (i.e., plurality of feature vectors))
calculating, in real-time during the performance of the feature engineering and using the plurality of feature vectors, a weighted distance between the feature vector and each of the plurality of feature vectors; ; ([Harris ¶ 0066 as detailed above]; “An edge may be associated with a numerical value, known as a weight, that may be assigned to the pairwise connection between the two nodes. The edge weight may be identified as a strength of connectivity between two nodes and/or may be related to a cost or distance” [Harris ¶ 0048]; “ An edge weight may also be used to express a cost or a distance required to move from one state or node to the next” [Harris ¶ 0024])
determining a first one of the plurality of features having a corresponding one of the weighted distances within a threshold distance to the feature vector; (“A predetermined amount of input features may be added to a current set of input features to develop a new feature set or current version of the model. For example, 50 new input features can be added to a current set of 200 input features for the current version of the model. The 50 new features can be identified based on the connection strength to the selected output nodes” [Harris ¶ 0066])
presenting the first one of the plurality of features with an option to replace the feature with the first one of the plurality of features in place of creating the feature as the new feature for the ML model ([Harris ¶ 0066, 0067] as detailed above; Wherein previous input features and new features may form the candidate features that are input to the downstream model, the features set may be further capped (e.g., top 50 features) wherein some previous input features are replaced by new input features (including, e.g., those drawn from a first cluster) in the next training batch based on measured importance).
However, Harris does not expressly teach presenting a user interface comprising a plurality of options for a machine learning (ML) model configuration of an ML model by an ML model training system, or presenting the first one of the plurality of features in the user interface.
In the same field of endeavor, Brumbaugh teaches a machine learning platform including a feature engineering framework (“Here, we introduce Bighead, a framework-agnostic, end-toend platform for machine learning. It offers a seamless user experience requiring only minimal efforts that span feature set management, prototyping, training, batch (offline) inference, realtime (online) inference, evaluation, and model lifecycle management. In contrast to existing platforms, it is designed to be highly versatile and extensible, and supports all major machine learning frameworks, rather than focusing on one particular framework” [Brumbaugh Abstract]; see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, incl. Feature Generation and Zipline component [Brumbaugh page 553]; “The lifecycle of a machine learning model typically starts with collecting datasets comprised of features for the model to train and perform inference on, followed by constant iterations on processing the features to extract relevant information, called feature engineering… Zipline is a framework built to tackle these challenges in feature management for machine learning projects. It provides a general-purpose feature processing, aggregation, and joining framework that guarantees point-in-time correctness for the feature sets” [Brumbaugh page 554 Zipline: Feature Management]) presenting a user interface comprising a plurality of options for a machine learning (ML) model configuration of an ML model by an ML model training system, and presenting features in the user interface (“Built with the design goals in mind, Bighead consists of multiple components (Figure 1):… Bighead UI: A user interface that provides easy operations and visualizations for models (Section IX) [Brumbaugh page 553 Architecture]; “More specifically, users can carry over highly customizable metadata from training to the Bighead UI, such as evaluation metrics and visualizations. For example, users can explore the model performance in the UI to determine which models should be deployed, via feature importance plots, ROC curves, and precision-recall curves” [Brumbaugh page 557 Bighead UI]; see Fig. 6 – “Screenshots of UI showing an example Model, its Versions and Artifacts, and details of an Artifact”, including Feature importance (top 20) plot [Brumbaugh page 558]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated presenting a user interface comprising a plurality of options for a machine learning (ML) model configuration of an ML model by an ML model training system and presenting the first one of the plurality of features in the user interface. as taught by Brumbaugh into Harris because they are both directed towards systems that utilize feature engineering frameworks for preprocessing input data to a machine learning model. Given that the Bighead platform is already explained to be highly versatile and extensible [Brumbaugh Abstract], a person of ordinary skill would be able to incorporate the teachings of Brumbaugh to thereby extend the autonomous learning platform of Harris into an operational, deployable end-to-end machine learning platform. Doing so would enable real-time deployment of the resulting platform in a consistent and reliable manner, thereby addressing scalability challenges and improving the user experience [Brumbaugh Abstract].
However, the combination does not expressly teach responsive to a selection of the option, replacing the feature with the first one of the plurality of features in the ML model configuration.
In the same field of endeavor, Chatzimparmpas teaches a feature engineering visual analytics system (“Despite that, while several visual analytics tools exist to monitor and control the different stages of the ML life cycle (especially those related to data and algorithms), feature engineering support remains inadequate. In this paper, we present FeatureEnVi, a visual analytics system specifically designed to assist with the feature engineering process. Our proposed system helps users to choose the most important feature, to transform the original features into powerful alternatives, and to experiment with different feature generation combinations” [Chatzimparmpas Abstract]) that responsive to a selection of the option, replac[es] the feature with the first one of the plurality of features in the ML model configuration (see Fig. 1 – “Selecting important features, transforming them, and generating new features with FeatureEnVi” [Chatzimparmpas page 1774]; “Also, instead of appending features progressively (called forward selection) or considering all features and then discarding some (known as backward elimination), we choose a stepwise selection approach. Therefore, we start with all features, but we can add or remove any number of features at different stages.” [Chatzimparmpas page 1774 Introduction]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated responsive to a selection of the option, replacing the feature with the first one of the plurality of features in the ML model configuration. as taught by Chatzimparmpas into Harris and Brumbaugh because they are all directed towards feature engineering systems, with Brumbaugh and Chatzimparmpas specifically being directed towards visual analytics. Incorporating the teachings of Chatzimparmpas would thereby allow for increased feature engineering support functions in the provided user interfaces [Chatzimparmpas Abstract].
However, the combination does not expressly teach precomput[ing] and stor[ing] feature scores and feature vectors in a device-side cache to be accessed in real-time during the performance of the variable engineering.
In the same field of endeavor, Tepper teaches a similarity-based feature detection procedure via clustering (“In process 100, input data 102 (e.g., an image of a couch) is provided. The input data 102 may be decomposed 104 into a high dimensional feature vector 106. The high dimensional feature vector 106 may be a representation of the input data 102. The high dimensional feature vector 106 may be modified from a second data format to a first data format that enables similarity searching… Thus, the modified vector 108 may be a representation of the high dimensional feature vector 106. Furthermore, as described below, the modified vector 108 may be compared against clusters in a database. For example, a database may be partitioned into clusters to reduce search speeds and times. The clusters may be in the form of the first data format to enable such similarity searches” [Tepper ¶ 0018-0020]) that precomput[es] and stor[es] feature scores and feature vectors in a device-side cache to be accessed in real-time during the performance of the variable engineering (“A clustering method that clusters the database and, simultaneously, computes cluster representatives (e.g., centroids) in Hamming space may be executed…The generation of clusters may be executed offline. The process 100 may then select a subset of clusters based on the modified vector 108, 110…The selector may be an SoC that includes one or more of a GPU, CPU, FPGA and/or other devices that have on-chip memories (e.g., caches) to store data and execute the aspects as described herein.” [Tepper ¶ 0021-0022]; “Example 1 includes a computing system including a data storage that is to store data associated with a plurality of clusters that are each in a first data format, a system-on-chip coupled to the data storage and that is to include a cache, a memory including a set of executable program instructions, which when executed by the system-on-chip, cause the computing system to store a plurality of classes that represent the plurality of clusters in the cache, wherein each of the classes is to represent a group of the plurality of clusters, modify input data from a second data format to the first data format, and conduct a similarity search based on the input data in the first data format to assign the input data to at least one class of the plurality of classes” [Tepper ¶ 0090]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated precomput[ing] and stor[ing] feature scores and feature vectors in a device-side cache to be accessed in real-time during the performance of the variable engineering as taught by Tepper into the combination because both Harris and Tepper are directed towards similarity-based feature detection procedures via clustering. Incorporating the teachings of Tepper would reduce cost of memory access and latency during query processing for subsequent similarity determinations (“The fast on-chip selector may exhibit a reduced memory footprint and allow for leveraging the benefits of a SAM, and fitting the selector data (e.g., the data the on-chip selector utilizes during similarity searching) in on-chip memory to reduce costly memory and data storage accesses. Thus, the similarity search process 100 may result in orders of magnitude faster searching based on an efficient and focused identification of partitions of a data store to explore for matches, and a focused Stochastic Associative Memory analysis (e.g., avoiding costly memory and data storage accesses) within the partitions” [Tepper ¶ 0016]).
Regarding claim 12, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 11, and Brumbaugh further teaches wherein the feature parameter is extracted from a data container for the feature or selected during the performance of the feature engineering for the ML model (see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, including External Feature Sources which are fetched for Zipline [Brumbaugh page 553]).
Regarding claim 13, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 12, and Harris further teaches wherein the feature parameter comprises one of a plurality of feature parameters for a configuration of the feature, ([Harris ¶ 0062] as detailed in claim 11 above; Input features may include configuration data) and wherein the generating the feature vector is further based on the plurality of feature parameters ([Harris ¶ 0073] as detailed in claim 11 above; Any collected parameters may be thereby expressed as nodes within the information space (i.e., variable vectors)).
Regarding claim 14, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 11, and Brumbaugh further teaches wherein the feature comprises a JavaScript Object Notation (JSON) data structure comprising the feature parameter, and wherein the feature parameter comprises one of source data for the feature or processing logic for the feature (“Finally, Zipline exposes an API for users to define a feature set. This API includes 1) metadata, such as owner and documentation, 2) source of the feature set, including a schema and relevant properties, and 3) a list of features and how to aggregate each of them. The feature set definitions are stored in a repository so that they can be easily discovered, inspected, and shared. Below is an example of a feature set definition. Zipline is aware of the timestamp of each raw event (ts), the primary key (listing), the aggregation (sum), and time windows (7d, 14d, etc.) for each feature” [Brumbaugh page 555 Zipline: Feature Management]; See example feature set definition which is in JSON format and includes source –
PNG
media_image1.png
578
726
media_image1.png
Greyscale
[Brumbaugh page 555]).
Regarding claim 15, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 11, and Harris further teaches accessing the plurality of features for the ML model training system; ([Harris ¶ 0058-0059,0062] as detailed in claim 11 above).
calculating the plurality of feature vectors for the plurality of features based on feature parameters for the plurality of features; ([Harris ¶ 0073] as detailed in claim 11 above; Any collected parameters may be thereby expressed as nodes within the information space (i.e., feature vectors))
clustering the plurality of features by the plurality of feature vectors using an ML clustering technique; ([Harris ¶ 0066] as detailed in claim 11 above; Wherein nodes of the topology graph represent features, an output node may be identified for determining clusters with sufficiently high connection (i.e. similarity) that comprise a group of nodes (i.e., plurality of variables)) and
generating the plurality of feature scores for a plurality of feature clusters from the plurality of features ([Harris ¶ 0048, 0024, 0066] as detailed in claim 11 above; For a selected output node, a selection of clusters each representing new input features can be identified based on relative connection strength (i.e., feature score)).
Tepper further teaches storing the plurality of feature scores and the plurality of feature vectors in the device-side cache ([Tepper ¶ 0021-0022, 0090] as detailed in claim 11 above)
Regarding claim 16, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 15, and Harris further teaches generating a cluster representative for each of the plurality of feature clusters representing one or more of the plurality of feature vectors in each of the plurality of feature clusters, wherein the plurality of feature scores are associated with the cluster representative (“Features may be manually selected from the graph by one skilled in the art based on knowledge about the information space. This may done based on common sense rules and/or using any data mining technique, such as cluster analysis. For example, k-means clustering in combination with pre-defined rules (e.g. node Y: ‘male’ should not be of the same feature as node A: ‘female’) may be used to group nearby nodes into a single feature” [Harris ¶ 0059]; K-means clustering expressly calculates centroids (i.e., representative vectors) for each cluster. For a selected output node (i.e., first variable), a selection of clusters each representing new input features can be identified based on relative connection strength (i.e., feature score), wherein connection strength may be further measured through edge weights representing any form of distance between nodes (e.g., Euclidean distance to cluster centroid)).
Regarding claim 17, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 11, and Brumbaugh further teaches wherein the first one of the plurality of features is presented in the user interface with summary indicating a similarity of one or more parts of a definitive function of the feature with the first one of the plurality of features (see Fig. 6 – “Screenshots of UI showing an example Model, its Versions and Artifacts, and details of an Artifact”, including Feature importance (top 20) plot [Brumbaugh page 558] as detailed in claim 11 above; Selection of a UI screen parameter may enable a user to view training details including feature importance (i.e., variable similarity data))
Regarding claim 18, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 17, and Brumbaugh further teaches wherein the user interface further comprises one or more menus that enable an establishment of at least one of source tables for the feature parameter or logic definitions for the feature parameter when creating the feature as the new feature for the ML model (“Users only need to define features once, and Zipline makes them available in both inference and training environments, guaranteeing a single, consistent source of data for model consumption” [Brumbaugh page 554 Zipline: Feature Management]; see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, including External Feature Sources which are fetched for Zipline [Brumbaugh page 553])
Regarding claim 19, Harris teaches A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations (“Various embodiments are directed to performing autonomous learning for updating input features used for an artificial intelligence model. For example, a method can comprise receiving updated data of an information space that includes a graph of nodes having a defined topology, the updated data including historical data of requests to the artificial intelligence model and output results associated with the requests, wherein different categories of input data corresponds to different input nodes of the graph…In addition, the method may also include grouping input nodes to identify candidate input features to the artificial intelligence model, wherein the input node(s) of a candidate input feature are correlated to each other, and wherein the candidate input features have strength values above a threshold. Furthermore, the method may comprise training an enlarged artificial intelligence model using the historical data for the candidate input features, the training providing a ranking of candidate input features relative to a measure of effect on an output value of the enlarged artificial intelligence model, and selecting the top N candidate input features to be used in the artificial intelligence model.” [Harris ¶ 0005]; “Other embodiments are directed to systems, devices, and computer readable media associated with methods described herein” [Harris ¶ 0007]) comprising:
receiving a plurality of machine learning (ML) features associated with at least one ML model, wherein each of the plurality of ML features is associated with a measurable datum used by one or more of the at least one ML model for ML model outputs ((“In some implementations, an information space can be analyzed to determine suitable features for training the AI model. The information space may be represented as a topological graph of nodes connected by edges, which can have assigned weight, distance, or cost” [Harris ¶ 0031]; “FIG. 3 shows a depiction of selecting input features from a topological graph. In graph 300, each node (A through G, S through Z) may represent a specific set (e.g., one piece or a collection) of information determined from collected data….The information of graph 300 may be used as training data for an artificial intelligence model” [Harris ¶ 0055, 0057]; “An input may be any known or pre-defined characteristic of the information space, and an output may be any predicted outcome that depends on one or more inputs. Examples of an input may include transaction data, user profile data, text in a conversation, etc. Examples of outputs may include a targeted recommendation, a probability of fraud, a response to initiate a purchase, etc.” [Harris ¶ 0062]; A topological graph (i.e., configuration) is accessed to obtain features (incl. e.g. measurable data) which are used as training data for an AI model (i.e., associated with an ML model));
converting feature description data for each of the plurality of ML features to a corresponding vector representable in a vector space; (“An artificial intelligence model may learn from a set of collected data that is expressed as a graph characterized by nodes whose connection to one another are expressed as edges. Nodes within the graph of the information space may correspond to specific pieces of information collected from data. For example, for a given information space representing data collected in a transaction network, any given piece of information extracted from a transaction may be expressed as a node (e.g. account number, device ID, merchant category code, transaction type, etc.)” [Harris ¶ 0073]; Any collected parameters may be thereby expressed as nodes within the information space (i.e., variable vectors)))
clustering the corresponding vectors for the plurality of ML features in the vector space using an ML clustering technique, wherein the ML clustering technique utilizes similarity scores between the corresponding vectors; ((“Clusters with sufficiently high connection to a given output node may be identified as a possible input feature. A predetermined amount of input features may be added to a current set of input features to develop a new feature set or current version of the model. For example, 50 new input features can be added to a current set of 200 input features for the current version of the model. The 50 new features can be identified based on the connection strength to the selected output nodes by, for example, selecting the top 50 features based on a shortest distance from an output node. These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; “An edge may be associated with a numerical value, known as a weight, that may be assigned to the pairwise connection between the two nodes. The edge weight may be identified as a strength of connectivity between two nodes and/or may be related to a cost or distance” [Harris ¶ 0048]; “ An edge weight may also be used to express a cost or a distance required to move from one state or node to the next” [Harris ¶ 0024])
generating a cluster representative vector for each cluster from a plurality of clusters resulting from the clustering; (Features may be manually selected from the graph by one skilled in the art based on knowledge about the information space. This may done based on common sense rules and/or using any data mining technique, such as cluster analysis. For example, k-means clustering in combination with pre-defined rules (e.g. node Y: ‘male’ should not be of the same feature as node A: ‘female’) may be used to group nearby nodes into a single feature” [Harris ¶ 0059]; K-means clustering expressly calculates centroids (i.e., representative vectors) for each cluster))
storing cluster comparison data comprising the cluster representative vectors and the plurality of clusters, wherein the cluster comparison data enables use with a feature engineering operation (“These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; The clusters (and associated features) are stored for input to the downstream model).
receiving, during the feature engineering operation, a new feature to be created for the at least one ML model; (“The information space may include a graph of nodes having a defined topology, where most of the nodes correspond to different inputs and some of the nodes correspond to outputs. An input may be any known or pre-defined characteristic of the information space, and an output may be any predicted outcome that depends on one or more inputs. Examples of an input may include transaction data, user profile data, text in a conversation, etc. Examples of outputs may include a targeted recommendation, a probability of fraud, a response to initiate a purchase, etc.” [Harris ¶ 0062]; “At step 430, the newly detected features may be smoothed (binned) to create a reduced feature set. The nodes of the graph may be grouped or formed into clusters, in which nodes that are close to each other in topology form a cluster and may have similar properties such as similar signal-to-noise ratio. Clusters with sufficiently high connection to a given output node may be identified as a possible input feature. A predetermined amount of input features may be added to a current set of input features to develop a new feature set or current version of the model. For example, 50 new input features can be added to a current set of 200 input features for the current version of the model These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; Wherein current nodes of the topology graph (i.e., model configuration) represent features, an output node may be identified for determining clusters of nodes with sufficiently high connection (i.e. similarity), and features of these nodes may be newly selected as input features (i.e., new variables), in conjunction with the existing current set of input features (i.e., including first variable) for training the model (i.e., determined to be predictive/suitable features during variable engineering process))
determining, in real-time during the feature engineering operation and using the cluster comparison data, a first cluster of the plurality of clusters matching the new feature based on a vector similarity between a new feature vector for the new feature and the cluster representative vector for the first cluster; (“Clusters with sufficiently high connection to a given output node may be identified as a possible input feature. A predetermined amount of input features may be added to a current set of input features to develop a new feature set or current version of the model. For example, 50 new input features can be added to a current set of 200 input features for the current version of the model. The 50 new features can be identified based on the connection strength to the selected output nodes by, for example, selecting the top 50 features based on a shortest distance from an output node. These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; “An edge may be associated with a numerical value, known as a weight, that may be assigned to the pairwise connection between the two nodes. The edge weight may be identified as a strength of connectivity between two nodes and/or may be related to a cost or distance” [Harris ¶ 0048]; “ An edge weight may also be used to express a cost or a distance required to move from one state or node to the next” [Harris ¶ 0024]; For a selected output node corresponding to the current set of input features (incl. first variable), a selection of clusters (incl. first cluster) each representing new input features can be identified (i.e., matched) based on relative connection strength (i.e., similarity), wherein connection strength may be further measured through edge weights representing any form of distance between nodes (e.g., Euclidean distance between vectors));
presenting an existing ML feature of the plurality of ML features from the first cluster with an option to replace the new feature with the existing ML feature in place of creating the new feature for the at least one ML model; ([Harris ¶ 0066] as detailed above; “At step 440, the new feature set may be used to retrain the model using the new features. The training of the enlarged artificial intelligence model may be performed using the historical data for candidate input features, and the training may provide a ranking of the candidate input features relative to a measure of their effect on an output value of the enlarged artificial intelligence model… In one embodiment, the features in the enlarged feature set may be ranked according to strength of correlation to a predicted outcome, and the feature set may then be capped to its original size (e.g. ranking 250 features and then selecting only the top 200 features as the new feature set)” [Harris ¶ 0067]; Wherein previous input features and new features may form the candidate features that are input to the downstream model, the features set may be further capped (e.g., top 50 features) wherein some previous input features are replaced by new input features (including, e.g., those drawn from a first cluster) in the next training batch based on measured importance).
However, Harris does not expressly teach the ML model being deployed with decision services of a server provider system or the recited feature engineering procedure being an operation of an application.
In the same field of endeavor, Brumbaugh teaches a machine learning platform including a feature engineering framework (“Here, we introduce Bighead, a framework-agnostic, end-toend platform for machine learning. It offers a seamless user experience requiring only minimal efforts that span feature set management, prototyping, training, batch (offline) inference, realtime (online) inference, evaluation, and model lifecycle management. In contrast to existing platforms, it is designed to be highly versatile and extensible, and supports all major machine learning frameworks, rather than focusing on one particular framework” [Brumbaugh Abstract]; see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, incl. Feature Generation and Zipline component [Brumbaugh page 553]; “The lifecycle of a machine learning model typically starts with collecting datasets comprised of features for the model to train and perform inference on, followed by constant iterations on processing the features to extract relevant information, called feature engineering… Zipline is a framework built to tackle these challenges in feature management for machine learning projects. It provides a general-purpose feature processing, aggregation, and joining framework that guarantees point-in-time correctness for the feature sets” [Brumbaugh page 554 Zipline: Feature Management]) wherein the ML model is deployed with decision services of a server provider system, the recited feature engineering procedure being an operation of an application (see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, incl. Feature Generation and Zipline component [Brumbaugh page 553]; “To address the challenges in the domain of machine learning outlined in the previous sections, Bighead is designed to be: Seamless: Offer a streamlined user experience from prototyping to production, across different frameworks. Enable users to easily write, iterate on, and deploy readable, robust, and reproducible machine learning models. Provide integration with existing data infrastructure and make data access and storage straightforward” [Brumbaugh page 553 Design Goals and Architecture]; Zipline, the feature engineering component, exists as a component within the deployable Bighead service).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated a system wherein the ML model is deployed with decision services of a server provider system, the recited feature engineering procedure being an operation of an application as taught by Brumbaugh into Harris because they are both directed towards systems that utilize feature engineering frameworks for preprocessing input data to a machine learning model. Given that the Bighead platform is already explained to be highly versatile and extensible [Brumbaugh Abstract], a person of ordinary skill would be able to incorporate the teachings of Brumbaugh to thereby extend the autonomous learning platform of Harris into an operational, deployable end-to-end machine learning platform. Doing so would enable real-time deployment of the resulting platform in a consistent and reliable manner, thereby addressing scalability challenges and improving the user experience [Brumbaugh Abstract].
However, the combination does not expressly teach responsive to a selection of the option, replacing the new feature with the existing feature for the at least one ML model.
In the same field of endeavor, Chatzimparmpas teaches a feature engineering visual analytics system (“Despite that, while several visual analytics tools exist to monitor and control the different stages of the ML life cycle (especially those related to data and algorithms), feature engineering support remains inadequate. In this paper, we present FeatureEnVi, a visual analytics system specifically designed to assist with the feature engineering process. Our proposed system helps users to choose the most important feature, to transform the original features into powerful alternatives, and to experiment with different feature generation combinations” [Chatzimparmpas Abstract]) that responsive to a selection of the option, replac[es] the new feature with the existing feature for the at least one ML model (see Fig. 1 – “Selecting important features, transforming them, and generating new features with FeatureEnVi” [Chatzimparmpas page 1774]; “Also, instead of appending features progressively (called forward selection) or considering all features and then discarding some (known as backward elimination), we choose a stepwise selection approach. Therefore, we start with all features, but we can add or remove any number of features at different stages.” [Chatzimparmpas page 1774 Introduction]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated teach responsive to a selection of the option, replacing the new feature with the existing feature for the at least one ML model as taught by Chatzimparmpas into Harris and Brumbaugh because they are all directed towards feature engineering systems, with Brumbaugh and Chatzimparmpas specifically being directed towards visual analytics. Incorporating the teachings of Chatzimparmpas would thereby allow for increased feature engineering support functions in the provided user interfaces [Chatzimparmpas Abstract].
However, the combination does not expressly teach storing cluster comparison data in a device-side cache to be accessed in real-time during the feature engineering operation.
In the same field of endeavor, Tepper teaches a similarity-based feature detection procedure via clustering (“In process 100, input data 102 (e.g., an image of a couch) is provided. The input data 102 may be decomposed 104 into a high dimensional feature vector 106. The high dimensional feature vector 106 may be a representation of the input data 102. The high dimensional feature vector 106 may be modified from a second data format to a first data format that enables similarity searching… Thus, the modified vector 108 may be a representation of the high dimensional feature vector 106. Furthermore, as described below, the modified vector 108 may be compared against clusters in a database. For example, a database may be partitioned into clusters to reduce search speeds and times. The clusters may be in the form of the first data format to enable such similarity searches” [Tepper ¶ 0018-0020]) that stores cluster comparison data in a device-side cache to be accessed in real-time during the feature engineering operation. (“A clustering method that clusters the database and, simultaneously, computes cluster representatives (e.g., centroids) in Hamming space may be executed…The generation of clusters may be executed offline. The process 100 may then select a subset of clusters based on the modified vector 108, 110…The selector may be an SoC that includes one or more of a GPU, CPU, FPGA and/or other devices that have on-chip memories (e.g., caches) to store data and execute the aspects as described herein.” [Tepper ¶ 0021-0022]; “Example 1 includes a computing system including a data storage that is to store data associated with a plurality of clusters that are each in a first data format, a system-on-chip coupled to the data storage and that is to include a cache, a memory including a set of executable program instructions, which when executed by the system-on-chip, cause the computing system to store a plurality of classes that represent the plurality of clusters in the cache, wherein each of the classes is to represent a group of the plurality of clusters, modify input data from a second data format to the first data format, and conduct a similarity search based on the input data in the first data format to assign the input data to at least one class of the plurality of classes” [Tepper ¶ 0090]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated storing cluster comparison data in a device-side cache to be accessed in real-time during the feature engineering operation as taught by Tepper into the combination because both Harris and Tepper are directed towards similarity-based feature detection procedures via clustering. Incorporating the teachings of Tepper would reduce cost of memory access and latency during query processing for subsequent similarity determinations (“The fast on-chip selector may exhibit a reduced memory footprint and allow for leveraging the benefits of a SAM, and fitting the selector data (e.g., the data the on-chip selector utilizes during similarity searching) in on-chip memory to reduce costly memory and data storage accesses. Thus, the similarity search process 100 may result in orders of magnitude faster searching based on an efficient and focused identification of partitions of a data store to explore for matches, and a focused Stochastic Associative Memory analysis (e.g., avoiding costly memory and data storage accesses) within the partitions” [Tepper ¶ 0016]).
Regarding claim 22, the combination of Harris, Brumbaugh, Chatzimparmpas, and Tepper teaches the limitations of parent claim 19, and Harris further teaches training the at least one ML model using the existing ML feature in place of the new feature based on the replacing (“These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; As explained above, wherein previous input features and new features may form the candidate features that are input to the downstream model, the features set may be further capped (e.g., top 50 features) wherein some previous input features are replaced by new input features (including, e.g., those drawn from a first cluster) in the next training batch based on measured importance). Brumbaugh further teaches deploying the at least one ML model trained using the existing ML feature with the decision services of the server provider system (see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, incl. Feature Generation and Zipline component [Brumbaugh page 553]; “To address the challenges in the domain of machine learning outlined in the previous sections, Bighead is designed to be: Seamless: Offer a streamlined user experience from prototyping to production, across different frameworks. Enable users to easily write, iterate on, and deploy readable, robust, and reproducible machine learning models. Provide integration with existing data infrastructure and make data access and storage straightforward” [Brumbaugh page 553 Design Goals and Architecture]; Zipline, the feature engineering component, exists as a component within the deployable Bighead service).
Response to Amendment and Arguments
The amendment filed 08/04/2026 has been entered.
Applicant’s amendment to the claims with respect to resolving objections has been considered, and the objections are consequently withdrawn.
The remarks filed 08/04/2026 have been fully considered.
Applicant’s remarks traversing the subject matter eligibility rejections under 35 U.S.C. 101 set forth in the office action mailed 05/04/2026, in view of claims 1-9, 11-19, and 21-22 as amended, have been considered and are persuasive.
Consequently, the rejections are withdrawn.
Applicant’s remarks traversing the prior art rejections under 35 U.S.C. 103 set forth in the office action mailed 05/04/2026, in view of claims 1-9, 11-19, and 21-22 as amended, have been considered, but are moot because the new grounds of rejection set forth above does not rely on the reference(s) applied in the prior rejection of record for the subject matter being specifically challenged in applicant' s argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VIJAY M BALAKRISHNAN whose telephone number is (571) 272-0455. The examiner can normally be reached 10am-5pm EST Mon-Thurs.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JENNIFER WELCH can be reached on (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/V.M.B./
Examiner, Art Unit 2143
/JENNIFER N WELCH/Supervisory Patent Examiner, Art Unit 2143
Prosecution Insights