DETAILED ACTION
This nonfinal action is in response to application 18/325,844 filed on 05/30/2023.
Claims 1-20 remain pending in the application. Claims 1, 11, and 19 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 5, 12-14 and 17 are objected to because of the following informalities:
In claim 5, “wherein the generating is precomputed” should read “wherein the generating of variable clusters is precomputed” to make clearer what “the generating” is in reference to.
In claim 12, “wherein the feature parameter is extracted from a data container for the feature or selected for the feature parameter during a feature engineering for the ML model” should read “wherein the feature parameter is extracted from a data container for the feature or selected during a feature engineering for the ML model” to avoid redundant language and improve clarity.
In claim 13, “wherein the feature parameter comprises one of a plurality of feature parameters for the feature in a plurality of feature fields for a configuration of the feature” should read “wherein the feature parameter comprises one of a plurality of feature parameters for a configuration of the feature” or be likewise amended to avoid redundant language and improve clarity.
In claim 14, “processing logic for feature” should read “processing logic for the feature” to correct an apparent typographical error.
In claim 17, “wherein the first one of the plurality of features is presented in the user interface with an option to replace the feature” should read “wherein the first one of the plurality of features is presented in the user interface as an option to replace the feature” to improve clarity and be consistent with description in the specification [¶ 0067].
Appropriate corrections are required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50 (“2019 PEG”).
Independent Claims (Claim 1, Claim 11, Claim 19):
Step 1: Claim 1 is drawn to a system/apparatus, claim 11 is drawn to a method, and claim 19 is drawn to a product. Therefore, each of these claims falls under one of the four categories of statutory subject matter (process/method, machine/apparatus, manufacture/product, or composition of matter).
Step 2A Prong 1: Claims 1, 11, and 19 each recite a judicially recognized exception of an abstract idea.
Claim 1 recites, inter alia:
determining a first variable from a plurality of variables that enables a similarity detection; determining a first cluster of variables matching the first variable based on a vector similarity for a first variable vector representing the first variable and a cluster representative vector representing the first cluster of variables; and presenting variable similarity data for at least one of the first cluster of variables or each variable in the first cluster of variables – This limitation recites a comparison of variables through dimensional vector representations, including dimensional vectors grouped based on proximity in vector space (cluster), and calculating “similarities” between vectors (e.g., distances) for further analysis, and thereby amounts to an abstract procedure of analysis that expressly recites mathematical calculation.
Claim 11 recites, inter alia:
generating a feature vector for the feature based at least on the feature parameter; accessing a plurality of feature scores for a plurality of features, wherein the plurality of feature scores are associated with a plurality of feature vectors for the plurality of features; calculating a weighted distance between the feature vector and each of the plurality of feature vectors; determining a first one of the plurality of features having a corresponding one of the weighted distances within a threshold distance to the feature vector; and presenting the first one of the plurality of features – This limitation recites a comparison of variables through dimensional vector (feature vector) representations and numerical (feature score) representations, and calculating weighted distances between features, and further comparing calculated distances to a numerical threshold. It thereby amounts to an abstract procedure of analysis that expressly recites mathematical calculation.
Claim 19 recites, inter alia:
converting feature description data for each of the plurality of ML features to a
corresponding vector representable in a vector space; clustering the corresponding vectors for the plurality of ML features in the vector space using similarity scores between the corresponding vectors; generating a cluster representative vector for each cluster from a plurality of clusters resulting from the clustering; and storing cluster comparison data comprising the cluster representative vectors and the plurality of clusters – This limitation recites a comparison of variables through dimensional vector representations, including dimensional vectors grouped based on proximity in vector space (clusters), and calculating “scores” of similarity (e.g., distances) between vectors, and thereby amounts to an abstract procedure of analysis that expressly recites mathematical calculation.
Step 2A Prong 2: The following additional elements recited in claims 1, 11, and 19 do not integrate the recited judicial exceptions into a practical application.
Claim 1 additionally recites:
A system comprising: a non-transitory memory; and one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations – These limitations amount to mere instructions to implement an abstract idea on a computer or computer components.
accessing a machine learning (ML) model configuration associated with an ML model deployable with a computing service of the system; [enabling a similarity detection] based on the ML model configuration; [presenting variable similarity data in a user interface] associated with performing a variable engineering for the ML model – These limitations do no more than generally link an abstract procedure of analysis and calculation to the field of use of data preprocessing for machine learning (ML) models.
Claim 11 additionally recites:
presenting a user interface comprising a plurality of options for a machine learning (ML) model configuration of an ML model by an ML model training system; receiving a feature parameter for a feature usable by the ML model for an ML model output task; [accessing features scores for a plurality of features] preexisting with the ML model training system; [presenting features] in the user interface – These limitations do no more than generally link an abstract procedure of analysis and calculation to the field of use of data preprocessing for machine learning (ML) models, as well as recite insignificant pre- and post- solution steps of gathering and outputting features which are insignificant extra-solution activity. They further merely invoke generic computer components (user interface) as tools to receive and transmit data for enabling abstract analysis.
Claim 19 additionally recites:
A non-transitory machine-readable medium having stored thereon machine- readable instructions executable to cause a machine to perform operations – This limitation amounts to mere instructions to implement an abstract idea on a computer or computer components.
receiving a plurality of machine learning (ML) features associated with at least one ML model deployed with decision services of a server provider system, wherein each of the plurality of ML features is associated with a measurable datum used by one or more of the at least one ML model for ML model outputs; [clustering corresponding vectors in the vector space] using an ML clustering technique – These limitations do no more than generally link an abstract procedure of analysis and calculation to the field of use of data preprocessing for machine learning (ML) models, as well as recite insignificant pre- solution steps of gathering features which is insignificant extra-solution activity.
storing [data] that enables use with a feature engineering operation of an application – This limitation does no more than further recite insignificant implementation of data gathering and storage via a computer, and thereby amounts to insignificant extra-solution activity.
Step 2B: The additional elements recited in claims 1, 11, and 19, viewed individually or as an ordered combination, do not provide an inventive concept or otherwise amount to significantly more than the recited abstract ideas themselves.
Claim 1 additionally recites:
A system comprising: a non-transitory memory; and one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations – Mere instructions to implement an abstract idea on a computer or computer components do not provide an inventive concept or significantly more to the recited abstract idea.
accessing a machine learning (ML) model configuration associated with an ML model deployable with a computing service of the system; [enabling a similarity detection] based on the ML model configuration; [presenting variable similarity data in a user interface] associated with performing a variable engineering for the ML model – Generally linking an abstract procedure of analysis and calculation to the field of use of data preprocessing for machine learning (ML) models does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 11 additionally recites:
presenting a user interface comprising a plurality of options for a machine learning (ML) model configuration of an ML model by an ML model training system; receiving a feature parameter for a feature usable by the ML model for an ML model output task; [accessing features scores for a plurality of features] preexisting with the ML model training system; [presenting features] in the user interface – Generally linking an abstract procedure of analysis and calculation to the field of use of data preprocessing for machine learning (ML) models does not provide an inventive concept or significantly more to the recited abstract idea. Further, receiving and transmitting data is well-understood, routine, and conventional activity (see MPEP § 2106.05(d); “Receiving or transmitting data over a network”) and thereby also does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 19 additionally recites:
A non-transitory machine-readable medium having stored thereon machine- readable instructions executable to cause a machine to perform operations –Mere instructions to implement an abstract idea on a computer or computer components do not provide an inventive concept or significantly more to the recited abstract idea.
receiving a plurality of machine learning (ML) features associated with at least one ML model deployed with decision services of a server provider system, wherein each of the plurality of ML features is associated with a measurable datum used by one or more of the at least one ML model for ML model outputs; [clustering corresponding vectors in the vector space] using an ML clustering technique – Generally linking an abstract procedure of analysis and calculation to the field of use of data preprocessing for machine learning (ML) models does not provide an inventive concept or significantly more to the recited abstract idea. Further, receiving and transmitting data is well-understood, routine, and conventional activity (see MPEP § 2106.05(d); “Receiving or transmitting data over a network”) and thereby also does not provide an inventive concept or significantly more to the recited abstract idea.
storing [data] that enables use with a feature engineering operation of an application – Storing data in memory is well-understood, routine, and conventional activity (see MPEP § 2106.05(d); “Storing and retrieving information in memory”) and thereby does not provide an inventive concept or significantly more to the recited abstract idea.
Even when considered as an ordered combination, the additional elements recited in the claims ultimately do no more than place the claims in the context of data preprocessing for a generic ML model. As such, claims 1, 11, and 19 are not patent eligible.
Dependent Claims (Claims 2-10, Claims 12-18, Claim 20):
Dependent claims 2-10, 12-18, and 20 narrow the scope of independent claims 1, 11, and 19, and likewise narrow the recited judicial exceptions. They recite abstract idea limitations that are similar to those recited within the independent claims (i.e., mental processes and/or mathematical concepts), and thereby merely expand on the already recited exceptions. The dependent claims also do not recite any further additional elements that successfully integrate the recited judicial exceptions into a practical application or provide significantly more than the recited abstract ideas themselves. Consequently, claims 2-10, 12-18, and 20 are also rejected under 35 U.S.C. 101.
Step 1: Claims 2-10 are drawn to a system/apparatus, claims 12-18 are drawn to a method, and claim 20 is drawn to a product. Therefore, each of these claims falls under one of the four categories of statutory subject matter (process/method, machine/apparatus, manufacture/product, or composition of matter).
Step 2A Prong 1: Claims 2-10, 12-18, and 20 each recite a judicially recognized exception of an abstract idea.
Claim 2 recites, inter alia:
generating a plurality of variable clusters that includes the first cluster of variables and at least one second cluster of variables based on a plurality of variable vectors for the plurality of variables; and performing the similarity detection using one or more of the stored plurality of variable clusters – This limitation further amounts to a comparison of variables through dimensional vector representations, including dimensional vectors grouped based on proximity in vector space (clusters), and calculating scores of similarity (e.g., distances) between vectors, and thereby amounts to an abstract procedure of analysis that expressly recites mathematical calculation.
Claim 3 recites the same judicial exception as claim 2.
Claim 4 recites, inter alia:
calculating a plurality of cluster representative vectors including the cluster representative vector based on the plurality of variable clusters, wherein the plurality of cluster representative vectors are stored with the plurality of variable clusters and usable for performing the similarity detection – This limitation further amounts to a comparison of variables through dimensional vector representations, including dimensional vectors grouped based on proximity in vector space (clusters), and calculating “scores” of similarity (e.g., distances) between vectors, and thereby amounts to an abstract procedure of analysis that expressly recites mathematical calculation.
Claim 5 recites the same judicial exception as claim 2.
Claim 6 recites, inter alia:
displaying the variable similarity data in a list comprising at least one variable from the at least one of the first cluster of variables or each variable in the first cluster of variables, wherein the list includes variable usage data for the at least one variable – This limitation recites further observation of previously calculated values through display in a list, and thereby further amounts to a mentally performable process step within the recited abstract analysis procedure.
Claim 7 recites, inter alia:
prior to determining the first cluster of variables, the operations further comprise: generating the first variable vector based on at least one variable parameter for the first variable; and calculating the vector similarity using a vector similarity technique – This limitation further amounts to a comparison of variables through dimensional vector representations and calculating “similarities” between vectors (e.g., distances) for further analysis, and thereby amounts to an abstract procedure of analysis that expressly recites mathematical calculation.
Claim 10 recites, inter alia:
receiving a request to replace the first variable with a second variable from the first cluster of variables; and replacing the first variable with the second variable in the ML model configuration – This limitation recites further processing and organization of data based on observation of previously calculated values, and thereby further amounts to a mentally performable process step within the recited abstract analysis procedure.
Claim 15 recites, inter alia:
calculating the plurality of feature vectors for the plurality of features based on feature parameters for the plurality of features; clustering the plurality of features by the plurality of feature vectors; and generating the plurality of feature scores for a plurality of feature clusters from the plurality of features – This limitation recites a comparison of variables through dimensional vector representations, including dimensional vectors grouped based on proximity in vector space (clusters), and calculating “scores” between vectors, and thereby amounts to an abstract procedure of analysis that expressly recites mathematical calculation.
Claim 16 recites, inter alia:
generating a cluster representative for each of the plurality of feature clusters representing one or more of the plurality of feature vectors in each of the plurality of feature clusters, wherein the plurality of feature scores are associated with the cluster representative – This limitation further amounts to a comparison of variables through dimensional vector representations, including dimensional vectors grouped based on proximity in vector space (clusters), and calculating “scores” between vectors, and thereby amounts to an abstract procedure of analysis that expressly recites mathematical calculation.
Claim 17 recites, inter alia:
wherein the first one of the plurality of features is presented with an option to replace the feature in place of creating the feature as a new feature for the ML model training system – This limitation recites further processing and organization of data based on observation of previously calculated values, and thereby amounts to a mentally performable process step within the recited abstract analysis procedure.
Claim 20 recites, inter alia:
comparing a new feature vector for the new feature to the cluster representative vectors for the plurality of clusters; and presenting a result of the comparing in association with the new feature – This limitation further amounts to a comparison of variables through dimensional vector representations, including dimensional vectors grouped based on proximity in vector space (clusters), and calculating “scores” between vectors, and thereby amounts to an abstract procedure of analysis that expressly recites mathematical calculation.
Step 2A Prong 2: Claims 4, 7, 10, and 16 do not recite any further additional elements besides those recited in the independent claims, and the following additional elements recited in claims 2-3, 5-6, 8-9, 12-15, 17-18, and 20 also do not integrate the recited judicial exceptions into a practical application.
Claim 2 additionally recites:
[generating a plurality of variable clusters based on] an ML clustering technique – This limitation does no more than generally link an abstract procedure of calculation to the field of use of machine learning (ML) algorithms.
storing the plurality of variable clusters – This limitation does no more than further recite insignificant implementation of data storage via a computer, and thereby amounts to insignificant extra-solution activity.
Claim 3 additionally recites:
wherein the ML clustering technique utilizes at least one of a k-nearest neighbors algorithm, a k-means algorithm, a means-shift algorithm, or a DB SCAN algorithm with a vector similarity scoring between at least one of each of the plurality of variable vectors or nearest neighbors of the plurality of variable vectors – This limitation does no more than generally link an abstract procedure of calculation to the field of use of machine learning (ML) algorithms.
Claim 5 additionally recites:
wherein the determining the first cluster of variables matching the first variable is performed in real-time when generating variables and based on an input of at least one variable definition declaration or variable description data, and wherein the generating is precomputed prior to the accessing the ML model configuration – This limitation does no more than specify insignificant steps of computer implementation with respect to gathering data for further analysis, as well as generally linking an abstract procedure of analysis and calculation to the field of use of data preprocessing for machine learning (ML) models.
Claim 6 additionally recites:
[displaying the variable similarity data in a list] in the user interface, wherein the list enables selection and viewing of the at least one variable – This limitation recites an insignificant post- solution step of outputting displayed features which is insignificant extra-solution activity. It further merely invokes generic computer components (user interface) as tools to transmit data.
Claim 8 additionally recites:
wherein the first variable and the plurality of variables each comprise a JavaScript Object Notation (JSON) container having variable logic comprising at least one of a source table, a transformation logic, a windowing parameter, or a filter definition – This limitation does no more than invoke an insignificant means of computer implementation with respect to gathering and transmitting data to enable abstract analysis, and therefore amounts to insignificant extra-solution activity.
Claim 9 additionally recites:
wherein each of the plurality of variables is associated with a measurable datum usable by the ML model to generate an ML model output, wherein the operations further comprise receiving a selection of at least one variable parameter defining variable logic for the first variable during the variable engineering of the ML model, and wherein the presenting the variable similarity data occurs during the variable engineering based on the selection of the at least one variable parameter – These limitations does no more than generally link an abstract procedure of analysis and calculation to the field of use of data preprocessing for machine learning (ML) models.
Claim 12 additionally recites:
wherein the feature parameter is extracted from a data container for the feature or selected for the feature parameter during a feature engineering for the ML model – This limitation further recites insignificant pre- solution steps of gathering data which is insignificant extra-solution activity.
Claim 13 additionally recites:
wherein the feature parameter comprises one of a plurality of feature parameters for the feature in a plurality of feature fields for a configuration of the feature, and wherein the generating the feature vector is further based on the plurality of feature parameters – This limitation further recites insignificant pre- solution steps of gathering data which is insignificant extra-solution activity.
Claim 14 additionally recites:
wherein the feature comprises a JavaScript Object Notation (JSON) data structure comprising the feature parameter, and wherein the feature parameter comprises one of source data for the feature or processing logic for feature – This limitation does no more than invoke an insignificant means of computer implementation with respect to gathering and transmitting data to enable abstract analysis, and therefore amounts to insignificant extra-solution activity.
Claim 15 additionally recites:
accessing the plurality of features for the ML model training system; – This limitation further recites insignificant pre- solution steps of gathering data which is insignificant extra-solution activity.
[clustering the features] using an ML clustering technique – This limitation does no more than generally link an abstract procedure of calculation to the field of use of machine learning (ML) algorithms.
Claim 17 additionally recites:
[feature is presented] in the user interface; [replace the feature] via the user interface – This limitation does no more than invoke an insignificant means of computer implementation with respect to gathering and transmitting data to enable abstract analysis, and therefore amounts to insignificant extra-solution activity.
Claim 18 additionally recites:
wherein the user interface comprises one or more menus that enable an establishment of at least one of source tables for the feature parameter or logic definitions for the feature parameter when creating the feature as the new feature for the ML model training system – This limitation does no more than further invoke an insignificant means of computer implementation with respect to gathering and transmitting data to enable abstract analysis, and therefore amounts to insignificant extra-solution activity.
Claim 20 additionally recites:
receiving a new feature being generated for the at least one ML model or a new ML model for the decision services; [presenting result] via the application – These limitations do no more than invoke an insignificant means of computer implementation with respect to gathering and transmitting data to enable abstract analysis, and therefore amount to insignificant extra-solution activity.
Step 2B: The additional elements recited in claims 2-3, 5-6, 8-9, 12-15, 17-18, and 20, viewed individually or as an ordered combination, do not provide an inventive concept or otherwise amount to significantly more than the recited abstract ideas themselves.
Claim 2 additionally recites:
[generating a plurality of variable clusters based on] an ML clustering technique – Generally linking an abstract procedure of calculation to the field of use of machine learning (ML) algorithms does not provide an inventive concept or significantly more to the recited abstract idea.
storing the plurality of variable clusters – Storing data is well-understood, routine, and conventional activity (see MPEP § 2106.05(d); “Storing and retrieving information in memory”) and thereby does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 3 additionally recites:
wherein the ML clustering technique utilizes at least one of a k-nearest neighbors algorithm, a k-means algorithm, a means-shift algorithm, or a DB SCAN algorithm with a vector similarity scoring between at least one of each of the plurality of variable vectors or nearest neighbors of the plurality of variable vectors – Generally linking an abstract procedure of calculation to the field of use of machine learning (ML) algorithms does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 5 additionally recites:
wherein the determining the first cluster of variables matching the first variable is performed in real-time when generating variables and based on an input of at least one variable definition declaration or variable description data, and wherein the generating is precomputed prior to the accessing the ML model configuration – Receiving data via computer implementation (e.g., over a network) is well-understood, routine, and conventional activity (see MPEP § 2106.05(d); “Storing and retrieving information in memory”, “Receiving or transmitting data over a network”) and thereby does not provide an inventive concept or significantly more to the recited abstract idea. Generally linking an abstract procedure of analysis and calculation to the field of use of data preprocessing for machine learning (ML) models likewise does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 6 additionally recites:
[displaying the variable similarity data in a list] in the user interface, wherein the list enables selection and viewing of the at least one variable – Receiving and transmitting data is well-understood, routine, and conventional activity (see MPEP § 2106.05(d); “Storing and retrieving information in memory”) and thereby does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 8 additionally recites:
wherein the first variable and the plurality of variables each comprise a JavaScript Object Notation (JSON) container having variable logic comprising at least one of a source table, a transformation logic, a windowing parameter, or a filter definition – Utilizing JSON file formats for data transfer and management is well-understood, routine, and conventional activity (see Liu et al., “JSON Data Management – Supporting Schema-less Development in RDBMS”, [page 1247 Introduction]) and thereby does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 9 additionally recites:
wherein each of the plurality of variables is associated with a measurable datum usable by the ML model to generate an ML model output, wherein the operations further comprise receiving a selection of at least one variable parameter defining variable logic for the first variable during the variable engineering of the ML model, and wherein the presenting the variable similarity data occurs during the variable engineering based on the selection of the at least one variable parameter – Generally linking an abstract procedure of analysis and calculation to the field of use of data preprocessing for machine learning (ML) models does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 12 additionally recites:
wherein the feature parameter is extracted from a data container for the feature or selected for the feature parameter during a feature engineering for the ML model – Receiving data is well-understood, routine, and conventional activity (see MPEP § 2106.05(d); “Receiving or transmitting data over a network”) and thereby does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 13 additionally recites:
wherein the feature parameter comprises one of a plurality of feature parameters for the feature in a plurality of feature fields for a configuration of the feature, and wherein the generating the feature vector is further based on the plurality of feature parameters – Receiving data is well-understood, routine, and conventional activity (see MPEP § 2106.05(d); “Receiving or transmitting data over a network”) and thereby does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 14 additionally recites:
wherein the feature comprises a JavaScript Object Notation (JSON) data structure comprising the feature parameter, and wherein the feature parameter comprises one of source data for the feature or processing logic for feature – Utilizing JSON file formats for data transfer and management is well-understood, routine, and conventional activity (see Liu et al., “JSON Data Management – Supporting Schema-less Development in RDBMS”, [page 1247 Introduction]) and thereby does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 15 additionally recites:
accessing the plurality of features for the ML model training system; – Receiving data is well-understood, routine, and conventional activity (see MPEP § 2106.05(d); “Receiving or transmitting data over a network”) and thereby does not provide an inventive concept or significantly more to the recited abstract idea.
[clustering the features] using an ML clustering technique – Generally linking an abstract procedure of calculation to the field of use of machine learning (ML) algorithms does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 17 additionally recites:
[feature is presented] in the user interface; [replace the feature] via the user interface – Receiving and transmitting data via computer implementation (e.g., over a network) is well-understood, routine, and conventional activity (see MPEP § 2106.05(d); “Storing and retrieving information in memory”, “Receiving or transmitting data over a network”) and thereby does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 18 additionally recites:
wherein the user interface comprises one or more menus that enable an establishment of at least one of source tables for the feature parameter or logic definitions for the feature parameter when creating the feature as the new feature for the ML model training system – Receiving data via computer implementation (e.g., over a network) is well-understood, routine, and conventional activity (see MPEP § 2106.05(d); “Storing and retrieving information in memory”, “Receiving or transmitting data over a network”) and thereby does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 20 additionally recites:
receiving a new feature being generated for the at least one ML model or a new ML model for the decision services; [presenting result] via the application – Receiving and transmitting data via computer implementation (e.g., over a network) is well-understood, routine, and conventional activity (see MPEP § 2106.05(d); “Storing and retrieving information in memory”, “Receiving or transmitting data over a network”) and thereby does not provide an inventive concept or significantly more to the recited abstract idea.
Even when considered as an ordered combination, the additional elements recited in the claims ultimately do no more than generically place the claims in the context of data preprocessing for a generic ML model, wherein data is processed, e.g., through a certain format and/or via a generic user interface. As such, claims 2-10, 12-18, and 20 are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-16 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Harris et al. (Pub. No. US 20180330258 A1, “Autonomous Learning Platform for Novel Feature Discovery”, filed 05/09/2017), hereinafter Harris, in view of Brumbaugh et al. (“Bighead: A Framework-Agnostic, End-to-End Machine Learning Platform”, available 2019), hereinafter Brumbaugh.
Regarding claim 1, Harris teaches A system (“Various embodiments are directed to performing autonomous learning for updating input features used for an artificial intelligence model. For example, a method can comprise receiving updated data of an information space that includes a graph of nodes having a defined topology, the updated data including historical data of requests to the artificial intelligence model and output results associated with the requests, wherein different categories of input data corresponds to different input nodes of the graph…In addition, the method may also include grouping input nodes to identify candidate input features to the artificial intelligence model, wherein the input node(s) of a candidate input feature are correlated to each other, and wherein the candidate input features have strength values above a threshold. Furthermore, the method may comprise training an enlarged artificial intelligence model using the historical data for the candidate input features, the training providing a ranking of candidate input features relative to a measure of effect on an output value of the enlarged artificial intelligence model, and selecting the top N candidate input features to be used in the artificial intelligence model.” [Harris ¶ 0005]; “Other embodiments are directed to systems, devices, and computer readable media associated with methods described herein” [Harris ¶ 0007]) comprising:
a non-transitory memory ([Harris ¶ 0007] as detailed above); and
one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations ([Harris ¶ 0007] as detailed above) comprising:
accessing a machine learning (ML) model configuration associated with an ML model (“In some implementations, an information space can be analyzed to determine suitable features for training the AI model. The information space may be represented as a topological graph of nodes connected by edges, which can have assigned weight, distance, or cost” [Harris ¶ 0031]; “FIG. 3 shows a depiction of selecting input features from a topological graph. In graph 300, each node (A through G, S through Z) may represent a specific set (e.g., one piece or a collection) of information determined from collected data….The information of graph 300 may be used as training data for an artificial intelligence model” [Harris ¶ 0055, 0057]; A topological graph (i.e., configuration) is accessed to obtain features which are used as training data for an AI model (i.e., associated with an ML model));
determining a first variable from a plurality of variables that enables a similarity detection based on the ML model configuration; (“Rather than score individual nodes for predictiveness, the AI model may score a collection of nodes, known as features…Features may be manually selected from the graph by one skilled in the art based on knowledge about the information space” [Harris ¶ 0058-0059]; “The information space may include a graph of nodes having a defined topology, where most of the nodes correspond to different inputs and some of the nodes correspond to outputs. An input may be any known or pre-defined characteristic of the information space, and an output may be any predicted outcome that depends on one or more inputs. Examples of an input may include transaction data, user profile data, text in a conversation, etc. Examples of outputs may include a targeted recommendation, a probability of fraud, a response to initiate a purchase, etc.” [Harris ¶ 0062]; “At step 430, the newly detected features may be smoothed (binned) to create a reduced feature set. The nodes of the graph may be grouped or formed into clusters, in which nodes that are close to each other in topology form a cluster and may have similar properties such as similar signal-to-noise ratio. Clusters with sufficiently high connection to a given output node may be identified as a possible input feature” [Harris ¶ 0066]; Wherein nodes of the topology graph (i.e., model configuration) represent features, an output node (i.e., first variable) may be identified for determining clusters with sufficiently high connection (i.e. similarity) that comprise a group of nodes (i.e., plurality of variables))
determining a first cluster of variables matching the first variable based on a vector similarity for a first variable vector representing the first variable and a cluster representative vector representing the first cluster of variables (“Clusters with sufficiently high connection to a given output node may be identified as a possible input feature. A predetermined amount of input features may be added to a current set of input features to develop a new feature set or current version of the model. For example, 50 new input features can be added to a current set of 200 input features for the current version of the model. The 50 new features can be identified based on the connection strength to the selected output nodes by, for example, selecting the top 50 features based on a shortest distance from an output node. These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; “An edge may be associated with a numerical value, known as a weight, that may be assigned to the pairwise connection between the two nodes. The edge weight may be identified as a strength of connectivity between two nodes and/or may be related to a cost or distance” [Harris ¶ 0048]; “ An edge weight may also be used to express a cost or a distance required to move from one state or node to the next” [Harris ¶ 0024]; For a selected output node (i.e., first variable), a selection of clusters (incl. first cluster) each representing new input features can be identified (i.e., matched) based on relative connection strength (i.e., similarity), wherein connection strength may be further measured through edge weights representing any form of distance between nodes (e.g., Euclidean distance between vectors)); and
presenting variable similarity data for at least one of the first cluster of variables or each variable in the first cluster of variables (“These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; “At step 440, the new feature set may be used to retrain the model using the new features. The training of the enlarged artificial intelligence model may be performed using the historical data for candidate input features, and the training may provide a ranking of the candidate input features relative to a measure of their effect on an output value of the enlarged artificial intelligence model… In one embodiment, the features in the enlarged feature set may be ranked according to strength of correlation to a predicted outcome, and the feature set may then be capped to its original size (e.g. ranking 250 features and then selecting only the top 200 features as the new feature set)” [Harris ¶ 0067]; Wherein the selected clusters (incl. first cluster) form the candidate features that are input for training, the features are further presented in a ranking based on the strength of their correlation (i.e., variable similarity) to a predicted outcome)
However, Harris does not expressly teach a system wherein the ML model is deployable with a computing service of the system, or wherein variable similarity data is presented in a user interface associated with performing a variable engineering for the ML model.
In the same field of endeavor, Brumbaugh teaches a machine learning platform including a feature engineering framework (“Here, we introduce Bighead, a framework-agnostic, end-toend platform for machine learning. It offers a seamless user experience requiring only minimal efforts that span feature set management, prototyping, training, batch (offline) inference, realtime (online) inference, evaluation, and model lifecycle management. In contrast to existing platforms, it is designed to be highly versatile and extensible, and supports all major machine learning frameworks, rather than focusing on one particular framework” [Brumbaugh Abstract]; see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, incl. Feature Generation and Zipline component [Brumbaugh page 553]; “The lifecycle of a machine learning model typically starts with collecting datasets comprised of features for the model to train and perform inference on, followed by constant iterations on processing the features to extract relevant information, called feature engineering… Zipline is a framework built to tackle these challenges in feature management for machine learning projects. It provides a general-purpose feature processing, aggregation, and joining framework that guarantees point-in-time correctness for the feature sets” [Brumbaugh page 554 Zipline: Feature Management]) wherein the ML model is deployable with a computing service of the system (see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, incl. Feature Generation and Zipline component [Brumbaugh page 553]; “To address the challenges in the domain of machine learning outlined in the previous sections, Bighead is designed to be: Seamless: Offer a streamlined user experience from prototyping to production, across different frameworks. Enable users to easily write, iterate on, and deploy readable, robust, and reproducible machine learning models. Provide integration with existing data infrastructure and make data access and storage straightforward” [Brumbaugh page 553 Design Goals and Architecture]; Zipline, the feature engineering component, exists as a component within the deployable Bighead service) and wherein variable similarity data is presented in a user interface associated with performing a variable engineering for the ML model (“Built with the design goals in mind, Bighead consists of multiple components (Figure 1):… Bighead UI: A user interface that provides easy operations and visualizations for models (Section IX) [Brumbaugh page 553 Architecture]; “More specifically, users can carry over highly customizable metadata from training to the Bighead UI, such as evaluation metrics and visualizations. For example, users can explore the model performance in the UI to determine which models should be deployed, via feature importance plots, ROC curves, and precision-recall curves” [Brumbaugh page 557 Bighead UI]; see Fig. 6 – “Screenshots of UI showing an example Model, its Versions and Artifacts, and details of an Artifact”, including Feature importance (top 20) plot [Brumbaugh page 558])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated a system wherein the ML model is deployable with a computing service of the system, and wherein variable similarity data is presented in a user interface associated with performing a variable engineering for the ML model as taught by Brumbaugh into Harris because they are both directed towards systems that utilize feature engineering frameworks for preprocessing input data to a machine learning model. Given that the Bighead platform is already explained to be highly versatile and extensible [Brumbaugh Abstract], a person of ordinary skill would be able to incorporate the teachings of Brumbaugh to thereby extend the autonomous learning platform of Harris into an operational, deployable end-to-end machine learning platform. Doing so would enable real-time deployment of the resulting platform in a consistent and reliable manner, thereby addressing scalability challenges and improving the user experience [Brumbaugh Abstract].
Regarding claim 2, the combination of Harris and Brumbaugh teaches the limitations of parent claim 1, and Harris further teaches generating a plurality of variable clusters that includes the first cluster of variables and at least one second cluster of variables based on a plurality of variable vectors for the plurality of variables and an ML clustering technique; ([Harris ¶ 0066] as detailed in claim 1 above; “Features may be manually selected from the graph by one skilled in the art based on knowledge about the information space. This may done based on common sense rules and/or using any data mining technique, such as cluster analysis. For example, k-means clustering in combination with pre-defined rules (e.g. node Y: ‘male’ should not be of the same feature as node A: ‘female’) may be used to group nearby nodes into a single feature” [Harris ¶ 0059])
storing the plurality of variable clusters; ([Harris ¶ 0066] as detailed in claim 1 above; The clusters (and associated features) are stored for input to the downstream model) and
performing the similarity detection using one or more of the stored plurality of variable clusters ([Harris ¶ 0066] as detailed in claim 1 above; For a selected output node (i.e., first variable), a selection of clusters (incl. first or second cluster) each representing new input features can be identified (i.e., matched) based on relative connection strength (i.e., similarity)).
Regarding claim 3, the combination of Harris and Brumbaugh teaches the limitations of parent claim 2, and Harris further teaches wherein the ML clustering technique utilizes at least one of a k-nearest neighbors algorithm, a k-means algorithm, a means-shift algorithm, or a DB SCAN algorithm with a vector similarity scoring between at least one of each of the plurality of variable vectors or nearest neighbors of the plurality of variable vectors ([Harris ¶ 0059] as detailed above; K-means clustering expressly may be used to create clusters)
Regarding claim 4, the combination of Harris and Brumbaugh teaches the limitations of parent claim 2, and Harris further teaches wherein the operations further comprise:
calculating a plurality of cluster representative vectors including the cluster representative vector based on the plurality of variable clusters, ([Harris ¶ 0059] as detailed in claim 2 above; K-means clustering expressly calculates centroids (i.e., representative vectors) for each cluster)
wherein the plurality of cluster representative vectors are stored with the plurality of variable clusters and usable for performing the similarity detection ([Harris ¶ 0066] as detailed in claim 1 above; For a selected output node (i.e., first variable), a selection of clusters (incl. first or second cluster) each representing new input features can be identified (i.e., matched) based on relative connection strength (i.e., similarity), wherein connection strength may be further measured through edge weights representing any form of distance between nodes (e.g., Euclidean distance to cluster centroid)).
Regarding claim 5, the combination of Harris and Brumbaugh teaches the limitations of parent claim 2, and Harris further teaches matching the first variable based on an input of at least one variable definition or variable description data, ([Harris ¶ 0066] as detailed in claim 1 above; “The information space may include a graph of nodes having a defined topology, where most of the nodes correspond to different inputs and some of the nodes correspond to outputs. An input may be any known or pre-defined characteristic of the information space, and an output may be any predicted outcome that depends on one or more inputs. Examples of an input may include transaction data, user profile data, text in a conversation, etc. Examples of outputs may include a targeted recommendation, a probability of fraud, a response to initiate a purchase, etc.” [Harris ¶ 0062]) and wherein the generating is precomputed prior to the accessing the ML model configuration (“These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; Engineering of candidate features is executed prior to accessing the downstream model). Brumbaugh further teaches feature engineering operations being performed in real-time (“Zipline is a framework built to tackle these challenges in feature management for machine learning projects. It provides a general-purpose feature processing, aggregation, and joining framework that guarantees point-in-time correctness for the feature sets.” [Brumbaugh page 554 Zipline: Feature Management]).
Regarding claim 6, the combination of Harris and Brumbaugh teaches the limitations of parent claim 1, and Harris further teaches displaying the variable similarity data in a list comprising at least one variable from the at least one of the first cluster of variables or each variable in the first cluster of variables, wherein the list includes variable usage data for the at least one variable ([Harris ¶ 0066, 0067] as detailed in claim 1 above; Wherein the selected clusters form the candidate features that are input for training, the features are further presented in a ranking based on the strength of their correlation (i.e., variable similarity) to a predicted outcome (i.e., their level of “usage” for generating said outcome)). Brumbaugh further teaches displaying variable similarity data in the user interface, wherein the list enables selection and viewing of the at least one variable ([Brumbaugh page 557 Bighead UI] as detailed in claim 1 above; see Fig. 6 – “Screenshots of UI showing an example Model, its Versions and Artifacts, and details of an Artifact”, including Feature importance (top 20) plot [Brumbaugh page 558]).
Regarding claim 7, the combination of Harris and Brumbaugh teaches the limitations of parent claim 1, and Harris further teaches wherein, prior to the determining the first cluster of variables, the operations further comprise:
generating the first variable vector based on at least one variable parameter for the first variable; (“An artificial intelligence model may learn from a set of collected data that is expressed as a graph characterized by nodes whose connection to one another are expressed as edges. Nodes within the graph of the information space may correspond to specific pieces of information collected from data. For example, for a given information space representing data collected in a transaction network, any given piece of information extracted from a transaction may be expressed as a node (e.g. account number, device ID, merchant category code, transaction type, etc.)” [Harris ¶ 0073]; Any collected parameters may be thereby expressed as nodes within the information space (i.e., variable vectors)) and
calculating the vector similarity using a vector similarity technique ([Harris ¶ 0066] as detailed in claim 1 above).
Regarding claim 8, the combination of Harris and Brumbaugh teach the limitations of parent claim 1, and Brumbaugh further teaches wherein the first variable and the plurality of variables each comprise a JavaScript Object Notation (JSON) container having variable logic comprising at least one of a source table, a transformation logic, a windowing parameter, or a filter definition (“Finally, Zipline exposes an API for users to define a feature set. This API includes 1) metadata, such as owner and documentation, 2) source of the feature set, including a schema and relevant properties, and 3) a list of features and how to aggregate each of them. The feature set definitions are stored in a repository so that they can be easily discovered, inspected, and shared. Below is an example of a feature set definition. Zipline is aware of the timestamp of each raw event (ts), the primary key (listing), the aggregation (sum), and time windows (7d, 14d, etc.) for each feature” [Brumbaugh page 555 Zipline: Feature Management]; See example feature set definition which is in JSON format and includes source –
PNG
media_image1.png
578
726
media_image1.png
Greyscale
[Brumbaugh page 555]).
Regarding claim 9, the combination of Harris and Brumbaugh teach the limitations of parent claim 1, and Harris further teaches wherein each of the plurality of variables is associated with a measurable datum usable by the ML model to generate an ML model output. ([Harris ¶ 0062] as detailed in claim 1 above; Input features may include measurable types of data (e.g., transaction data)). Brumbaugh further teaches wherein the operations further comprise receiving a selection of at least one variable parameter defining variable logic for the first variable during the variable engineering of the ML model, and wherein the presenting the variable similarity data occurs during the variable engineering based on the selection of the at least one variable parameter (see Fig. 6 – “Screenshots of UI showing an example Model, its Versions and Artifacts, and details of an Artifact”, including Feature importance (top 20) plot [Brumbaugh page 558] as detailed in claim 1 above; Selection of a UI screen parameter may enable a user to view training details including feature importance (i.e., variable similarity data)).
Regarding claim 10, the combination of Harris and Brumbaugh teach the limitations of parent claim 1, and Harris further teaches receiving a request to replace the first variable with a second variable from the first cluster of variables; and replacing the first variable with the second variable in the ML model configuration ([Harris ¶ 0066, 0067] as detailed above; Wherein previous input features and new features may form the candidate features that are input to the downstream model, the features set may be further capped wherein some previous input features are replaced by new input features (including, e.g., those drawn from a first cluster) based on measured importance)
Regarding claim 11, Harris teaches receiving a feature parameter for a feature usable by the ML model for an ML model output task; (“Rather than score individual nodes for predictiveness, the AI model may score a collection of nodes, known as features…Features may be manually selected from the graph by one skilled in the art based on knowledge about the information space” [Harris ¶ 0058-0059]; “The information space may include a graph of nodes having a defined topology, where most of the nodes correspond to different inputs and some of the nodes correspond to outputs. An input may be any known or pre-defined characteristic of the information space, and an output may be any predicted outcome that depends on one or more inputs. Examples of an input may include transaction data, user profile data, text in a conversation, etc. Examples of outputs may include a targeted recommendation, a probability of fraud, a response to initiate a purchase, etc.” [Harris ¶ 0062])
generating a feature vector for the feature based at least on the feature parameter; (“An artificial intelligence model may learn from a set of collected data that is expressed as a graph characterized by nodes whose connection to one another are expressed as edges. Nodes within the graph of the information space may correspond to specific pieces of information collected from data. For example, for a given information space representing data collected in a transaction network, any given piece of information extracted from a transaction may be expressed as a node (e.g. account number, device ID, merchant category code, transaction type, etc.)” [Harris ¶ 0073]; Any collected parameters may be thereby expressed as nodes within the information space (i.e., variable vectors)))
accessing a plurality of feature scores for a plurality of features preexisting with the ML model training system, wherein the plurality of feature scores are associated with a plurality of feature vectors for the plurality of features; (“Rather than score individual nodes for predictiveness, the AI model may score a collection of nodes, known as features…Features may be manually selected from the graph by one skilled in the art based on knowledge about the information space” [Harris ¶ 0058-0059]; “The information space may include a graph of nodes having a defined topology, where most of the nodes correspond to different inputs and some of the nodes correspond to outputs. An input may be any known or pre-defined characteristic of the information space, and an output may be any predicted outcome that depends on one or more inputs. Examples of an input may include transaction data, user profile data, text in a conversation, etc. Examples of outputs may include a targeted recommendation, a probability of fraud, a response to initiate a purchase, etc.” [Harris ¶ 0062]; “At step 430, the newly detected features may be smoothed (binned) to create a reduced feature set. The nodes of the graph may be grouped or formed into clusters, in which nodes that are close to each other in topology form a cluster and may have similar properties such as similar signal-to-noise ratio. Clusters with sufficiently high connection to a given output node may be identified as a possible input feature” [Harris ¶ 0066]; Wherein nodes of the topology graph (i.e., model configuration) represent features, an output node (i.e., first variable) may be identified for determining clusters with sufficiently high connection (i.e. scores) that comprise a group of nodes (i.e., plurality of feature vectors))
calculating a weighted distance between the feature vector and each of the
plurality of feature vectors; (“An edge may be associated with a numerical value, known as a weight, that may be assigned to the pairwise connection between the two nodes. The edge weight may be identified as a strength of connectivity between two nodes and/or may be related to a cost or distance” [Harris ¶ 0048]; “ An edge weight may also be used to express a cost or a distance required to move from one state or node to the next” [Harris ¶ 0024];)
determining a first one of the plurality of features having a corresponding one of the weighted distances within a threshold distance to the feature vector; (“A predetermined amount of input features may be added to a current set of input features to develop a new feature set or current version of the model. For example, 50 new input features can be added to a current set of 200 input features for the current version of the model. The 50 new features can be identified based on the connection strength to the selected output nodes” [Harris ¶ 0066]) and
presenting the first one of the plurality of features (“These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]).
However, Harris does not expressly teach presenting a user interface comprising a plurality of options for a machine learning (ML) model configuration of an ML model by an ML model training system, or presenting the first one of the plurality of features in the user interface.
In the same field of endeavor, Brumbaugh teaches a machine learning platform including a feature engineering framework (“Here, we introduce Bighead, a framework-agnostic, end-toend platform for machine learning. It offers a seamless user experience requiring only minimal efforts that span feature set management, prototyping, training, batch (offline) inference, realtime (online) inference, evaluation, and model lifecycle management. In contrast to existing platforms, it is designed to be highly versatile and extensible, and supports all major machine learning frameworks, rather than focusing on one particular framework” [Brumbaugh Abstract]; see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, incl. Feature Generation and Zipline component [Brumbaugh page 553]; “The lifecycle of a machine learning model typically starts with collecting datasets comprised of features for the model to train and perform inference on, followed by constant iterations on processing the features to extract relevant information, called feature engineering… Zipline is a framework built to tackle these challenges in feature management for machine learning projects. It provides a general-purpose feature processing, aggregation, and joining framework that guarantees point-in-time correctness for the feature sets” [Brumbaugh page 554 Zipline: Feature Management]) presenting a user interface comprising a plurality of options for a machine learning (ML) model configuration of an ML model by an ML model training system, and presenting features in the user interface (“Built with the design goals in mind, Bighead consists of multiple components (Figure 1):… Bighead UI: A user interface that provides easy operations and visualizations for models (Section IX) [Brumbaugh page 553 Architecture]; “More specifically, users can carry over highly customizable metadata from training to the Bighead UI, such as evaluation metrics and visualizations. For example, users can explore the model performance in the UI to determine which models should be deployed, via feature importance plots, ROC curves, and precision-recall curves” [Brumbaugh page 557 Bighead UI]; see Fig. 6 – “Screenshots of UI showing an example Model, its Versions and Artifacts, and details of an Artifact”, including Feature importance (top 20) plot [Brumbaugh page 558]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated presenting a user interface comprising a plurality of options for a machine learning (ML) model configuration of an ML model by an ML model training system and presenting the first one of the plurality of features in the user interface. as taught by Brumbaugh into Harris because they are both directed towards systems that utilize feature engineering frameworks for preprocessing input data to a machine learning model. Given that the Bighead platform is already explained to be highly versatile and extensible [Brumbaugh Abstract], a person of ordinary skill would be able to incorporate the teachings of Brumbaugh to thereby extend the autonomous learning platform of Harris into an operational, deployable end-to-end machine learning platform. Doing so would enable real-time deployment of the resulting platform in a consistent and reliable manner, thereby addressing scalability challenges and improving the user experience [Brumbaugh Abstract].
Regarding claim 12, the combination of Harris and Brumbaugh teach the limitations of parent claim 11, and Brumbaugh further teaches wherein the feature parameter is extracted from a data container for the feature or selected for the feature parameter during a feature engineering for the ML model (see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, including External Feature Sources which are fetched for Zipline [Brumbaugh page 553]).
Regarding claim 13, the combination of Harris and Brumbaugh teach the limitations of parent claim 12, and Harris further teaches wherein the feature parameter comprises one of a plurality of feature parameters for the feature in a plurality of feature fields for a configuration of the feature, ([Harris ¶ 0062] as detailed in claim 11 above; Input features may include configuration data) and wherein the generating the feature vector is further based on the plurality of feature parameters ([Harris ¶ 0073] as detailed in claim 11 above; Any collected parameters may be thereby expressed as nodes within the information space (i.e., variable vectors)).
Regarding claim 14, the combination of Harris and Brumbaugh teach the limitations of parent claim 11, and Brumbaugh further teaches wherein the feature comprises a JavaScript Object Notation (JSON) data structure comprising the feature parameter, and wherein the feature parameter comprises one of source data for the feature or processing logic for feature (“Finally, Zipline exposes an API for users to define a feature set. This API includes 1) metadata, such as owner and documentation, 2) source of the feature set, including a schema and relevant properties, and 3) a list of features and how to aggregate each of them. The feature set definitions are stored in a repository so that they can be easily discovered, inspected, and shared. Below is an example of a feature set definition. Zipline is aware of the timestamp of each raw event (ts), the primary key (listing), the aggregation (sum), and time windows (7d, 14d, etc.) for each feature” [Brumbaugh page 555 Zipline: Feature Management]; See example feature set definition which is in JSON format and includes source –
PNG
media_image1.png
578
726
media_image1.png
Greyscale
[Brumbaugh page 555]).
Regarding claim 15, the combination of Harris and Brumbaugh teach the limitations of parent claim 11, and Harris further teaches accessing the plurality of features for the ML model training system; ([Harris ¶ 0058-0059,0062] as detailed in claim 11 above).
calculating the plurality of feature vectors for the plurality of features based on feature parameters for the plurality of features; ([Harris ¶ 0073] as detailed in claim 11 above; Any collected parameters may be thereby expressed as nodes within the information space (i.e., feature vectors))
clustering the plurality of features by the plurality of feature vectors using an ML clustering technique; ([Harris ¶ 0066] as detailed in claim 11 above; Wherein nodes of the topology graph represent features, an output node may be identified for determining clusters with sufficiently high connection (i.e. similarity) that comprise a group of nodes (i.e., plurality of variables)) and
generating the plurality of feature scores for a plurality of feature clusters from the plurality of features ([Harris ¶ 0048, 0024, 0066] as detailed in claim 11 above; For a selected output node, a selection of clusters each representing new input features can be identified based on relative connection strength (i.e., feature score)).
Regarding claim 16, the combination of Harris and Brumbaugh teaches the limitations of parent claim 15, and Harris further teaches generating a cluster representative for each of the plurality of feature clusters representing one or more of the plurality of feature vectors in each of the plurality of feature clusters, wherein the plurality of feature scores are associated with the cluster representative (“Features may be manually selected from the graph by one skilled in the art based on knowledge about the information space. This may done based on common sense rules and/or using any data mining technique, such as cluster analysis. For example, k-means clustering in combination with pre-defined rules (e.g. node Y: ‘male’ should not be of the same feature as node A: ‘female’) may be used to group nearby nodes into a single feature” [Harris ¶ 0059]; K-means clustering expressly calculates centroids (i.e., representative vectors) for each cluster. For a selected output node (i.e., first variable), a selection of clusters each representing new input features can be identified based on relative connection strength (i.e., feature score), wherein connection strength may be further measured through edge weights representing any form of distance between nodes (e.g., Euclidean distance to cluster centroid)).
Regarding claim 19, Harris teaches A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations (“Various embodiments are directed to performing autonomous learning for updating input features used for an artificial intelligence model. For example, a method can comprise receiving updated data of an information space that includes a graph of nodes having a defined topology, the updated data including historical data of requests to the artificial intelligence model and output results associated with the requests, wherein different categories of input data corresponds to different input nodes of the graph…In addition, the method may also include grouping input nodes to identify candidate input features to the artificial intelligence model, wherein the input node(s) of a candidate input feature are correlated to each other, and wherein the candidate input features have strength values above a threshold. Furthermore, the method may comprise training an enlarged artificial intelligence model using the historical data for the candidate input features, the training providing a ranking of candidate input features relative to a measure of effect on an output value of the enlarged artificial intelligence model, and selecting the top N candidate input features to be used in the artificial intelligence model.” [Harris ¶ 0005]; “Other embodiments are directed to systems, devices, and computer readable media associated with methods described herein” [Harris ¶ 0007]) comprising:
receiving a plurality of machine learning (ML) features associated with at least one ML model, wherein each of the plurality of ML features is associated with a measurable datum used by one or more of the at least one ML model for ML model outputs ((“In some implementations, an information space can be analyzed to determine suitable features for training the AI model. The information space may be represented as a topological graph of nodes connected by edges, which can have assigned weight, distance, or cost” [Harris ¶ 0031]; “FIG. 3 shows a depiction of selecting input features from a topological graph. In graph 300, each node (A through G, S through Z) may represent a specific set (e.g., one piece or a collection) of information determined from collected data….The information of graph 300 may be used as training data for an artificial intelligence model” [Harris ¶ 0055, 0057]; “An input may be any known or pre-defined characteristic of the information space, and an output may be any predicted outcome that depends on one or more inputs. Examples of an input may include transaction data, user profile data, text in a conversation, etc. Examples of outputs may include a targeted recommendation, a probability of fraud, a response to initiate a purchase, etc.” [Harris ¶ 0062]; A topological graph (i.e., configuration) is accessed to obtain features (incl. e.g. measurable data) which are used as training data for an AI model (i.e., associated with an ML model));
converting feature description data for each of the plurality of ML features to a corresponding vector representable in a vector space; (“An artificial intelligence model may learn from a set of collected data that is expressed as a graph characterized by nodes whose connection to one another are expressed as edges. Nodes within the graph of the information space may correspond to specific pieces of information collected from data. For example, for a given information space representing data collected in a transaction network, any given piece of information extracted from a transaction may be expressed as a node (e.g. account number, device ID, merchant category code, transaction type, etc.)” [Harris ¶ 0073]; Any collected parameters may be thereby expressed as nodes within the information space (i.e., variable vectors)))
clustering the corresponding vectors for the plurality of ML features in the vector space using an ML clustering technique, wherein the ML clustering technique utilizes similarity scores between the corresponding vectors; ((“Clusters with sufficiently high connection to a given output node may be identified as a possible input feature. A predetermined amount of input features may be added to a current set of input features to develop a new feature set or current version of the model. For example, 50 new input features can be added to a current set of 200 input features for the current version of the model. The 50 new features can be identified based on the connection strength to the selected output nodes by, for example, selecting the top 50 features based on a shortest distance from an output node. These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; “An edge may be associated with a numerical value, known as a weight, that may be assigned to the pairwise connection between the two nodes. The edge weight may be identified as a strength of connectivity between two nodes and/or may be related to a cost or distance” [Harris ¶ 0048]; “ An edge weight may also be used to express a cost or a distance required to move from one state or node to the next” [Harris ¶ 0024])
generating a cluster representative vector for each cluster from a plurality of clusters resulting from the clustering; (Features may be manually selected from the graph by one skilled in the art based on knowledge about the information space. This may done based on common sense rules and/or using any data mining technique, such as cluster analysis. For example, k-means clustering in combination with pre-defined rules (e.g. node Y: ‘male’ should not be of the same feature as node A: ‘female’) may be used to group nearby nodes into a single feature” [Harris ¶ 0059]; K-means clustering expressly calculates centroids (i.e., representative vectors) for each cluster)) and
storing cluster comparison data comprising the cluster representative vectors and the plurality of clusters that enables use with a feature engineering operation (“These previous input features and new features may form the candidate features that can be input into the artificial intelligence model for training and evaluation” [Harris ¶ 0066]; The clusters (and associated features) are stored for input to the downstream model).
However, Harris does not expressly teach the ML model being deployed with decision services of a server provider system or the recited feature engineering procedure being an operation of an application.
In the same field of endeavor, Brumbaugh teaches a machine learning platform including a feature engineering framework (“Here, we introduce Bighead, a framework-agnostic, end-toend platform for machine learning. It offers a seamless user experience requiring only minimal efforts that span feature set management, prototyping, training, batch (offline) inference, realtime (online) inference, evaluation, and model lifecycle management. In contrast to existing platforms, it is designed to be highly versatile and extensible, and supports all major machine learning frameworks, rather than focusing on one particular framework” [Brumbaugh Abstract]; see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, incl. Feature Generation and Zipline component [Brumbaugh page 553]; “The lifecycle of a machine learning model typically starts with collecting datasets comprised of features for the model to train and perform inference on, followed by constant iterations on processing the features to extract relevant information, called feature engineering… Zipline is a framework built to tackle these challenges in feature management for machine learning projects. It provides a general-purpose feature processing, aggregation, and joining framework that guarantees point-in-time correctness for the feature sets” [Brumbaugh page 554 Zipline: Feature Management]) wherein the ML model is deployed with decision services of a server provider system, the recited feature engineering procedure being an operation of an application (see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, incl. Feature Generation and Zipline component [Brumbaugh page 553]; “To address the challenges in the domain of machine learning outlined in the previous sections, Bighead is designed to be: Seamless: Offer a streamlined user experience from prototyping to production, across different frameworks. Enable users to easily write, iterate on, and deploy readable, robust, and reproducible machine learning models. Provide integration with existing data infrastructure and make data access and storage straightforward” [Brumbaugh page 553 Design Goals and Architecture]; Zipline, the feature engineering component, exists as a component within the deployable Bighead service).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated a system wherein the ML model is deployed with decision services of a server provider system, the recited feature engineering procedure being an operation of an application as taught by Brumbaugh into Harris because they are both directed towards systems that utilize feature engineering frameworks for preprocessing input data to a machine learning model. Given that the Bighead platform is already explained to be highly versatile and extensible [Brumbaugh Abstract], a person of ordinary skill would be able to incorporate the teachings of Brumbaugh to thereby extend the autonomous learning platform of Harris into an operational, deployable end-to-end machine learning platform. Doing so would enable real-time deployment of the resulting platform in a consistent and reliable manner, thereby addressing scalability challenges and improving the user experience [Brumbaugh Abstract].
Regarding claim 20, the combination of Harris and Brumbaugh teaches the limitations of parent claim 19, and Harris further teaches comparing a new feature vector for the new feature to the cluster representative vectors for the plurality of clusters; and presenting a result of the comparing in association with the new feature ([Harris ¶ 0066, 0067] as detailed in claim 1 above). Brumbaugh further teaches presenting a result of the comparing via the application (Brumbaugh page 557 Bighead UI] as detailed in claim 1 above; see Fig. 6 – “Screenshots of UI showing an example Model, its Versions and Artifacts, and details of an Artifact”, including Feature importance (top 20) plot [Brumbaugh page 558]) and receiving a new feature being generated for the at least one ML model or a new ML model for the decision services; (see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, including External Feature Sources which are fetched for Zipline [Brumbaugh page 553]).
Claims 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Harris and Brumbaugh, as applied to claim 11 above, further in view of Chatzimparmpas et al. (“FeatureEnVi: Visual Analytics for Feature Engineering Using Stepwise Selection and Semi-Automatic Extraction Approaches”, available April 2022), hereinafter Chatzimparmpas.
Regarding claim 17, the combination of Harris and Brumbaugh teaches the limitations of parent claim 15.
However, the combination does not expressly teach wherein the first one of the plurality of features is presented in the user interface with an option to replace the feature in place of creating the feature as a new feature for the ML model training system via the user interface.
In the same field of endeavor, Chatzimparmpas teaches a feature engineering visual analytics system (“Despite that, while several visual analytics tools exist to monitor and control the different stages of the ML life cycle (especially those related to data and algorithms), feature engineering support remains inadequate. In this paper, we present FeatureEnVi, a visual analytics system specifically designed to assist with the feature engineering process. Our proposed system helps users to choose the most important feature, to transform the original features into powerful alternatives, and to experiment with different feature generation combinations” [Chatzimparmpas Abstract]) wherein features are presented in the user interface with an option to replace the feature in place of creating the feature as a new feature for the ML model training system via the user interface (see Fig. 1 – “Selecting important features, transforming them, and generating new features with FeatureEnVi” [Chatzimparmpas page 1774]; “Also, instead of appending features progressively (called forward selection) or considering all features and then discarding some (known as backward elimination), we choose a stepwise selection approach. Therefore, we start with all features, but we can add or remove any number of features at different stages.” [Chatzimparmpas page 1774 Introduction]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated features being presented in the user interface with an option to replace the feature in place of creating the feature as a new feature for the ML model training system via the user interface as taught by Chatzimparmpas into Harris and Brumbaugh because they are all directed towards feature engineering systems, with Brumbaugh and Chatzimparmpas specifically being directed towards visual analytics. Incorporating the teachings of Chatzimparmpas would thereby allow for increased feature engineering support functions in the provided user interfaces [Chatzimparmpas Abstract].
Regarding claim 18, the combination the combination of Harris, Brumbaugh, and Chatzimparmpas teaches the limitations of parent claim 17, and Brumbaugh further teaches wherein the user interface comprises one or more menus that enable an establishment of at least one of source tables for the feature parameter or logic definitions for the feature parameter when creating the feature as the new feature for the ML model training system (“Users only need to define features once, and Zipline makes them available in both inference and training environments, guaranteeing a single, consistent source of data for model consumption” [Brumbaugh page 554 Zipline: Feature Management]; see Fig. 1 – “An overview of the architecture of Bighead, depicting all the components”, including External Feature Sources which are fetched for Zipline [Brumbaugh page 553])
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VIJAY M BALAKRISHNAN whose telephone number is (571) 272-0455. The examiner can normally be reached 10am-5pm EST Mon-Thurs.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JENNIFER WELCH can be reached on (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/V.M.B./
Examiner, Art Unit 2143
/JENNIFER N WELCH/Supervisory Patent Examiner, Art Unit 2143
Prosecution Insights