DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Argument
Applicant’s arguments (pgs. 10 – 14) with respect to amended claim(s) have been considered but are moot, because arguments/remarks are directed to amended claim limitations that were not previously examined by the examiner. The rejections are noted in the current office action to address amended claim limitations.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 – 2, 7, 11, 15 and 18 – 19 are rejected under 35 U.S.C. 103 as being unpatentable over Misra et al., Pub. No.: US20180357511A1, in view of Stojanovic, Pub. No.: US10210246B2 and SHAH et al., Pub. No.: US20210349865A1.
Regarding claim 1, Misra teaches: A computer-implemented method for feature engineering in an artificial intelligence/machine learning (AI/ML) computing system, the method comprising:
(Misra, “[0016] … machine learning technique used [feature engineering in an artificial intelligence/machine learning (AI/ML) computing system] (e.g., supervised, unsupervised, or a combination of supervised and unsupervised machine learning technique, including support vector machine (SVM), conditional random field (CRF), Naive Bayes, convolutional neural networks, K-nearest neighbors, and/or the like), features used (e.g., key words, n-grams, word relationships, and/or the like), feature relevance scores used (e.g., feature weights assigned to the features, indicating a measure of importance of the feature to the machine learning technique), and/or the like.”)
receiving, via a data acquisition engine of an analytics application provided to a subscriber computing device, an input dataset comprising data regarding operations of a subscriber entity;
(Misra, “[0012] Some implementations, described herein, provide an analytics recommendation platform (e.g., cloud computing platform, server computer, and/or the like) designed to identify a recommended machine learning technique, recommended features, and recommended feature relevance scores (e.g., weights) for an analytics application. For example, the analytics recommendation platform may receive [receiving, via a data acquisition engine of an analytics application provided to a subscriber computing device], from a user device (e.g., a personal computer, mobile device, server computer, and/or the like), data defining one or more characteristics of an analytics application (e.g., a description of the analytics application, training data with annotations, test data, portions of data to be analyzed, and/or the like) [an input dataset comprising data regarding operations of a subscriber entity].”)
Misra does not teach:
generating, by a feature engineering engine of the analytics application, a reduced discovery dataset comprising a representative sample of records from the input dataset and
storing at least a portion of the reduced discovery dataset in cache memory associated with the analytics application;
while displaying, via a graphical user interface (GUI) associated with the analytics application, at least a portion of the reduced discovery dataset, performing feature engineering operations comprising:
performing, by the feature engineering engine, an entity resolution operation on the input dataset, comprising applying a first machine learning model to a set of items in the input dataset and a set of features retrieved from a feature catalogue to perform a match operation
based on output of the match operation between the set of items in the input dataset and the set of features retrieved from the feature catalogue, generating an instantiated set of features by associatively storing the set of items in the input dataset to the set of features in the feature catalogue;
generate for display, applying a second trained machine learning model on to the instantiated set of features,
a recommended operation for modifying the reduced discovery dataset
responsive to receiving, via the GUI, at least one user-guided operation to modify the reduced discovery dataset, perform, by the feature engineering engine, the at least one user-guided operation
to modify remaining data from the input dataset that is not included in the representative sample of records.
based on fuzzy logic;
wherein the second trained machine learning model is automatically selected from a plurality of models based on a performance metric determined for the instantiated set of features;
Stojanovic teaches:
generating, by a feature engineering engine of the analytics application, a reduced discovery dataset comprising a representative sample of records from the input dataset and
(Stojanovic, (col. 4 line [34 – 44]), “The method may include comparing the input data set to one or more reference data sets obtained from a reference source. The reference source may be a knowledge source provided by a knowledge service. The input data set may be compared to the one or more reference data sets using one or more of graph matching or semantic similarity matching [generating, by a feature engineering engine of the analytics application, a reduced discovery dataset comprising a representative sample of records from the input dataset]. The method may include computing a similarity metric for each of the one or more reference data sets, the similarity metric indicating a measure of similarity of each of the one or more reference data sets in comparison to the input data set.”)
storing at least a portion of the reduced discovery dataset in cache memory associated with the analytics application;
(Stojanovic, (col. 2 line [58 – 64]), “In some embodiments, the data enrichment service can compare an input data set to reference data sets stored in a knowledge source to identify similarly related data [storing at least a portion of the reduced discovery dataset in cache memory associated with the analytics application]. Matching can be performed between the input data and the reference data set(s) without supervised training (e.g., machine learning) and extraction precision can be improved over time via adaptive feedback from an end user.”)
while displaying, via a graphical user interface (GUI) associated with the analytics application, at least a portion of the reduced discovery dataset, performing feature engineering operations comprising:
(Stojanovic, (col. 8 line [25 – 31]), “In some embodiments, a publish stage 112 can provide data source metadata captured during enrichment and any data source enrichments or repairs to one or more visualization systems for analysis (e.g., display recommended data transformations, enrichments, and/or other modifications to a user) [while displaying, via a graphical user interface (GUI) associated with the analytics application, at least a portion of the reduced discovery dataset, performing feature engineering operations]. The publishing sub-system can deliver the processed data to one or more data targets.”)
performing, by the feature engineering engine, an entity resolution operation on the input dataset, comprising applying a first machine learning model to a set of items in the input dataset and a set of features retrieved from a feature catalogue to perform a match operation
(Stojanovic, (col. 21 line [34 – 42]), “Using the input data set, which may include added data, matching module 312 can implement matching methods (e.g., a graph matching method) to compare the words from the augmented data set [performing, by the feature engineering engine, an entity resolution operation on the input dataset, comprising applying a first machine learning model to a set of items in the input dataset] to categories of data from knowledge source 340 [a set of features retrieved from a feature catalogue to perform a match operation ]. Similarity metric module 314 can implement a method to determine the semantic similarity between the augmented data set and each category in knowledge source 340 to identify a name for the category. The name of the category may be chosen based on a highest similarity metric.”)
based on output of the match operation between the set of items in the input dataset and the set of features retrieved from the feature catalogue, generating an instantiated set of features by associatively storing the set of items in the input dataset to the set of features in the feature catalogue;
(Stojanovic, (col. 4 line [55 – 64]), “In some embodiments, the method may include: storing the input data set [between the set of items in the input dataset] with matching information [based on output of the match operation] that indicates the similarity metric computed for each of the one or more reference data sets [the set of features retrieved from the feature catalogue] and that indicates the match identified between the input data set and the one or more reference data sets; identifying a category label for the input data set based on identifying the match between the input data set and the one or more reference data sets; and storing the input data set in association with the category label [generating an instantiated set of features by associatively storing the set of items in the input dataset to the set of features in the feature catalogue].”)
generate for display, applying a second trained machine learning model on to the instantiated set of features,
(Stojanovic, (col. 8 line [25 – 31]), “In some embodiments, a publish stage 112 can provide data source metadata captured during enrichment and any data source enrichments or repairs to one or more visualization systems for analysis (e.g., display recommended data transformations, enrichments, and/or other modifications to a user) [generate for display, applying a second trained machine learning model]. The publishing sub-system can deliver the processed data to one or more data targets [to the instantiated set of features].”)
a recommended operation for modifying the reduced discovery dataset
(Stojanovic, (col. 9 line [3 – 13]), “Enrichment actions (e.g., the addition and/or transformation of data) can be performed on the data stored in the distributed storage system 210. Transformation of data may include modification to add missing data or data to enrich the data. Transformation of data may include modifying errors in the data or repairing the data [a recommended operation for modifying the reduced discovery dataset]. The publish processing stage 112 can include a publish engine, a syndication engine, and a metadata results manager. In some embodiments, various open source technologies can be used to implement some functionality within the various processing stages and/or processing engines.”)
responsive to receiving, via the GUI, at least one user-guided operation to modify the reduced discovery dataset, perform, by the feature engineering engine, the at least one user-guided operation
(Stojanovic, (col. 8 line [25 – 32]), “In some embodiments, a publish stage 112 can provide data source metadata captured during enrichment and any data source enrichments or repairs to one or more visualization systems for analysis (e.g., display recommended data transformations, enrichments, and/or other modifications to a user) [responsive to receiving, via the GUI, at least one user-guided operation to modify the reduced discovery dataset, perform, by the feature engineering engine, the at least one user-guided operation]. The publishing sub-system can deliver the processed data to one or more data targets. A data target may correspond to a place where the processed data can be sent.”)
to modify remaining data from the input dataset that is not included in the representative sample of records.
(Stojanovic, (col. 3 line 61 – col. 4 line 2), “The data enrichment service may implement several different techniques to determine the similarity of an input data set to one or more data sets. A input data set can be associated with or labeled with a category (e.g., domain) associated with the data set having the best match (e.g., a highest degree of similarity) with the input data set. As such, the input data set can be modified [to modify remaining data from the input dataset that is not included in the representative sample of records], or enriched with a category name that enables a user to better identify the input data set.”)
Stojanovic and Misra are related to the same field of endeavor (i.e.: recommendation platform). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Stojanovic with teachings of Misra to add a similarity based data enrichment technique that identifies related datasets using semantic similarity metrics and retrieves additional information from knowledge source to improve the relevance of feature and machine learning technique scores and enabling more informed recommendations or actions. (Stojanovic, Abstract).
Misra in view of Stojanovic do not teach:
… based on fuzzy logic;
wherein the second trained machine learning model is automatically selected from a plurality of models based on a performance metric determined for the instantiated set of features;
SHAH teaches:
… based on fuzzy logic;
(SHAH, “[0032] … The data mapper may perform the comparisons using character-based comparison techniques, fuzzy logic techniques [based on fuzzy logic], a semantic-based comparison, and/or the like. The data mapper may perform the comparisons above for each object of the one or more objects of the source data and for each field of each object…”)
wherein the second trained machine learning model is automatically selected from a plurality of models based on a performance metric determined for the instantiated set of features;
(SHAH, “[0079]… The machine learning system may then train each machine learning model using the entire training set 420 (e.g., without cross-validation), and may test each machine learning model using the test set 425 to generate a corresponding performance score for each machine learning model. The machine learning model may compare the performance scores for each machine learning model, and may select the machine learning model with the best (e.g., highest accuracy, lowest error, closest to a desired threshold, and/or the like) [wherein the second trained machine learning model is automatically selected from a plurality of models based on a performance metric determined for the instantiated set of features] performance score as the trained machine learning model 445.”)
SHAH, Misra and Stojanovic are related to the same field of endeavor (i.e.: recommendation platform). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of SHAH with teachings of Misra and Stojanovic to add a template based model assisted data migration and schema mapping mechanism with user verification to improve data integrity and reduced operational overhead. (SHAH, Abstract).
Claim 19 recites limitations analogous to claim 1, so is rejected under the same rationale.
Regarding claim 2, Misra, Stojanovic and SHAH teach the method of claim 1.
Misra further teaches: wherein the analytics application is provided by a provider entity associated with the AI/ML computing system, and wherein the analytics application is on a virtual network associated with the subscriber entity.
(Misra, “[0012] Some implementations, described herein, provide an analytics recommendation platform (e.g., cloud computing platform, server computer, and/or the like) [wherein the analytics application is provided by a provider entity associated with the AI/ML computing system] designed to identify a recommended machine learning technique, recommended features, and recommended feature relevance scores (e.g., weights) for an analytics application. For example, the analytics recommendation platform may receive, from a user device (e.g., a personal computer, mobile device, server computer, and/or the like) [wherein the analytics application is on a virtual network associated with the subscriber entity], data defining one or more characteristics of an analytics application (e.g., a description of the analytics application, training data with annotations, test data, portions of data to be analyzed, and/or the like).”)
Claim 18, recites limitations analogous to claim 2, so is rejected under the same rationale.
Regarding claim 7, Misra, Stojanovic and SHAH teach the method of claim 1.
Stojanovic further teaches: wherein performing feature engineering operations further comprises:
providing, via the GUI, an analytics engine selection control; and
(Stojanovic, (col. 8 line [25 – 32]), “In some embodiments, a publish stage 112 can provide data source metadata captured during enrichment and any data source enrichments or repairs to one or more visualization systems for analysis (e.g., display recommended data transformations, enrichments, and/or other modifications to a user) [wherein performing feature engineering operations further comprises:
providing, via the GUI]. The publishing sub-system can deliver the processed data to one or more data targets [an analytics engine selection control]. A data target may correspond to a place where the processed data can be sent.”)
responsive to detecting a selection using the analytics engine selection control, invoking an executable associated with the selected analytics engine to perform operations comprising: generating a visual summary of an item in the instantiated set of features; and
(Stojanovic, (col. 42 line [53 – 63]), “This big data and related applications can be hosted and/or manipulated by an infrastructure system on many levels and at different scales. Tens, hundreds, or thousands of processors linked in parallel can act upon such data in order to present it or simulate external forces on the data or what it represents. These data sets can involve structured data, such as that organized in a database or otherwise according to a structured model, and/or unstructured data [responsive to detecting a selection using the analytics engine selection control, invoking an executable associated with the selected analytics engine to perform operations comprising: generating a visual summary of an item in the instantiated set of features] (e.g., emails, images, data blobs (binary large objects), web pages, complex event processing).”)
causing the GUI to display the visual summary along with the instantiated set of features.
(Stojanovic, (col. 8 line [25 – 32]), “In some embodiments, a publish stage 112 can provide data source metadata captured during enrichment and any data source enrichments or repairs to one or more visualization systems for analysis (e.g., display recommended data transformations, enrichments, and/or other modifications to a user) [wherein performing feature engineering operations further comprises: providing, via the GUI]. The publishing sub-system can deliver the processed data to one or more data targets [an analytics engine selection control]. A data target may correspond to a place where the processed data can be sent.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Stojanovic with teachings of Misra and SHAH for the same reasons disclosed for claim 1.
Regarding claim 11, Misra, Stojanovic and SHAH teach the method of claim 1.
Misra further teaches: wherein the recommendation comprises at least one of: a score, a probability, a discovered cluster, or a data visualization.
(Misra, “[0012] Some implementations, described herein, provide an analytics recommendation platform (e.g., cloud computing platform, server computer, and/or the like) designed to identify a recommended machine learning technique, recommended features, and recommended feature relevance scores (e.g., weights) for an analytics application [wherein the recommendation comprises at least one of: a score,]. For example, the analytics recommendation platform may receive, from a user device (e.g., a personal computer, mobile device, server computer, and/or the like), data defining one or more characteristics of an analytics application (e.g., a description of the analytics application, training data with annotations, test data, portions of data to be analyzed, and/or the like).”)
Regarding claim 15, Misra, Stojanovic and SHAH teach the method of claim 1.
Misra further teaches: wherein performing the feature engineering operations further comprises: determining the set of features in the feature catalogue based on a previously generated feature definition mark-up file.
(Misra, “[0004] … determine, for the first analytics application, a first feature associated with the first analytics application, the first feature being based on the measure of similarity and a second feature associated with the second analytics application; [wherein performing the feature engineering operations further comprises: determining the set of features in the feature catalogue based on a previously generated feature definition mark-up file] determine, for the first analytics application, a first machine learning technique associated with the first analytics application, the first machine learning technique being based on the measure of similarity and a second machine learning technique associated with the second analytics application; and perform an action based on the first feature and the first machine learning technique.”)
Claim(s) 3 – 4 are rejected under 35 U.S.C. 103 as being unpatentable over Misra in view of Stojanovic, SHAH and in further view of Walenstein et al., Pub. No.: US20200026636A1.
Regarding claim 3, Misra, Stojanovic and SHAH teach the method of claim 1.
Misra in view of Stojanovic and SHAH do not teach:
further comprising generating the reduced discovery dataset using random sampling.
Walenstein teaches:
further comprising generating the reduced discovery dataset using random sampling.
(Walenstein, “[0033] A number of under-sampling methods can be used, for example, random majority under-sampling, under-sampling with cluster centroids, and extraction of majority-minority Tomek links. For example, the under-sampling with cluster centroids method under-samples the majority class by replacing a cluster of majority data entries with the cluster centroid (middle of a cluster) of a k-means algorithm. The random majority under-sampling method [further comprising generating the reduced discovery dataset using random sampling] under-samples the majority class by randomly picking samples with or without replacement.”)
Walenstein, Misra, Stojanovic and SHAH are related to the same field of endeavor (i.e.: recommendation platform). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Walenstein with teachings of Misra, Stojanovic and SHAH to add dataset sampling, feature based filtering and classification of new inputs into predefined categories using a trained machine learning model. (Walenstein, Abstract).
Regarding claim 4, Misra in view of Stojanovic and SHAH teach the method of claim 1.
Misra in view of Stojanovic and SHAH do not teach:
further comprising generating the reduced discovery dataset using stratified sampling.
Walenstein teaches:
further comprising generating the reduced discovery dataset using stratified sampling.
(Walenstein, “[0040] Various sampling methods can be used to split the modified data set and generate a balanced training data set and a balanced testing data set (i.e., the data set has a similar number of data entries for each class). For example, a stratified sampling method [further comprising generating the reduced discovery dataset using stratified sampling] or a stratified-k-fold sampling method can be used for splitting the modified data set. The idea of the stratified sampling method is to use a probabilistic sampling to divide an entire population into different strata which are randomly chosen but balanced and proportional. The stratified-k-fold sampling method is different from the stratified sampling method because the stratified-k-fold sampling method generates more than a single fold of the stratified sampling.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Walenstein with teachings of Misra, Stojanovic and SHAH for the same reasons disclosed for claim 3.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Misra in view of Stojanovic, SHAH and in further view of Anderson, Pub. No.: US9449057B2.
Misra in view of Stojanovic and SHAH teach the method of claim 1.
Misra in view of Stojanovic and SHAH do not teach:
wherein a size of the reduced discovery dataset is optimized by performing at least one of: generating the reduced discovery dataset to be at or under a predetermined size limit, extracting a predetermined number of records from the input dataset, or extracting a predetermined percentage of records from the input dataset.
Anderson teaches:
wherein a size of the reduced discovery dataset is optimized by performing at least one of: generating the reduced discovery dataset to be at or under a predetermined size limit, extracting a predetermined number of records from the input dataset, or extracting a predetermined percentage of records from the input dataset.
(Anderson, (col. 14 Line [51 – 61]), “To address this situation, in some implementations, a data pattern code can be formulated for the production data, i.e., the input data to the application. A dataset of test records may be created based on the data pattern code, and applied to the application. For example, a small number of records (e.g., a single record or another predetermined number of records) [wherein a size of the reduced discovery dataset is optimized by performing at least one of: generating the reduced discovery dataset to be at or under a predetermined size limit] can be selected for each distinct data pattern code. Having been taken from the full production data, this dataset may be deemed to represent a comprehensive set of test cases that the application can be configured to handle properly.”)
Anderson, Misra, Stojanovic and SHAH are related to the same field of endeavor (i.e.: recommendation platform). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Anderson with teachings of Misra, Stojanovic and SHAH to add structured encoding of dataset records into pattern representative features to improve computational efficiency and pattern extraction from large datasets. (Anderson, Abstract).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Misra in view of Stojanovic, SHAH and in further view of Chen et al., Pub. No.: US20200409945A1.
Misra in view of Stojanovic and SHAH teach the method of claim 1.
Misra in view of Stojanovic and SHAH do not teach:
wherein performing the entity resolution operations comprises de-duplicating an item in the input dataset.
Chen teaches:
wherein performing the entity resolution operations comprises de-duplicating an item in the input dataset.
(Chen, “[0072] At block 380, a scoring model is used to generate a score for each result in the first and second sets of results. For example, scoring model 219 generates a score for each target entity record that was identified by at least one of the searches. Block 390 may involve deduplicating the union of the first and second sets of results [wherein performing the entity resolution operations comprises de-duplicating an item in the input dataset] since some search results from the second query may be the same as some search results from the first query.”)
Chen, Misra, Stojanovic and SHAH are related to the same field of endeavor (i.e.: recommendation platform). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Chen with teachings of Misra, Stojanovic and SHAH to add a specific multi-query, score based entity resolution mechanism for matching records across different data sources to provide implementation detail for how cross dataset association or similarity determination is performed. (Chen, Abstract).
Claim(s) 8 – 10 are rejected under 35 U.S.C. 103 as being unpatentable over Misra in view of Stojanovic, SHAH and in further view of Nourian et al., Pub. No.: US20210049503A1.
Regarding claim 8, Misra in view of Stojanovic and SHAH teach the method of claim 7.
Misra in view of Stojanovic and SHAH do not teach:
wherein the item is a derived item, and wherein the visual summary relates to a local explainability statistic for the item
Nourian teaches:
wherein the item is a derived item, and wherein the visual summary relates to a local explainability statistic for the item.
(Nourian, “[0007] In accordance with some implementations of the disclosed subject matter, computer-implemented machines, systems and methods are disclosed for providing insights about a machine learning model, the machine learning model trained, during a training phase, to learn patterns to correctly classify input data associated with risk analysis. One or more features of the machine learning model may be analyzed [wherein the item is a derived item], the one or more features being defined based on one or more constraints associated with one or more values and relationships and whether said one or more values and relationships satisfy at least one of the one or more constraints. One or more visual indicators may be displayed based on an analysis of the one or more features and training data used to train the machine learning model, the one or more visual indicators providing a summary [wherein the visual summary] of the machine learning model's performance or efficacy. In response to further analyzing the one or more features and the training data, at least one or more of a global explanation about the machine learning model or a local explanation about the machine learning model may be provided [relates to a local explainability statistic for the item].”)
Nourian, Misra, Stojanovic and SHAH are related to the same field of endeavor (i.e.: recommendation platform). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Nourian with teachings of Misra, Stojanovic and SHAH to add model insight functionality including constraint based feature analysis and visual performance indicators to improve transparency of the machine learning model’s effectiveness. (Nourian, Abstract).
Regarding claim 9, Misra in view of Stojanovic, SHAH and Nourian teach the method of claim 7.
Nourian further teaches: wherein the visual summary relates to a global explainability statistic for at least a subset of the instantiated set of features.
(Nourian, “[0007] In accordance with some implementations of the disclosed subject matter, computer-implemented machines, systems and methods are disclosed for providing insights about a machine learning model, the machine learning model trained, during a training phase, to learn patterns to correctly classify input data associated with risk analysis. One or more features of the machine learning model may be analyzed, the one or more features being defined based on one or more constraints associated with one or more values and relationships and whether said one or more values and relationships satisfy at least one of the one or more constraints. One or more visual indicators may be displayed based on an analysis of the one or more features and training data used to train the machine learning model, the one or more visual indicators providing a summary of the machine learning model's performance or efficacy. In response to further analyzing the one or more features and the training data, at least one or more of a global explanation about the machine learning model [wherein the visual summary relates to a global explainability statistic for at least a subset of the instantiated set of features] or a local explanation about the machine learning model may be provided.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Nourian with teachings of Misra, Stojanovic and SHAH for the same reasons disclosed for claim 8.
Regarding claim 10, Misra in view of Stojanovic, SHAH and Nourian teach the method of claim 9.
Nourian further teaches: further comprising generating and displaying a GUI control structured (Nourian, “[0008] The global explanation may provide general information about one or more functionalities of the machine learning model and at least one of a visualization that summarizes the machine learning model's global behavior with respect to the one or more features [comprising generating and displaying a GUI control structured], or an identification of a first set of features that are influential in generating one or more identifiable outcomes. The further analyzing may be performed based on at least one of a model-dependent approach or a model-independent approach, wherein the model-dependent approach takes into consideration unique properties of the machine learning model, such as the model's structure in calculating a feature's importance to the machine learning's operation for correctly classifying the input data.”)
to enable a modification of a threshold relating to the global explainability statistic.
(Nourian, “[0010] In at least one implementation, a first threshold may be determined and the local explanation provides an understanding of how possible changes to the instance's feature values adjust or shift an expected result or projected outcome beyond the first threshold [to enable a modification of a threshold relating to the global explainability statistic]. In response to understanding how the machine learning model behaves in the first instance, the machine learning model may be tuned to select outcomes that best suit an expected result in a first set of instances. For example, the machine learning model may be tuned to minimally change one or more of the machine learning model's features in a first set of features having a first characteristic. The machine learning model may be tuned by adjusting instances that demand changes to the fewest number of features or instances with a least amount of change to the most important features of the machine learning model.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Nourian with teachings of Misra, Stojanovic and SHAH for the same reasons disclosed for claim 8.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Misra in view of Stojanovic, SHAH and Gupta et al., Pub. No.: US20220230181A1.
Misra in view of Stojanovic and SHAH teach the method of claim 1.
Misra in view of Stojanovic and SHAH do not teach:
wherein the input dataset is indicative of one or more activities, and wherein generating the recommendation comprises determining a next best activity for an activity in a set of one or more activities
Gupta teaches:
wherein the input dataset is indicative of one or more activities, and wherein generating the recommendation comprises determining a next best activity for an activity in a set of one or more activities.
(Gupta, “[0040] Accordingly, it is now appreciated that there is a need to provide a narrowly tailored subset of recommended actions [wherein the input dataset is indicative of one or more activities] as best recommended actions [and wherein generating the recommendation comprises determining a next best activity for an activity in a set of one or more activities], including the most applicable recommended actions, in view of the context. That is, based on certain conditions (e.g., probability of success related to the customer, business interest, etc.), the most applicable recommended actions may be provided to the agent to quickly and efficiently resolve the customer request.”)
Gupta, Misra, Stojanovic and SHAH are related to the same field of endeavor (i.e.: recommendation platform). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Gupta with teachings of Misra, Stojanovic and SHAH to add context based recommendation and ranking of actions to support decision through prioritized recommendation. (Gupta, Abstract).
Claim(s) 13 – 14 are rejected under 35 U.S.C. 103 as being unpatentable over Misra in view of Srivastava, Chen, SHAH, Gupta and Rausch et al., Pub. No.: US20170153914A1.
Regarding claim 13, Misra in view of Stojanovic and SHAH teach the method of claim 1.
Stojanovic further teaches: generating and displaying a visual summary of the instantiated set of features,
(Stojanovic, “In some embodiments, a publish stage 112 can provide data source metadata captured during enrichment and any data source enrichments or repairs to one or more visualization systems for analysis (e.g., display recommended data transformations, enrichments, and/or other modifications to a user) [responsive to receiving, via the GUI, at least one user-guided operation to modify the reduced discovery dataset, perform, by the feature engineering engine, the at least one user-guided operation]. The publishing sub-system can deliver the processed data to one or more data targets. A data target may correspond to a place where the processed data can be sent.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Stojanovic with teachings of Misra and SHAH for the same reasons disclosed for claim 1.
Misra, Stojanovic and SHAH do not teach:
wherein the instantiated set of features is shown as a linking item between a first node in a first set of nodes, the first node indicative of the input dataset, and
a second node in a second set of nodes, the second node indicative of the set of features.
Gupta teaches:
wherein the instantiated set of features is shown as a linking item between a first node in a first set of nodes, the first node indicative of the input dataset,
(Gupta, “[0050] Each of the node devices may additionally analyze its corresponding data set [the first node indicative of the input dataset] portion to determine one or more features of the data items therein at least partially in parallel with others of the node devices [wherein the instantiated set of features is shown as a linking item between a first node in a first set of nodes]. By way of example, each node device may determine the type(s) of data within its corresponding data set portion, including and not limited to, binary values, integer values, floating point values, text characters, bit mapped still images, frames of video data, samples of audio data, etc. Also by way of example, each node device may determine a density and/or sparsity of the data items, and/or may determine a size of the data items within its corresponding data set portion.”)
Gupta, Misra, Stojanovic and SHAH are related to the same field of endeavor (i.e.: recommendation platform). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Gupta with teachings of Misra, Stojanovic and SHAH to add context based recommendation and ranking of actions to support decision through prioritized recommendation. (Gupta, Abstract).
Misra, Stojanovic, SHAH and Gupta do not teach:
a second node in a second set of nodes, the second node indicative of the set of features
Rausch teaches:
a second node in a second set of nodes, the second node indicative of the set of features.
(Rausch, “[0051] In some embodiments, such analyses and provision of observation data indicative of features of the data items within each of the data set portions [and a second node in a second set of nodes, the second node indicative of the set of features] to the coordinating device may be at least partially enabled by the provision of the annotated metadata to each of the node devices. Thus, one or more of the node devices may employ the indications of structural features of the whole of the selected data set provided in the annotated metadata as input to such an analysis of the data items within one or more of the data set portions.”)
Rausch, Misra, Stojanovic, SHAH and Gupta are related to the same field of endeavor (i.e.: recommendation platform). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Rausch with teachings of Misra, Stojanovic, SHAH and Gupta to add automated action selection and execution based on similarities to previous analyzed datasets to leverage past actions to efficiently process new datasets. (Rausch, Abstract).
Regarding claim 14, Misra in view of Stojanovic, SHAH, Gupta and Rausch teach the method of claim 13.
Misra further teaches: wherein the detail record includes at least one of: a project identifier for a project that includes the instantiated feature, an instantiated feature identifier, an instantiated feature configuration parameter, a SQL query associated with the instantiated feature, or feature versioning information.
(Misra, “[0018] … The analytics application information may be stored in a variety of data structures that enable access to the analytics application information, such as a database that enables querying the database to obtain analytics application information based on the queries (e.g., queries for analytics applications associated with various analytics application characteristics, machine learning techniques, features, feature relevance scores, and/or the like) [wherein the detail record includes at least one of: a project identifier for a project that includes the instantiated feature, an instantiated feature identifier, an instantiated feature configuration parameter, a SQL query associated with the instantiated feature, or feature versioning information]. In this way, the analytics recommendation platform identifies and stores information associated with analytics applications in a manner designed to enable the information to be obtained later (e.g., for comparison with analytics applications for which the analytics recommendation platform is to make a recommendation).”)
Stojanovic further teaches: further comprising: upon detecting a user interaction with the linking item, generating and displaying, along with the visual summary, a detail record for a particular feature associated with the linking item,
(Stojanovic, (col. 8 line [25 – 32]), “In some embodiments, a publish stage 112 can provide data source metadata captured during enrichment and any data source enrichments or repairs to one or more visualization systems for analysis (e.g., display recommended data transformations, enrichments, and/or other modifications to a user) [further comprising: upon detecting a user interaction with the linking item, generating and displaying, along with the visual summary, a detail record for a particular feature associated with the linking item]. The publishing sub-system can deliver the processed data to one or more data targets. A data target may correspond to a place where the processed data can be sent.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Stojanovic with teachings of Misra, SHAH, Gupta and Rausch for the same reasons disclosed for claim 13.
Claim(s) 16 – 17 are rejected under 35 U.S.C. 103 as being unpatentable over Misra in view of Stojanovic, SHAH and Gupta.
Regarding claim 16, Stojanovic teaches: based on a feature configuration file, determining a feature catalogue to reference; and
(Stojanovic, (col. 23 line [19 – 26]), “Knowledge scoring module 316 can determine information about knowledge groups (e.g., domains or categories). Information about knowledge groups can be presented in a graphical user interface [based on a feature configuration file, determining a feature catalogue to reference], such as the example shown in FIG. 10. Information about knowledge domains may include a metric (e.g., a knowledge score) indicating a measure of similarity between a knowledge domain and an input data set of terms.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Stojanovic with teachings of Misra and SHAH for the same reasons disclosed for claim 1.
Misra in view of Stojanovic and SHAH do not teach:
using the instantiated set of features, applying a second trained machine learning model to determine a next best activity for an activity in the set of activities;
Gupta teaches:
using the instantiated set of features, applying a second trained machine learning model to determine a next best activity for an activity in the set of activities;
(Gupta, “[0040] Accordingly, it is now appreciated that there is a need to provide a narrowly tailored subset of recommended actions [using the instantiated set of features, applying a second trained machine learning model] as best recommended actions [to determine a next best activity for an activity in the set of activities], including the most applicable recommended actions, in view of the context. That is, based on certain conditions (e.g., probability of success related to the customer, business interest, etc.), the most applicable recommended actions may be provided to the agent to quickly and efficiently resolve the customer request.”)
Gupta, Misra, Stojanovic and SHAH are related to the same field of endeavor (i.e.: recommendation platform). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Gupta with teachings of Misra, Stojanovic and SHAH to add context based recommendation and ranking of actions to support decision through prioritized recommendation. (Gupta, Abstract).
The rest of the limitations are analogous to claim 1, so are rejected under similar rationale.
Regarding claim 17, Misra in view of Stojanovic, SHAH and Gupta teach the method of claim 16.
Gupta further teaches: further comprising: generating a plurality of customer conversion communication paths; and using the plurality of customer conversion communication paths,
(Gupta, “[0070] Generating recommended actions may also involve historical data 508. The historical data 508 may include past data related to the customer. The historical data 508 may include data indicating each customer action (e.g., customer financed home loan through the bank) or customer-client interaction (e.g., customer conversation with the agent of the bank) since association with the client [generating a plurality of customer conversion communication paths; and using the plurality of customer conversion communication paths], such as since the date that the customer opened an account with the client.”)
determining the next best activity.
(Gupta, “[0072] … In some instances, the number of recommended actions to provide to the agent may default to three recommended actions 514, in which the top or highest ranked of the three recommend actions is the best recommended action (e.g., next best action) of best recommended actions [determining the next best activity]. Specifically, the machine learning algorithm may learn about the agent, similarly situated agents, the customer, the department, the client interest 509 over time, and/or other business related information, and generate and rank the subset of ranked recommended actions 514 based on a predicted likelihood that the customer will act in accordance with the recommended actions (e.g., apply for the recommended credit card, apply for a new home loan, etc.).”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Gupta with teachings of Misra, Stojanovic and SHAH for the same reasons disclosed for claim 16.
Claim(s) 20 is rejected under 35 U.S.C. 103 as being unpatentable over Misra in view of Stojanovic, SHAH, Gupta and Rausch.
Regarding claim 20, Misra in view of Stojanovic and SHAH teach the method of claim 19.
Stojanovic further teaches: generating and displaying a visual summary of the instantiated set of features,
(Stojanovic, (col. 8 line [25 – 32]), “In some embodiments, a publish stage 112 can provide data source metadata captured during enrichment and any data source enrichments or repairs to one or more visualization systems for analysis (e.g., display recommended data transformations, enrichments, and/or other modifications to a user) generating and displaying a visual summary of the instantiated set of features,]. The publishing sub-system can deliver the processed data to one or more data targets. A data target may correspond to a place where the processed data can be sent.”)
upon detecting a user interaction with the linking item, generating and displaying, along with the visual summary, a detail record for a particular feature associated with the linking item, wherein the detail record includes at least one of: a project identifier for a project that includes the instantiated feature, an instantiated feature identifier, an instantiated feature configuration parameter, a SQL query associated with the instantiated feature, or feature versioning information.
(Stojanovic, (col. 25 line [24 – 33]), “The graphical dashboard may indicate a plurality of metrics, each of the plurality of metrics indicating a real time metric of the data relative to a time that the data is profiled [upon detecting a user interaction with the linking item, generating and displaying, along with the visual summary, a detail record for a particular feature associated with the linking item, wherein the detail record includes at least one of:]. A graphical visualization may be displayed in a user interface. For example, the graphical visualization that is generated may be sent to a client device to cause the client device to display the graphical visualization in a user interface at the client device. In some embodiments, a graphical visualization may provide profiling results [a project identifier for a project that includes the instantiated feature, an instantiated feature identifier, an instantiated feature configuration parameter, a SQL query associated with the instantiated feature, or feature versioning information].”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Stojanovic with teachings of Misra and SHAH for the same reasons disclosed for claim 1.
Misra in view of Stojanovic and SHAH do not teach:
wherein the instantiated set of features is shown as a linking item between a first node in a first set of nodes, the first node indicative of the input dataset,
a second node in a second set of nodes, the second node indicative of the set of features.
Gupta teaches:
wherein the instantiated set of features is shown as a linking item between a first node in a first set of nodes, the first node indicative of the input dataset,
(Gupta, “[0050] Each of the node devices may additionally analyze its corresponding data set [the first node indicative of the input dataset] portion to determine one or more features of the data items therein at least partially in parallel with others of the node devices [wherein the instantiated set of features is shown as a linking item between a first node in a first set of nodes]. By way of example, each node device may determine the type(s) of data within its corresponding data set portion, including and not limited to, binary values, integer values, floating point values, text characters, bit mapped still images, frames of video data, samples of audio data, etc. Also by way of example, each node device may determine a density and/or sparsity of the data items, and/or may determine a size of the data items within its corresponding data set portion.”)
Gupta, Misra, Stojanovic and SHAH are related to the same field of endeavor (i.e.: recommendation platform). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Gupta with teachings of Misra, Stojanovic and SHAH to add context based recommendation and ranking of actions to support decision through prioritized recommendation. (Gupta, Abstract).
Misra, Stojanovic, SHAH and Gupta do not teach:
a second node in a second set of nodes, the second node indicative of the set of features.
Rausch teaches:
a second node in a second set of nodes, the second node indicative of the set of features.
(Rausch, “[0051] In some embodiments, such analyses and provision of observation data indicative of features of the data items within each of the data set portions [and a second node in a second set of nodes, the second node indicative of the set of features] to the coordinating device may be at least partially enabled by the provision of the annotated metadata to each of the node devices. Thus, one or more of the node devices may employ the indications of structural features of the whole of the selected data set provided in the annotated metadata as input to such an analysis of the data items within one or more of the data set portions.”)
Rausch, Misra, Stojanovic, SHAH and Gupta are related to the same field of endeavor (i.e.: recommendation platform). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Rausch with teachings of Misra, Stojanovic, SHAH and Gupta to add automated action selection and execution based on similarities to previous analyzed datasets to leverage past actions to efficiently process new datasets. (Rausch, Abstract).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Nia et al., Pub. No.: US20220198277A1.
Nia teaches generative adversarial networks to ensure realistic local samples and surrogate models to provide machine learning (ML) explainability (MLX). Based on many features, an embodiment trains an ML model. The ML model inferences an original inference for original feature values respectively for many features.
Patra et al., Pub. No.: US11205050B2.
Patra teaches learning property graph representations edge-by-edge. In an embodiment, an input graph is received. The input graph comprises a plurality of vertices and a plurality of edges. Each vertex of the plurality of vertices is associated with vertex properties of the respective vertex. A vertex-to-property mapping is generated for each vertex of the plurality of vertices.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to MATIYAS T MARU whose telephone number is (571)270-0902 or via email: matiyas.maru@uspto.gov. The examiner can normally be reached Monday - Friday (8:00am - 4:00pm) EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to
use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
Michelle Bechtold can be reached on (571)431-0762. The fax phone number for the organization were this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users.
To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA)
or 571-272-1000.
/M.T.M./ Examiner, Art Unit 2148
/MICHELLE T BECHTOLD/ Supervisory Patent Examiner, Art Unit 2148