DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Allowable Subject Matter
Claims 5, 6, 12, and 14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-4, 8-11, 18, and 20 are rejected under 35 U.S.C. 102(a)(1) and (a)(2) as being anticipated by Mermoud et al. (US Pub. 20190372859).
Referring to claim 1, Mermoud discloses A non-transitory computer-readable storage medium with instructions stored thereon, the instructions executable by a machine to cause the machine to [fig. 2; par. 88; device 200, memory 240, network assurance process 248, processor 220]:
receive, from a computer system, feature data, wherein the feature data describes a set of patterns within a dataset hosted on the computer system [fig. 6, step 610; pars. 88-91; a remote service receives, from (a network assurance service of) a local network, aggregated metrics (i.e., feature data); the aggregated metrics include clusters (i.e., patterns) of measurements obtained from the local network];
generate a synthetic data set based on the feature data, wherein the synthetic data is to include the set of patterns [fig. 6, step 615; pars. 88-91; the remote service forms a synthetic training dataset using the aggregated metrics (including the clusters)];
train a machine learning model using the synthetic data [fig. 6, step 620; pars. 88-91; the remote service trains a machine learning-based model using the synthetic training dataset]; and
send the trained machine learning model to the computer system for use in inferences run on the computer system [fig. 6, step 625; pars. 36-39 and 88-91; the remote service sends the trained machine learning-based model to the network assurance service; the network assurance service uses the trained machine learning-based model to analyze (e.g., predict) performance of the local network].
Referring to claim 2, Mermoud discloses The storage medium of Claim 1, wherein access to the dataset is restricted and the synthetic data is to model characteristics of the dataset based on the feature data [pars. 65 and 88-91; data (e.g., the aggregated metrics) is shared between the network assurance service and the remote service consistent with very restrictive (privacy) constraints in terms of types and sensitivities of the data that can be shared; the aggregated metrics, from which the synthetic training dataset is formed, are an anonymized and differentially private summary of its underlying dataset].
Referring to claim 3, Mermoud discloses The storage medium of Claim 1, wherein the instructions are executable to further cause the machine to: receive feedback data from the computer system based on the inferences run on the computer system [pars. 39, 81-91; the network assurance service (e.g., MRM) monitors (prediction) performance of the trained machine learning-based model and requests further model training if the performance is below a threshold amount; the request for model training (i.e., feedback data) is provided from the network assurance service to the remote service]; generate new synthetic data based on the feedback data [fig. 6; pars. 39, 81-91; in response to receiving the request for model training, the remote service generates a synthetic training dataset based on aggregated metrics included in the request for model training]; train the machine learning model using the new synthetic data to generated a new trained version of the machine learning model [fig. 6; pars. 39, 81-91; the machine learning-based model is (re-)trained based on the synthetic training dataset]; and send the new trained version of the machine learning model to the computer system [fig. 6; pars. 39, 81-91; the (re-)trained machine learning-based model is sent to the network assurance service].
Referring to claim 4, Mermoud discloses The storage medium of Claim 3, wherein the feedback data comprises additional feature data, wherein the additional feature data describes additional patterns to be considered beyond the set of patterns included in the feature data, and the new synthetic data is generated to include the set of patterns and additional patterns [fig. 6; pars. 39, 81, 82, and 84-91; the network assurance service may add noise to the aggregated metrics, and the aggregated metrics (including the noise) are used to form the synthetic training dataset].
Referring to claim 8, Mermoud discloses The storage medium of Claim 1, wherein the machine learning model comprises one of a predictive model, a forecasting model, or a classification model [pars. 36-39 and 48-51; the machine learning-based model may include a cognitive analytics (i.e., classification) model, a predictive analytics model, or a trending analytics (i.e., forecasting) model].
Referring to claim 9, Mermoud discloses A non-transitory computer-readable storage medium with instructions stored thereon, the instructions executable by a machine to cause the machine to [fig. 2; par. 83; device 200, memory 240, network assurance process 248, processor 220]:
identify a dataset within a corpus of data hosted on a first computing system [fig. 5; pars. 82-87; a network assurance service obtains measurements from a local network];
perform a statistical analysis of the dataset to detect a set of patterns in the dataset [fig. 5, step 510; pars. 72, 73, 82-87; the network assurance service clusters the measurements into measurement clusters (i.e., patterns) using one or more (statistical) clustering techniques];
generate feature data from the statistical analysis to describe a set of features within the dataset based on the set of patterns [fig. 5, step 515; pars. 82-87; aggregated metrics are computed for the measurement clusters];
send the feature data to a second computing system [fig. 5, step 520; pars. 82-87; a model computation request including the aggregated metrics is sent to a remote service];
receive a trained machine learning model from the second computing system, wherein the trained machine learning model is trained by the second computing system using synthetic data generated on the second computing system based on the feature data [fig. 5, step 525; pars. 82-87; a trained machine learning-based model is received from the remote service, where the trained machine learning-based model is trained using a synthetic training dataset formed using the aggregated metrics];
provide input data from the corpus of data to the trained machine learning model to perform inferences based on the input data [fig. 5, step 530; pars. 36-39 and 82-87; the trained machine learning-based mode is used to analyze (e.g., perform predictions based on) measurements from the local network]; and
determine a degree of accuracy of the trained machine learning model based on the inferences [pars. 39, 81-91; the network assurance service (e.g., MRM) monitors (prediction) performance of the trained machine learning-based model and requests further model training if the performance is below a threshold amount (i.e., a required degree of accuracy)].
Referring to claim 10, Mermoud discloses The storage medium of Claim 9, wherein the instructions are executable to further cause the machine to send the feature data with a request to develop the trained machine learning model at the second computing system [fig. 5, step 520; pars. 82-87; note the model computation request], and access to the corpus of data is withheld from the second computing system [pars. 65 and 82-87; data (e.g., the aggregated metrics) is shared between the network assurance service and the remote service consistent with very restrictive (privacy) constraints in terms of types and sensitivities of the data that can be shared; the aggregated metrics, from which the synthetic training dataset is formed, are an anonymized and differentially private summary of its underlying dataset].
Referring to claim 11, Mermoud discloses The storage medium of Claim 9, wherein the instructions are executable to further cause the machine to: generate feedback data based on results of the inferences [pars. 39 and 81-87; the network assurance service (e.g., MRM) monitors (prediction) performance of the trained machine learning-based model and requests further model training (i.e., provides feedback data) if the performance is below a threshold amount]; send the feedback data to the second computing system [pars. 39 and 81-87; the request for model training (i.e., the feedback data) is provided from the network assurance service to the remote service]; and receive a new version of the trained machine learning model generated by the second computing system based on the feedback data [fig. 6; pars. 39, 81-87; the network assurance service receives the (re-)trained machine learning-based model that is trained based on aggregated metrics included in the request for model training].
Referring to claim 18, Mermoud discloses A system comprising:
a first computing system [fig. 2; par. 83; device 200 providing a network assurance service] comprising:
one or more first data processors [fig. 2; par. 83; processor 220];
a data store [fig. 2; par. 83; memory 240] to store a dataset [pars. 65 and 66; the network assurance service stores measurements obtained from a local network (e.g., an underlying dataset)]; and
a feature extraction tool [fig. 2; par. 83; the network assurance service], executable by the one or more first data processors [fig. 2; par. 83; processor 220] to perform a statistical analysis of the dataset to generate feature description data to describe a set of features within the dataset [pars. 72, 73, 82-87; the network assurance service clusters the measurements into measurement clusters (i.e., feature description data to describe a set of features within the dataset) using one or more (statistical) clustering techniques]; and
a second computing system coupled to the first computing system by a network [fig. 2; par. 88; (a second) device 200 provides a remote service], wherein the dataset is inaccessible to the second computing system [pars. 65 and 82-87; the measurements are shared between the network assurance service and the remote service consistent with very restrictive (privacy) constraints in terms of types and sensitivities of data that can be shared; only aggregated metrics, which are an anonymized and differentially private summary of the underlying dataset, are provided to the remote service], and the second computing comprises:
a data synthesizer [fig. 2; par. 88; the remote service], executable by the one or more second data processors to [fig. 2; par. 88; processor 220]:
receive the feature description data [pars. 88-91; the remote service receives the aggregated metrics]; and
generate a synthetic dataset based on the feature description data [pars. 88-91; the aggregated metrics are used to form a synthetic training dataset], wherein the synthetic dataset models the dataset and is to include the set of features [pars. 65 and 88-91; note that the aggregated metrics, from which the synthetic training dataset is formed, are an anonymized and differentially private summary of the underlying dataset]; and
a model trainer [fig. 2; par. 88; the remote service], executable by the one or more second data processors to: train a machine learning model with the synthetic data set to generate a trained machine learning model [pars. 88-91; a machine learning-based model is trained using the synthetic training dataset], wherein the trained machine learning model is for use by the first computing system and is to use data from the data store as an input [fig. 2; pars. 82-87 and 88-91; the trained machine learning-based model is sent to the network assurance service and is used to analyze measurements obtained from the local network (i.e., the underlying dataset)].
Referring to claim 20, Mermoud discloses The system of Claim 18, wherein the feature extraction tool is provided to the first computing system by the second computing system in association with training of the machine learning model [par. 70; the remote service (e.g., cloud service 302) may provide some degree of control over the network assurance service (e.g., local service 302a), such as changing its configuration, adjusting its operation, or the like; the network assurance service may be seen as an extension of the remote service].
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Mermoud in view of Wang et al. (US Pub. 20260087326).
Referring to claim 7, Mermoud does not appear to explicitly disclose The storage medium of Claim 1, wherein the synthetic data is generated through an artificial intelligence algorithm using the feature data as an input.
However, Wang discloses The storage medium of Claim 1, wherein the synthetic data is generated through an artificial intelligence algorithm using the feature data as an input [pars. 58-63; a synthetic data service evaluates data for patterns, labels the data, then communicates the labeled data to a synthetic data engine, which may be a machine learning or artificial intelligence engine for generating synthetic data].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the remote service taught by Mermoud so that the remote service uses a machine learning or artificial intelligence engine as taught by Wang to generate the synthetic training dataset, with a reasonable expectation of success. The motivation for doing so would have been to improve the quality of synthetic data so that it closely mimics real data, including data with missing values having complex interactions [Wang, pars. 24-28 and 53-55].
Claims 13 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Mermoud in view of Bhargava et al. (US Pub. 20230368070).
Referring to claim 13, Mermoud discloses The storage medium of Claim 11, wherein the instructions are executable to further cause the machine to determine whether the inferences meet a set of key performance indicators (KPIs) for the machine learning model, wherein the feedback data identifies the degree to which the inferences met the set of KPIs [pars. 39, 81-91; the network assurance service (e.g., MRM) monitors (prediction) performance of the trained machine learning-based model and requests further model training if the performance is below a threshold amount (i.e., a required degree of performance); the request for model training (i.e., feedback data) is provided from the network assurance service to the remote service].
Mermoud does not appear to explicitly disclose determining whether the inferences meet a set of key performance indicators (KPIs) and identifying the degree to which the inferences met the set of KPIs.
However, Bhargava discloses determining whether the inferences meet a set of key performance indicators (KPIs) and identifying the degree to which the inferences met the set of KPIs [pars. 95-99; a new/existing predictive model is trained using a training synthetic dataset then validated using a testing synthetic dataset, where the validating comprises adjusting hyperparameters (i.e., re-training or tuning) the predictive model based on key performance indicators (KPIs), which entails determining whether and to what degree the KPI are met].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the model training based on a performance threshold taught by Mermoud so that performance is validated using KPI as taught by Bhargava, with a reasonable expectation of success. The motivation for doing so would have been to evaluate the performance of the machine learning-based model using objective metrics [Bhargava, par. 42].
Referring to claim 19, Mermoud discloses The system of Claim 18, wherein the first computing system further comprises: an inference engine [fig. 2; par. 83; the network assurance service], executable by the one or more first data processors to use the trained machine learning model to perform inferences on data in the corpus of data [fig. 2; pars. 36-39, 82-87, and 88-91; note that the trained machine learning-based model is used to analyze (e.g., perform predictions based on) measurements obtained from the local network (i.e., the underlying dataset)]; and a results analyzer [fig. 2; par. 83; the network assurance service], executable by the one or more first data processors to: determine from the inferences whether the machine learning model meets a set of key performance indicators for the machine learning model; and send feedback data to the second computing system to indicate how the machine learning model meets or does not meet the key performance indicators [pars. 39, 81-91; the network assurance service (e.g., MRM) monitors (prediction) performance of the trained machine learning-based model and requests further model training if the performance is below a threshold amount (i.e., a required degree of performance); the request for model training (i.e., feedback data) is provided from the network assurance service to the remote service].
Mermoud does not appear to explicitly disclose determining whether the machine learning model meets a set of key performance indicators (KPIs) and indicating how the machine learning model meets or does not meet the set of KPIs.
However, Bhargava discloses determining whether the inferences meet a set of key performance indicators (KPIs) and indicating how the machine learning model meets or does not meet the set of KPIs [pars. 95-99; a new/existing predictive model is trained using a training synthetic dataset then validated using a testing synthetic dataset, where the validating comprises adjusting hyperparameters (i.e., re-training or tuning) the predictive model based on key performance indicators (KPIs), which entails determining whether and indicating how the KPI are met or not met].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the model training based on a performance threshold taught by Mermoud so that performance is validated using KPI as taught by Bhargava, with a reasonable expectation of success. The motivation for doing so would have been to evaluate the performance of the machine learning-based model using objective metrics [Bhargava, par. 42].
Claims 15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Mermoud in view of Koide et al. (US Pub. 20230179627).
Referring to claim 15, Mermoud does not appear to explicitly disclose The storage medium of Claim 9, wherein the dataset comprises image data or video data, and the set of features comprises a set of graphical features recurring at statistically significant frequencies within the image data or video data.
However, Koide discloses The storage medium of Claim 9, wherein the dataset comprises image data or video data, and the set of features comprises a set of graphical features recurring at statistically significant frequencies within the image data or video data [par. 36; a training model is generated using, as training data, one or more word/phrase appearance frequency features, image appearance frequency features, HTML features, and communication log features].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the synthetic training dataset taught by Mermoud so that the synthetic training dataset includes features particular to a task (e.g., detecting false information removal websites) as taught by Koide, with a reasonable expectation of success. The motivation for doing so would have been to enable highly accurate detection of false information removal websites (i.e., perform the task more accurately) [Koide, pars. 12, 36, and 74].
Referring to claim 16, Mermoud does not appear to explicitly disclose The storage medium of Claim 9, wherein the dataset comprises text data, and the set of features comprises one or more of recurring letter patterns, recurring word patterns, or recurring sentence fragments within the text data.
However, Koide discloses The storage medium of Claim 9, wherein the dataset comprises text data, and the set of features comprises one or more of recurring letter patterns, recurring word patterns, or recurring sentence fragments within the text data [par. 36; a training model is generated using, as training data, one or more word/phrase appearance frequency features, image appearance frequency features, HTML features, and communication log features].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the synthetic training dataset taught by Mermoud so that the synthetic training dataset includes features particular to a task (e.g., detecting false information removal websites) as taught by Koide, with a reasonable expectation of success. The motivation for doing so would have been to enable highly accurate detection of false information removal websites (i.e., perform the task more accurately) [Koide, pars. 12, 36, and 74].
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Mermoud in view of Foschini et al. (US Pub. 20230245777).
Referring to claim 17, Mermoud does not appear to explicitly disclose The storage medium of Claim 9, wherein the dataset comprises time series sensor data, and the set of features correspond to patterns recurring at statistically significant frequencies within the time series sensor data.
However, Foschini discloses The storage medium of Claim 9, wherein the dataset comprises time series sensor data, and the set of features correspond to patterns recurring at statistically significant frequencies within the time series sensor data [pars. 24 and 72; extracted features and learned representations from sensor data are used to cluster and segment received data records to search for similar users or days, detect trends and outliers in health and behavior, and visualize complex time series datasets to build understanding and identify issues].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the synthetic training dataset taught by Mermoud so that the synthetic training dataset includes extracted features and learned representations from sensor data as taught by Foschini, with a reasonable expectation of success. The motivation for doing so would have been to search for similar data, detect trends and outliers, and visualize complex time series datasets to build understanding and identify issues [Foshini, par. 72].
Conclusion
The following prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Chandelier et al. (US Pub. 20210232705) discloses determining identifying features for identifying patterns in images, reports, 1-D data such as EKG data and generating synthetic anonymized data based on the features.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GRACE PARK whose telephone number is (571)270-7727. The examiner can normally be reached M-F 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, TAMARA KYLE can be reached at (571)272-4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Grace Park/Primary Examiner, Art Unit 2144
