Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This non-final office action is responsive to the U.S. patent application no. 18/885,502 filed on September 13, 2024.
Claims 1-20 are pending.
Claims 1-20 are rejected.
Priority
The application claims priority under 35 U.S.C. 120 to U.S. non-provisional application No. 18/132,859 filed on April 10, 2023, which claims priority under 35 U.S.C. 120 to U.S. non-provisional application No. 16/892,935 filed on June 4, 2020, which claims priority under 35 U.S.C. 119(e) to U.S. Provisional application No. 62/900,537 filed on September 14, 2019.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 9/13/2024 and 8/18/2025 are compliant with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements have been considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) as failing to comply with the enablement requirement. The claims contains subject matter which were not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention.
Claim 1 recites the limitation
“analyzing the one or more first parameters to identify an ontology for the first machine learning model;”
However, this limitation is not found anywhere in the application’s disclosure.
The most relevant disclosure Examiner is able to identify is in paragraph [0356], which discloses that
“[0356] In various embodiments, the method can include generating an ontology based at least in part on analyzing the attributes of the one or more data objects.” Note that the ontology disclosed here is an ontology obtained through analyzing one or more data objects rather than one or more parameters that identify an define properties of the first machine learning model.
Therefore the claim limitation “analyzing the one or more first parameters to identify an ontology for the first machine learning model” is not enabled by the application disclosure.
Independent claims 8 and 15 contain the same limitation therefore have the same issue as claim 1.
Claims 2-7, 9-14 and 16-20 depend from claims 1, 8, and 15 respectively, therefore inherit the enablement issue from their independent claims.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b), as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claim 1 recites the limitation “generating a third result by processing the first data set using the first data set.” It appears that Applicant might have intended to recite “generating a third result by processing the first data set using the first machine learning model.”
Independent claims 8 and 15 contain the same limitation therefore have the same issue as claim 1.
Dependent claims 2-7, 9-14 and 16-20 inherit the issue from their respective independent claims.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 12,118,474. Although the claims at issue are not identical, they are not patentably distinct from each other. The table below shows the mapping between claims 1-7 of the two documents. Claims 8-20 in these two documents can be similarly mapped.
Application no. 18/885,502
Patent No. 12,118,474
1. A computer-implemented method comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters to identify and define properties of the first machine learning model;
generating a first result for the first machine learning model using a first data set as an input to the first machine learning model;
generating a second result for the first machine learning model using a second data set as the input to the first machine learning model, wherein the second data set comprises a labeled data set;
analyzing the one or more first parameters to identify an ontology for the first machine learning model;
using the ontology to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
generating a third result by processing the first data set using the first data set; generating a fourth result by processing the second data set using the second machine learning model;
determining that a model-replacement condition is satisfied based on the third result and the fourth result; and
in response to the determination that the model-replacement condition is satisfied, replacing the first machine learning model with the second machine learning model in a machine learning application.
2. The computer-implemented method of claim 1, further comprising:
storing the second machine learning model in a memory.
3. The computer-implemented method of claim 1,
wherein the replacing the first machine learning model with the second machine learning model is performed in a shadow mode until it is determined that one or more conditions for auto-promoting the second machine learning model to production is satisfied.
4. The computer-implemented method of claim 1, further comprising: generating a log comprising an identification of at least one of: the one or more first parameters, the first machine learning model, the second machine learning model, the first result, and the second result; and storing the log in a memory.
5. The computer-implemented method of claim 4, further comprising analyzing the log to determine one or more patterns.
6. The computer-implemented method of claim 4, further comprising saving supplemental metadata in relation to the second machine learning model, wherein the supplemental metadata includes at least the one or more first parameters the second result.
7. The computer-implemented method of claim 1, wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.
1. A computer-implemented method, the method comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters to identify and define properties of the first machine learning model;
generating a first result for the first machine learning model using a first data set as an input to the first machine learning model;
identifying one or more metrics for the first machine learning model, wherein the metrics define how a performance of the first machine learning model is measured;
generating a second result for the first machine learning model using a second data set as the input to the first machine learning model, wherein the second data set comprises a labeled data set;
comparing the first result with the second result to calculate a first scoring;
analyzing the one or more first parameters to identify an ontology for the first machine learning model;
using the ontology to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
testing the second machine learning model on the first data set to produce a third result; generating a fourth result for the second machine learning model using the second data set as the input to the second machine learning model;
comparing the third result to the fourth result to generate a second scoring; and
based on the second scoring being less than the first scoring, replacing the first machine learning model with the second machine learning model in a machine learning application.
2. The method of claim 1, further comprising:
storing the second machine learning model in a memory.
3. The method of claim 1,
wherein the replacing the first machine learning model with the second machine learning model is performed in a shadow mode until it the second machine learning model satisfies one or more conditions for auto-promoting the second machine learning model to production.
4. The method of claim 1, further comprising: generating a log comprising the one or more first parameters, the first machine learning model, the second machine learning model, the first result, and the second result; and storing the log in a memory.
5. The method of claim 4, further comprising analyzing the log to determine one or more patterns.
6. The method of claim 4, further comprising saving supplemental metadata concerning the second machine learning model, wherein the supplemental metadata includes at least the one or more first parameters the second result.
7. The method of claim 1, wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 11,625,648. Although the claims at issue are not identical, they are not patentably distinct from each other. The table below shows the mapping between claims 1-7 of the two documents. Claims 8-20 in these two documents can be similarly mapped.
Application no. 18/885,502
Patent No. 11,625,648
1. A computer-implemented method comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters to identify and define properties of the first machine learning model;
generating a first result for the first machine learning model using a first data set as an input to the first machine learning model;
generating a second result for the first machine learning model using a second data set as the input to the first machine learning model, wherein the second data set comprises a labeled data set;
analyzing the one or more first parameters to identify an ontology for the first machine learning model;
using the ontology to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
generating a third result by processing the first data set using the first data set;
generating a fourth result by processing the second data set using the second machine learning model;
determining that a model-replacement condition is satisfied based on the third result and the fourth result; and in response to the determination that the model-replacement condition is satisfied, replacing the first machine learning model with the second machine learning model in a machine learning application.
3. The computer-implemented method of claim 1, wherein the replacing the first machine learning model with the second machine learning model is performed in a shadow mode until it is determined that one or more conditions for auto-promoting the second machine learning model to production is satisfied.
4. The computer-implemented method of claim 1, further comprising: generating a log comprising an identification of at least one of: the one or more first parameters, the first machine learning model, the second machine learning model, the first result, and the second result; and storing the log in a memory.
5. The computer-implemented method of claim 4, further comprising analyzing the log to determine one or more patterns.
6. The computer-implemented method of claim 4, further comprising saving supplemental metadata in relation to the second machine learning model, wherein the supplemental metadata includes at least the one or more first parameters the second result.
7. The computer-implemented method of claim 1, wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.
1. A method for automatically composing a machine learning pipeline for use in a production environment executed as a background process of a machine learning application, the method comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters, wherein the one or more first parameters identify and define properties of the first machine learning model;
identifying the one or more first parameters that affect a speed or a quality of a learning process of the first machine learning model;
generating one or more first results for a first data set as an input to the first machine learning model and a subset of the one or more first parameters;
identifying one or more metrics for the first machine learning model, wherein the metrics define how a performance of the first machine learning model is measured;
comparing the one or more first results with a second data set to calculate a first scoring based at least in part on a difference in the one or more first results with the second data set, wherein the second data set comprises ground truth data;
analyzing the first data set to identify an ontology;
using the ontology of the first data set to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
testing the second machine learning model on the first data set to produce one or more second results;
comparing the one or more second results with the second data set to generate a second scoring; and
based on the second scoring being less than the first scoring, replacing the first machine learning model with the second machine learning model for the machine learning application.
3. The method of claim 1, wherein the replacing the first machine learning model is replaced with the second machine learning model in a shadow mode until it the second machine learning model satisfies one or more conditions for auto-promoting the second machine learning model to production.
4. The method of claim 1, further comprising: generating a log comprising the one or more first parameters, the first machine learning model, the second machine learning model, the one or more first results, and the one or more second results; and storing the log in a memory.
5. The method of claim 4, further comprising analyzing the log to determine one or more patterns.
6. The method of claim 4, further comprising saving supplemental metadata concerning the second machine learning model, wherein the supplemental metadata includes at least the one or more first parameters the one or more second results.
7. The method of claim 1, wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-6, 8-13 and 15-20 are rejected under 35 U.S.C. 102(a)(2) as being unpatentable over Di Pietro et al. (U.S. 2019/0370218).
Regarding claim 1, Di Pietro disclosed a computer-implemented method comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters to identify and define properties of the first machine learning model (Di Pietro disclosed in [0036] that “network assurance process 248 may employ one or more supervised, unsupervised, or semi-supervised machine learning models” and in [0035] that “One very common pattern among machine learning techniques is the use of an underlying model M, whose parameters are optimized for minimizing the cost function associated to M, given the input data.”);
generating a first result for the first machine learning model using a first data set as an input to the first machine learning model (Di Pietro disclosed in [0064] that “a network assurance service uses a first machine-learning based model that is locally deployed to a network to assess a set of input features comprising measurements from the network.” De Prieto also disclosed in [0039, 0046, 0049, 0077 and 0103] that “Predictive Analytics Model(s) … may be configured to predict user experiences” said “user experience” prediction anticipates the first result in the claim);
generating a second result for the first machine learning model using a second data set as the input to the first machine learning model, wherein the second data set comprises a labeled data set (Di Pietro disclosed in [0049] that “and “The predictive analytics model(s) of analyzer 312 may be able to predict other metrics such as the expected throughput for a client using a specific application. In yet another example, the predictive analytics model(s) may predict the user experience for voice/video quality using network variables (e.g., a predicted user rating of 1-5 stars for a given session, etc.), as function of the network state.” Said “throughput” prediction anticipates the second result in the claim);
analyzing the one or more first parameters to identify an ontology for the first machine learning model (Di Pietro disclosed in [0100] that “the input features of the generated model 416 can be chosen by selecting a subset of input features that are available to local service 302a (e.g., as provided by the customer profile message sent to cloud service 302 and which is compatible with the available resources (as provided by the stats message from PPM 410.” This is equivalent to identifying the ontology of the input features, i.e. first parameters )
using the ontology to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model (Di Pietro, [0098, 0099, 0100], “MSE 412… may be configured to implement a model replacement strategy that includes … choosing one replacement candidate from a large library of pre-computed models 416; and/or triggering the re-computation of a custom model 416 by training engine 414, based on a subset of the available datasets. Doing so allows cloud service 302 to create a model 416 which matches exactly the current requirements of the on-premise infrastructure”);
generating a third result by processing the first data set using the first data set (Di Pietro, Fig. 5, step 515 and [0103], “the network assurance service may monitor the performance of the first machine learning-based model”);
generating a fourth result by processing the second data set using the second machine learning model; determining that a model-replacement condition is satisfied based on the third result and the fourth result (Di Pietro disclosed in Fig. 5, steps 520, 525 and [0105] that “…the service may attempt to select a model that better optimizes the tradeoff between resource consumption by a model and the ability of the model to properly predict or assess the behavior of the network. … the service may select the second model, if the second model consumes fewer resources than the first model currently deployed to the network and/or is expected to have higher precision or recall.” Di Pietro also disclosed in [0079] that “model selection engine (MSE) 412, which is configured to select a machine learning-based model for execution by local service 302a, based on the outputs of MPM 408 and/or PPM 410.”); and
in response to the determination that the model-replacement condition is satisfied, replacing the first machine learning model with the second machine learning model in a machine learning application (Di Pietro disclosed in Fig. 5, step 530 and [0106] that “the network assurance service may deploy the selected second machine learning-based model to the network as a replacement for the first machine learning-based model”).
Claim 8 lists substantially the same elements as claim 1 in product form rather than method form. Therefore, the rejection rationale for claim 1 applies equally as well to claim 8.
Claim 15 lists substantially the same elements as claim 1 in system form rather than method form. Therefore, the rejection rationale for claim 1 applies equally as well to claim 15.
Regarding claims 2, 9 and 16, Di Pietro disclosed the subject matter of claims 1, 8 and 15 respectively. Di Pietro further disclosed storing the second machine learning model in a memory (Di Pietro disclosed in Fig. 4 and [0047] that “Machine learning-based analyzer 312 may include any number of machine learning models” implying that the models are stored in a memory on-premise).
Regarding claims 3, 10 and 17, Di Pietro disclosed the subject matter of claims 1, 8 and 15 respectively. Di Pietro further disclosed wherein the replacing the first machine learning model with the second machine learning model is performed in a shadow mode until it is determined that one or more conditions for auto-promoting the second machine learning model to production is satisfied (Di Pietro disclosed in Fig. 5, steps 520, 525 and [0105] that “…the service may attempt to select a model that better optimizes the tradeoff between resource consumption by a model and the ability of the model to properly predict or assess the behavior of the network. … the service may select the second model, if the second model consumes fewer resources than the first model currently deployed to the network and/or is expected to have higher precision or recall.” Which means that the selection and replacement is performed in the background and the second model is not selected unless its performance (e.g., resource consumption, precision) is better than that of the first model that is in use.).
Regarding claims 4, 11 and 18, Di Pietro disclosed the subject matter of claims 1, 8 and 15 respectively. Di Pietro further disclosed generating a log comprising an identification of at least one of: the one or more first parameters, the first machine learning model, the second machine learning model, the first result, and the second result; and storing the log in a memory (Di Pietro disclosed in [0070-0078] that MPM 408 may assess the performance of model(s) 406 based on any or all of the following: feedback provided by one or more users via the UI regarding alerts raised by model(s) 406; feedback from one or more other systems in the network, such as a network security system, etc., that are fed the outputs of model(s) 406; various performance metrics generated by a model 406 itself (e.g., prediction error, etc.)” the feedback information disclosed by Di Pietro anticipates the log in the claim).
Regarding claims 5, 12 and 19, Di Pietro disclosed the subject matter of claims 4, 11 and 18 respectively. Di Pietro further disclosed analyzing the log to determine one or more patterns (Di Pietro disclosed in [0070] that the model performance monitor (MPM) may monitor and assess the performance of machine learning-based model(s) 406 executed on-premise by local service 302a using feedbacks/logs).
Regarding claims 6, 13 and 20, Di Pietro disclosed the subject matter of claims 4, 11 and 18 respectively. Di Pietro further disclosed saving supplemental metadata in relation to the second machine learning model, wherein the supplemental metadata includes at least the one or more first parameters the second result (Di Pietro disclosed in [0077] that “another form of feedback that MPM 408 may use to compute the performance score for model(s) 406 may come from another system that leverages the assessment results of model(s) 406”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 7 and 14 are rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Di Pietro as applied to claims 1 and 8 above, and further in view of Faulhaber et al. (U.S. 2019/0156247).
Regarding claims 7 and 14, Di Pietro disclosed the subject matter of claims 1 and 8 respectively.
Di Pietro might not have disclosed but Faulhaber disclosed wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model (Faulhaber disclosed in [0089] that “the ML training container 830 generates model data and stores the model data in a file system of the ML training container 830. The model data includes characteristics of the machine learning model being trained, such as a number of layers in the machine learning model, hyperparameters of the machine learning model, coefficients of the machine learning model, weights of the machine learning model, and/or the like”).
One of ordinary skill in the art, before the effective filing date of the claimed invention, would have been motivated to combine Di Pietro and Faulhaber because both references disclosed methods for dynamically deployment of machine learning models based so as to ensure a level of performance/precision of models in use (Di Pietro, Abstract and [0074, 0105]; Faulhaber, Abstract).
Therefore it would have been obvious for one to combine Faulhaber’s teaching of the model data (i.e. metadata) with Di Pietro’s disclosure to realize that the model metadata in Di Pietro can include information about the model’s number of layers or number of parameters.
Related Prior Art
Parker (US 2017/0286839) is directed to Systems and methods of selecting machine learning models/algorithms for a candidate dataset.
Larson et al. (US 16,650,928) disclosed a computer network architecture for a pipeline of models with machine learning and artificial intelligence for healthcare outcomes
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHIRLEY X ZHANG whose telephone number is (571)270-5012. The examiner can normally be reached 8:30am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Joon H Hwang can be reached at 571-272-4036. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHIRLEY X ZHANG/Primary Examiner, Art Unit 2447
Prosecution Insights