DETAILED ACTION
This action is responsive to the application filed on 11/20/2025. Claims 1-19 are pending and have been examined.
This action is Non-final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C.
120, 121, 365(c), or 386(c) is acknowledged.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11/20/2025 has been entered.
Response to Arguments
Argument 1: The applicant argues that the claims are not directed to an abstract idea because they recite a specific, technical way of monitoring and verifying machine-learning training: repeatedly computing training and validation quality measures over successive training epochs, forming corresponding learning progressions, and inputting those progressions into a verification model during the learning process to generate an assessment of training quality; they contend this is more than just “analyzing data” because it is tied to improving the training process (e.g., faster, more reliable training with less expert intervention) in the context of a microscopy system, rather than merely organizing information or presenting results.
Response to Argument 1: The Examiner has considered Applicant’s arguments but finds them unpersuasive. The claims still focus on an abstract idea, namely, collecting and calculating training/validation performance values, organizing those values into “learning progressions,” and using those progressions to make a judgment about training quality. That is, at its core, data analysis and evaluation (mathematical calculations and mental judgment), even if performed by a computer. While Applicant argues this improves training in a technical way, the claims mainly recite desired results (calculate measures, form progressions, generate an assessment) rather than a specific technological improvement to the computer or training system itself. The microscopy elements (microscope, camera, computing device, imaging program) are also stated at a high level and are used in their normal way, so they mainly describe the environment where the idea is applied, not a practical application that adds “significantly more.” Therefore, the claims remain ineligible under 35 U.S.C. 101 and the rejection is maintained.
Argument 2: The applicant argues that the prior art rejections under 102/103 are flawed because the examiner’s mapping relies on an incorrect reading and does not show the newly emphasized limitations: they contend the examiner treated Wang’s “hyperparameters” (settings used to train different models) as if they were the claimed “trainable internal model parameters,” and that Wang mainly describes comparing performance across different training runs/models (e.g., different hyperparameter sets) rather than repeatedly computing training and validation quality measures across different epochs of the same training process, forming training/validation learning progressions from those successive measures, and using those progressions to generate a training-quality assessment during learning; for the 103 combination, they argue the secondary reference(s) cited for “quality measures/progressions” likewise evaluate metrics across different models/classifiers (not epoch-by-epoch for the same parameter set), so the combination still does not supply the missing epoch-based progression features, and they also argue the examiner’s motivation-to-combine is too conclusory (e.g., just stating it would improve accuracy/interpretability without explaining the specific modification and predictable result); additionally, for the claim(s) relying on Hutter, they argue Hutter is effectively multiple references because it is a compilation with different authors/chapters, so the rejection is really an improper multi-way combination.
Response to Argument 2: The Examiner has considered Applicant’s arguments but finds them unpersuasive because Feurer, rather than Wang, supplies the explicit teachings distinguishing trainable internal model parameters from hyperparameters and teaches iteration/epoch-based learning curves and quality assessment during training (e.g., measured each iteration, modeled/extrapolated for predictive termination). Wang is applied to the training system/workflow aspects of iteratively running training portions and evaluating performance (including use of held-out or testing data), and the combination yields the predictable result of repeatedly computing training/validation measures across stages of learning, forming progressions, and assessing training quality during learning. Applicant’s “Hutter is multiple references” argument is also unpersuasive because a compilation is a single printed publication and reliance on multiple chapters/sections does not transform it into an improper multi-way combination.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition
of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the
conditions and requirements of this title.
Claims 1-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1,
Step 1: Claim 1 is directed to a system, which is in the machine category, and one of four allowable subject matters.
Step 2A Prong 1:(a) “calculating a validation data quality measure for the current values of the trainable internal model parameters during the learning process, the validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from validation data during the learning process,” – The limitation is directed to calculating a training data quality measure and a validation data quality measure for current values of trainable internal model parameters during a learning process. The limitation is a process that can be completed in the human mind using evaluation, observation, and judgment, and thus it is directed to a mental process.
(b) “which uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs” – The limitation is directed to use same set of trainable internal model parameters and quality measures being determined for different values of same set of parameters across epochs. The limitation amounts to observation and evaluation of calculated results across iterations and thus is directed to a mental process.
(c) “forming a training learning progression from successively determined training data quality measures and forming a validation learning progression from successively determined validation data quality measures” – The limitation is directed to forming progressions (i.e., tracking/organizing successively determined quality measures) for both training and validation. The limitation is directed to a process that can be performed using evaluation, observation, and judgment (with aid of pen and paper), and thus the limitation is considered to be a mental process.
(d) “wherein the training system comprises a verification model, which is fed with the training learning progression and the validation learning progression during the learning process; and wherein the verification model is configured to generate a quality assessment of the learning process… depending on the training learning progression and the validation learning progression” – This limitation is directed to generating a quality assessment based on the training and validation learning progressions, which is considered to be a mental process that amounts to observation and evaluation of the progressions to judge the quality of the learning process.
Step 2A Prong 2 and Step 2B:(a) “A microscopy system comprising: at least one microscope, which comprises at least one camera for capturing a microscope image and a computing device, wherein the computing device comprises an imaging processing program for processing the microscope image using a machine learning model,” – This limitation recites a microscope system where a camera captures an image and a computing device executes an imaging processing program to process the image using a machine learning model. This limitation recites mere instructions to apply a camera and an image processing program to be generically executed on a computer (computing device), and the microscope being used in its ordinary capacity/high generality, for which cannot be integrated into a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
(b) “wherein the machine learning model includes a set of trainable internal model parameters for processing input training data to calculate a model output, the trainable internal model parameters being distinct from hyperparameters that define training settings or model architecture…repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process” – This limitation is directed to using a machine learning model with trainable internal model parameters (distinct from hyperparameters) to process input training data and calculate an output. The limitation is directed to manipulating and producing results from data using a machine learning model, which is an insignificant, extra-solution activity that cannot be integrated into a practical application (see MPEP 2106.05(g)). Further, under Step 2B, using a machine learning model with parameters to process input data and produce an output, and repetitive calculations is a well-understood, routine, and conventional activity that cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
(c) “and a training system for carrying out a learning process of the machine learning model, wherein the training system is configured for: adjusting values of the trainable internal model parameters of the machine learning model successively in the learning process depending on training data processed with the trainable internal model parameters;” – This limitation recites a training system configured to adjust parameter values successively based on training data. Mere data manipulation/adjustment based on data is considered to be an insignificant, extra-solution activity and cannot be integrated into a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, the act of adjusting values depending on gathered, training data processed in the parameters is a well-understood, routine, and conventional activity (WURC) that cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
(d) “wherein the validation data is not used in the learning process to adjust the values of the trainable internal model parameters values and instead the validation data is only used for calculating the validation data quality measure;” – This limitation amounts to no more than specifying how validation data is used (evaluation only), which is a further limitation to a field of use/environment and does not integrate into a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(h)).
(e) “repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process which uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs;” – This limitation is directed to repetitive calculation across epochs, which is an insignificant, extra-solution activity that cannot be integrated into a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, the act of repetitive calculation is also a well-understood, routine, and conventional activity (WURC), and cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
(f) “wherein the training system comprises a verification model, which is fed with the training learning progression and the validation learning progression during the learning process” – This limitation recites feeding the progressions into a verification model during the learning process, which is considered to be mere limiting to a particular field of use/particular environment (type of model and progressions within technology), and therefore cannot be integrated into a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 1 is non-patent eligible.
Regarding claim 2,
Step 1: This claim is directed to a method, which is considered to be a process, and it is an allowable subject matter.
Step 2A Prong 1:
“generates a quality assessment of the learning process of the machine learning model depending on the training learning progression and the validation learning progression.” – This limitation is directed assessing the progress of the learning process on the model based on learning and validation learning progression, which can be performed using evaluation/judgement, and therefore is considered a mental process.
“calculating a training data quality measure for current values of the trainable internal model parameters during the learning process, the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data; and calculating a validation data quality measure for the current values of the trainable internal model parameters values during the learning process, the validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters values from validation data during the learning process,..forming a training learning progression from successively determined training data quality measures; and forming a validation learning progression from successively determined validation data quality measures,” -- The limitations are directed to calculating quality measures of values and forming training model parameters. The limitation is directed to the use of mathematical calculations/operations, and thus it is directed to math.
Step 2A Prong 2 and Step 2B:
“A method for monitoring a learning process of a machine learning model, wherein the machine learning model includes a set of trainable internal model parameters for processing input training data to calculate a model output” -- The limitation is directed to a method of monitoring a learning process of a ML model which includes model parameters, trained for processing inputs to calculate a mode output. Similar to the above claim, it is an insignificant, extra-solution activity that cannot be integrated into a practical application (see MPEP 2106.05(g)). Further, under Step 2B, using a machine learning model with parameters to process input data and produce an output, and repetitive calculations is a well-understood, routine, and conventional activity that cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“the method comprising: launching a learning process including adjusting values of the trainable internal model parameters-values successively during the learning process depending on training data processed with the trainable internal model parameters;” -- The limitation recites a method that comprises launching a learning process that includes adjusting parameters depending on training data with the internal model parameters, and it is considered mere data observation. The limitation is considered to be an insignificant, extra-solution activity, and cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under step 2B, the act of storing/and retrieving information in memory (training system), is a well-understood, routine, and conventional activity (WURC), and cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)(iv).
“the trainable internal model parameters being distinct from hyperparameters that define training settings or model architecture” -- The limitation recites model parameters being distinct from the hyperparameters that will define training settings/model architecture. The limitation amounts to no more than mere further limiting to a field of use/environment, and it is not integrating to a practical application, nor providing significantly more than the judicial exception (see MPEP 2106.05(h)).
“repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process which uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs;” -- Analogous to claim 1 limitations, it is seen as an insignificant, extra-solution activity that cannot be integrated to a practical application (see MPEP 2106.05(g)). It is also a well-understood, routine, and conventional activity (WURC) that cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“wherein the training learning progression and the validation learning progression are fed to a verification model during the learning process; and” -- The limitation recites that the learning and validation progressions will merely be applied to the model during the learning process. The limitation does not integrate to a practical application, nor does it provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Regarding claim 3,
Step 1: Claim 3 depends on claim 2, which is deemed as a process. Claim 3 satisfies Step 1.
Step 2A Prong 1
“during an ongoing training of the machine learning model before a predetermined stopping criterion of the training is reached.” – This limitation is directed to ongoing training of a ML model before a pre-set threshold of training has been reached, which can be done by observation, evaluation, and judgement of the human mind, and therefore is considered a mental process.
Step 2A Prong 2 and step 2B:
“The method as defined in claim 2, wherein the training learning progression and the validation learning progression are fed to the verification model” -- This limitation recites a training system that further comprises a verification model that will be fed into the training of the learning and validation progression. This limitation is considered to be mere limiting to a particular field of use/particular environment (type of model, progression taking place within technology), and therefore cannot be integrated to a practical application, not provide significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 3 is non-patent eligible.
Regarding claim 4,
Step 1: Claim 4 is dependent on claim 3, which is dependent on claim 2, which is deemed as a process. Claim 4 satisfies Step 1.
Step 2A Prong 1:
“The method as defined in claim 3, wherein a decision is made based on the quality assessment whether to continue or abort the ongoing training of the machine learning model” – This limitation is directed to a method where a decision is made based on the quality assessment on if training should continue on the model, which can be completed using evaluation and judgement of the human mind, and therefore is considered to be a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 4 is non-patent eligible.
Regarding claim 5,
Step 1: Claim 5 is dependent on claim 2, which is dependent on claim 1, which is deemed as a process. Claim 4 satisfies Step 1.
Step 2A Prong 1:
“which is trained to generate the quality assessment as output from the training learning progression and the validation learning progression as input data.” – This limitation is directed to training that will involve doing an evaluation on performance to be the output of the learning progression that is done on input data, which can be performed using evaluation as judgement, and therefore is seen as a mental process.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 2, wherein the verification model comprises a verification machine learning model” – This limitation recites a method where a verification model will further comprise a verification machine learning model. This is recited in a high level of generality and is seen as mere instructions to apply generically onto a computer (ML model) , therefore the imitation cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 5 is non-patent eligible.
Regarding claim 6,
Step 1: Claim 6 is dependent on claim 5, which is dependent on claim 2, which is deemed as a process. Claim 6 satisfies Step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 5, wherein the verification machine learning model is trained by an unsupervised learning process, in which a plurality of training learning progressions and associated validation learning progressions are used as verification machine learning model training data, or wherein the verification machine learning model is trained by a supervised learning process, in which a plurality of training learning progressions and associated validation learning progressions with a predetermined quality assessment are used as verification machine learning model training data.” – This limitation recites unsupervised and supervised learning at a high level of generality; therefore it amounts to mere instructions to apply a judicial exception (which is the quality assessment= evaluation and judgment) on a computer, and cannot be integrated to a practical application, not provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 6 is non-patent eligible.
Regarding claim 7,
Step 1: Claim 7 is dependent on claim 5, which is dependent on claim 2, which is deemed as a process. Claim 7 satisfies Step 1.
Step 2A Prong 1:
There are no elements to be evaluated under step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 5, wherein the training learning progression and the validation learning progression are respectively fed to the verification machine learning model as a sequence of quality measure values;” -- This limitation recites a training system that further comprises a verification model that will be fed into the training of the learning and validation progression. This limitation is considered to be mere limiting to a particular field of use/particular environment (type of model, progression taking place within technology), and therefore cannot be integrated to a practical application, not provide significantly more than the judicial exception (see MPEP 2106.05(h)).
“and wherein the verification machine learning model comprises a recurrent neural network.” – This limitation recites mere instructions that a verification ML model (computer) will comprise (instruct to apply) a recurrent NN, which is considered to a generic statement of application to a computer, and is not a practical application nor able to provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 7 is non-patent eligible.
Regarding claim 8,
Step 1: Claim 8 is dependent on claim 5, which is dependent on claim 2, which is deemed as a process. Claim 8 satisfies Step 1.
Step 2A Prong 1:
There are no elements to be evaluated under step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 5, wherein the training learning progression and the validation learning progression are respectively fed to the verification machine learning model as a sequence of quality measure values;” -- This limitation recites a training system that further comprises a verification model that will be fed into the training of the learning and validation progression. This limitation is considered to be mere limiting to a particular field of use/particular environment (type of model, progression taking place within technology), and therefore cannot be integrated to a practical application, not provide significantly more than the judicial exception (see MPEP 2106.05(h)).
“and wherein the verification machine learning model comprises a convolutional neural network.” – This limitation recites mere instructions that a verification ML model (computer) will comprise (instruct to apply) a convolutional NN, which is considered to a generic statement of application to a computer, and is not a practical application nor able to provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 8 is non-patent eligible.
Regarding claim 9,
Step 1: Claim 9 is dependent on claim 2, which is deemed as a process. Claim 9 satisfies Step 1.
Step 2A Prong 1:
“The method as defined in claim 2, wherein the verification model takes one or more of the following factors into account for the quality assessment: …number of epochs after which the training learning progression or validation learning progression saturates, and a value of the quality measures during saturation; - divergence of the training learning progression or validation learning progression; - initial fluctuations in the training learning progression and validation learning progression and subsequent monotonous training learning progression and validation learning progression” – This limitation is directed to a method for which multiple factors are listed for the quality assessment, which involves jumps in the learning and validation progression, epochs after progression saturation, and a value of quality that is made after saturation, which are all directed to the use of a mathematical operation and involve known mathematical concepts known in the art (calculating differences, divergence in progression). All these factors are ultimately done to judge the quality, which amounts to evaluation and judgment/mental.
“-jumps in the training learning progression or validation learning progression; - whether an optimum of the values of the trainable internal model parameters at which a quality measure is below a predetermined limit value is reached early.” – This limitation is directed to jumps in trained data and assessing if trainable, internal parameter values at quality measure are below a predetermined value, which can be performed with observation, evaluation, and judgement of the human mind or given with pen and paper, therefore, it is a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 9 is non-patent eligible.
Regarding claim 10,
Step 1: Claim 9 is dependent on claim 2, which is deemed as a process. Claim 9 satisfies Step 1.
Step 2A Prong 1:
“The method as defined in claim 2, wherein the quality assessment comprises a suggestion for a modification of training parameters during the ongoing learning process or for a new learning process to be initiated.” -- This limitation is directed to a suggestion of modifying parameters during a current or learning process. This limitation can be performed using observation, evaluation, and judgement of the human mind to suggest a modification to a learning process, therefore it is considered to be a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 10 is non-patent eligible.
Regarding claim 11,
Step 1: Claim 11 is dependent on claim 10, and claim 10 is dependent on claim 2, which is deemed as a process. Claim 11 satisfies Step 1.
Step 2A Prong 1:
“The method as defined in claim 10, wherein the modification of training parameters comprises at least one of: a modification of a learning rate and a modification of a set number of epochs” -- This limitation is directed to a modification of learning rate and a set number of epochs that are done. Modifying learning rates and number of epochs involves the use of a mathematical concept, and therefore is seen as a mathematical concept/calculation. (see [0076] of instant application).
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 11 is non-patent eligible.
Regarding claim 12,
Step 1: Claim 12 is dependent on claim 10, and claim 10 is dependent on claim 2, which is deemed as a process. Claim 12 satisfies Step 1.
Step 2A Prong 1:
There are no elements to be evaluated under step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 10, wherein, in the event that the quality assessment assumes a local optimum of the values of the trainable internal model parameters, the modification of training parameters comprises a one-time or repeated increase of a learning rate in order to escape the local optimum of the values of the trainable internal model parameters.” – This limitation recites a method where local optimum of parameter values is assumed and modifying parameter values consists of a repetitive or non-repetitive increasing of learning rate as an attempt to resolve local optimum. Merely changing data values is considered to be an insignificant, extra solution activity, and therefore cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, the limitation, especially with the performance of repetitive calculations, is a well-understood, routine and conventional activity in the field of machine learning, and therefore cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Thus, claim 12 is non-patent eligible.
Regarding claim 13,
Step 1: Claim 13 is dependent on claim 10, and claim 10 is dependent on claim 2, which is deemed as a process. Claim 13 satisfies Step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 10, wherein the modification of training parameters comprises a modification of the values of the trainable internal model parameters;” – This limitation recites a method of mere instructions (generically explained) to comprise (apply) a modification of parameter values, which cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
“wherein the verification machine learning model also receives to this end, in addition to the training learning progression and the validation learning progression, associated values of the trainable internal model parameters of the machine learning model as inputs.” -- This limitation recites receiving data (associated model parameters) as inputs for the verification machine learning model, which would involve mere data gathering, and is seen to be an insignificant, extra-solution activity, and cannot be integrated to a practical application. Furthermore, under step 2B, receiving data over memory/computer is considered to be a well-understood, routine and conventional activity (WURC), and cannot be significantly more than the judicial exception (see MPEP 2106.05(d)(II).
Thus, claim 13 is non-patent eligible.
Regarding claim 14,
Step 1: Claim 14 is dependent on claim 13, and claim 13 is dependent on claim 10, and claim 10 is dependent on claim 2, which is deemed as a process. Claim 14 satisfies Step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 13, wherein the verification machine learning model comprises a neural network trained by a supervised learning process, in which training data comprises a plurality of training learning progressions, validation learning progressions and associated values of the trainable internal model parameters, as well as modifications of the values of the trainable internal model parameters as target data or wherein the verification machine learning model comprises a neural network trained by a reinforcement learning method,” – This limitation recites a verification machine model that comprises mere instructions to apply a trained NN by a learning process, for which the trained data will further comprise a group of learning and validation learning progressions as well as parameter values, and modified model parameter values to be target data. The limitation is mere instructions to apply the concept of training and modifying different values and trained data onto a computer, which cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
“in which a reinforcement learning agent learns using a predefined training environment” – This limitation recites reinforcement learning using a predefined training environment, which is seen as mere limiting to a particular field of use (ML) and particular environment, which cannot be integrated to a practical application, nor provide significantly more than the judicial exception, (see MPEP 2106.05(h)).
“for an input comprising a training learning progression, a validation learning progression and associated values of the trainable internal model parameters, how to modify the model parameter values of the trainable internal model parameters in order to optimize the quality assessment.” -- This limitation recites inputs to learning progression and parameter values and how to modify the values, which is directed to mere data gathering, and is considered an insignificant, extra-solution activity that cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under step 2B, the act of receiving data to be manipulated is a well-understood, routine and conventional activity (WURC), and cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II).
Thus, claim 14 is non-patent eligible.
Regarding claim 15,
Step 1: Claim 15 is dependent on claim 2, and claim 2 is deemed a process. Claim 15 satisfies step 1.
There are no elements to be evaluated under step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 2, wherein the training learning progression and the validation learning progression are input into a prediction machine learning model that is trained to predict future progressions of an input training learning progression and validation learning progression from the input training learning progression and validation learning progression and to add the predicted future progressions onto the input training learning progression and validation learning progression, wherein the training learning progression and validation learning progression supplemented by the prediction are output to a user or to the verification model.” – This limitation recites a method of inputting progressions to a prediction ML model that will be trained to output predict future progressions with a prediction to be outputted to either the user or the model itself, and then add them to the trained learning and validation progressions. This limitation is directed to mere data gathering and manipulation of data, for which is considered to be an insignificant, extra-solution activity, and cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under step 2B, the act of transmitting data over a network (ML model) is a well-understood, routine, and conventional activity (WURC) in the field of ML, and therefore does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II).
Thus, claim 15 is non-patent eligible.
Regarding claim 16,
Step 1: Claim 16 is dependent on claim 2, and claim 2 is deemed a process. Claim 16 satisfies step 1.
Step 2A Prong 1:
“The method as defined in claim 2, wherein the verification machine learning model is configured to conduct an anomaly detection in order to determine deviations from typical training progression” – This limitation is directed to a method of conduction anomaly detections to determine deviations of the training progression, which is considered to be a mental process that can be performed in the human mind using observation, evaluation, and judgement.
Step 2A Prong 2 and Step 2B:
“wherein the verification machine learning model for the anomaly detection is designed as an autoencoder trained with training learning progressions and validation learning progressions that do not contain any anomalies.” -- This limitation recites a verification ML model will be instructed to apply trained progressions that don’t have anomaly detections, which is recited in a high level of generality, and thus cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 16 is non-patent eligible.
Regarding claim 17,
Step 1: Claim 17 is dependent on claim 2, and claim 2 is deemed a process. Claim 17 satisfies step 1.
Step 2A Prong 1:
“define assessment criteria for the generation of the quality assessment depending on contextual data, wherein the contextual data relate to the machine learning model or the training of the machine learning model, wherein the contextual data comprise information regarding one or more of the following aspects: - type of a learning method, wherein a distinction is made at least between supervised training, unsupervised training, reinforcement learning and a use of adversarial networks; - type of a task of the machine learning model, wherein a distinction is made at least between a classification, segmentation, detection or regression; - architecture of the machine learning model; - values of hyperparameters of the machine learning model; - a type of the training/validation data.” – This limitation is directed to defining assessment criteria for the ML model based on data (contextual data), which can all be performed using evaluation and observation with the human mind, and therefore is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 2, wherein the verification model is designed to… wherein the contextual data comprise information regarding one or more of the following aspects: - type of a learning method, wherein a distinction is made at least between supervised training, unsupervised training, reinforcement learning and a use of adversarial networks; - type of a task of the machine learning model, wherein a distinction is made at least between a classification, segmentation, detection or regression; - architecture of the machine learning model; - values of hyperparameters of the machine learning model; - a type of the training/validation data.” – This limitation recites a verification model and its mere instructions to be applied onto the computer at a high level of generality, and cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 17 is non-patent eligible.
Regarding claim 18,
Step 1: Claim 18 is analogous to method claim 2, and thus satisfy step 1 as claim 2 did under the category of process.
There are no elements to be evaluated under step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“A non-transitory computer-readable medium storing a computer program with commands that, when executed by a computer, cause the execution of the method defined in claim 2.” – This limitation recites that the non-transitory CRM that stores a computer with commands that will cause the execution of the method defined in claim 2. The limitation does not amount to no more than mere instructions to apply onto a computer, and thus it cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 18 is non-patent eligible.
Regarding claim 19,
Step 1: The claim is directed to a method, which falls under the category of a process. The claim satisfies step 1.
Step 2A Prong 1:
“calculating, during the learning process, at least one quality measure for respectively current values of the trainable internal model parameters” – The limitation is directed to calculating a quality measure for current model parameter values. Under broadest reasonable interpretation, the limitation recites a process that can be completed in the human mind using evaluation, observation, and judgment (with the aid of pen and paper) to complete, and thus it is directed to a mental process.
“forming at least one training learning progression from the quality measures of different epochs” -- The limitation is directed to forming training progression from quality measures of different epochs (period of time). The limitation is directed to a process that can be performed in the human mind using evaluation, observation, and judgement, and thus the limitation is directed to a mental process.
“wherein the verification machine learning model is trained by a supervised learning process, in which a plurality of training learning progressions with predetermined quality assessments are used as verification machine learning model training data.” – The limitation is directed to training a machine learning model through a plurality of training learning progressions with predetermined quality assessments as training data. Training a model using predetermined quality assessment is a process that can be done using evaluation, and judgment, therefore it is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“A method for monitoring a learning process of a machine learning model including a generative adversarial network, wherein the machine learning model includes trainable internal model parameters for processing input training data to calculate a model output,” – The limitation recites a method of monitoring learning of a machine learning model that will further include a GAN, and includes model parameters for input processing training data to calculate the model output. The limitation does not amount to no more than mere further limits to a field of use/environment, and it cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(h)).
“the method comprising launching a learning process using training data to adjust values of the trainable internal model parameters of the machine learning model;”-- This limitation recites a method to launch a learning process that involves adjusting trainable parameter values based/depending on gathered and manipulated data. Mere data manipulation is considered to be an insignificant, extra-solution activity, and cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under step 2B, the act of storing/and retrieving information in memory (training system), is a well-understood, routine, and conventional activity (WURC), and cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)(iv)).
“the quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from an input to the machine learning model;” -- The limitation is directed to the quality measure indicated a quality for a calculated result with the current model parameters, wherein it will be further limited to not be utilized in the learning process to adjust the model parameter values, but instead using the validation data quality measure. The limitation amounts to no more than further limiting to a field of use/environment, and does not integrate to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(h)).
“repeating calculating the quality measure in different epochs of the learning process;” -- The limitation is directed to repetitive calculation of the training and validation data quality measure in different epochs. Repetitive calculation is an insignificant, extra-solution activity that cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, the act of repetitive calculation is also a well-understood, routine, and conventional activity (WURC), and cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“inputting the at least one training learning progression into a verification model during the learning process” -- The limitation is directed to inputting training progression into a verification model during the learning process of the model. Inputting/outputting gathered data is an insignificant, extra-solution activity, and it cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, the limitation is also considered to be a well-understood, routine, and conventional activity, that cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“wherein the verification machine learning model is trained by an unsupervised learning process, in which a plurality of training learning progressions are used as verification machine learning model training data.” – The limitation recites to training a model by unsupervised learning process, which entails using a plurality of training learning progressions as the training data. The limitation is directed to mere instructions to apply onto a computer, and it cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
“such that the training learning progressions are input into the verification machine learning model and deviations of the verification machine learning model's outputs to the predetermined quality assessments are used to update parameters of the verification machine learning model.” -- The limitation recites that the learning progressions are input to a model and that deviations of the outputs to the quality assessments are used to update parameters of the model. The limitation is directed to an insignificant, extra-solution activity that cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, the act of updating parameters and input/output relations over a model is a well-understood, routine, and conventional activity (WURC) that cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Thus, claim 19 is non-patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this
Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not
identically disclosed as set forth in section 102, if the differences between the claimed invention and the
prior art are such that the claimed invention as a whole would have been obvious before the effective filing
date of the claimed invention to a person having ordinary skill in the art to which the claimed invention
pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are
summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1 is rejected under 35 U.S.C. 103 as being unpatentable over Moore et. al, US11927738B2 (referred herein as Moore) in view of NPL reference “Hyperparameter Optimization”, by Feurer et al. (referred herein as Feurer) in view of US-20220051049-A1 by Wang et. al. (referred herein as Wang) further in view of US 8275723B2 by Guyon et. al. (referred herein as Guyon).
Regarding claim 1, Moore teaches:
A microscopy system comprising: at least one microscope, which comprises at least one camera for capturing a microscope image and a computing device, wherein the computing device comprises an imaging processing program for processing the microscope image using a machine learning model, ([Moore, col 12, lines 5-20, col 26, lines 15-17], “initiating movement of the camera and lighting apparatus for sample illumination and image capture… a plurality of images may be captured…the captured image from the main camera includes two copies of the sample…assessing the sample using feature extraction and/or one or more machine and/or deep learning models… the RANSAC method is an iterative method to estimate parameters of a mathematical model”, wherein the examiner interprets “the camera…image capture” to be the same as at least one camera capturing a microscope image, and “assessing the sample using…machine and/or deep learning models” to be the same as a computing device comprising an imaging processing program for processing the microscope image using a machine learning model because they are both directed to capturing microscope images with an imaging device and processing those images using machine learning–based analysis executed by a computing device.)
Moore does not teach wherein the machine learning model includes a set of trainable internal model parameters for processing input training data to calculate a model output, the trainable internal model parameters being distinct from hyperparameters that define training settings or model architecture; and a training system for carrying out a learning process of the machine learning model, wherein the training system is configured for: adjusting values of the trainable internal model parameters values of the machine learning model successively in the learning process depending on training data processed with the trainable internal model parameters; calculating a training data quality measure for current values of the trainable internal model parameters values during the learning process, the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data; and calculating a validation data quality measure for the current values of the trainable internal model parameters during the learning process, the validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from validation data during the learning process, wherein the validation data is not used in the learning process to adjust the values of the trainable internal model parameters and instead the validation data is only used for calculating the validation data quality measure; repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process which uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs; and forming a training learning progression from successively determined training data quality measures and forming a validation learning progression from successively determined validation data quality measures, wherein the training system comprises a verification model, which is fed with the training learning progression and the validation learning progression during the learning process; and wherein the verification model is configured to generate a quality assessment of the learning process of the machine learning model depending on the training learning progression and the validation learning progression.
Feurer teaches wherein the machine learning model includes a set of trainable internal model parameters for processing input training data to calculate a model output,([Feurer, Chapter 4, sec 4.2, page 83] “A learning algorithm A maps a set of training data points di to such a function, which is often expressed via a vector of model parameters”, wherein the examiner interprets “A learning algorithm A maps a set of training data points” and “expressed via a vector of model parameters” to be the same as a “set of trainable internal model parameters for processing input training data to calculate a model output” because they are both directed to internal parameters of a machine learning model that are derived from processing training data to produce an output function)
the trainable internal model parameters being distinct from hyperparameters that define training settings or model architecture; ([Feurer, Chapt 4, sec 4.2, page 83] “Most learning algorithms A further expose hyperparameters, which change the way the learning algorithm itself works. For example, hyperparameters are used to describe a description-length penalty, the number of neurons in a hidden layer, the number of data points that a leaf in a decision tree must contain to be eligible for splitting”, wherein the examiner interprets “hyperparameters, which change the way the learning algorithm itself works” and “the number of neurons in a hidden layer” to be the same as “hyperparameters that define training settings or model architecture” because they are both directed to external configuration values that control how the learning algorithm operates and determine the structural design of the model, and wherein the examiner interprets “model parameters” as described in the preceding sentence to be the same as trainable internal model parameters because they are both directed to values learned from data that are distinct from the externally-set hyperparameters.)
adjusting values of the trainable internal model parameters of the machine learning model ([Feurer, Chapter 2, Sec 2.4.2, page 49] “During training, they use their own weights as additional input data and observe their own errors to learn how to modify these weights in response to the new task at hand. The updating of the weights is defined in a parametric form that is differentiable end-to-end and can jointly optimize both the network and training algorithm using gradient descent”, wherein the examiner interprets “During training, they use their own weights” and “learn how to modify these weights” and “The updating of the weights” and “using gradient descent” to be the same as adjusting values of the trainable internal model parameters values of the machine learning model successively in the learning process because they are both directed to modifying learned model weights iteratively during the training procedure through optimization algorithms such as gradient descent)
calculating a training data quality measure for current values of the trainable internal model parameters values during the learning process, ([Feurer, Chapter 1, Section 1.4.1, p. 14] “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration”, wherein the examiner interprets “the performance of an iterative algorithm measured for each iteration” to be the same as “calculating a training data quality measure for current values of the trainable internal model parameters values during the learning process” because they are both directed to measuring a performance metric at successive iterations during training where the model parameters have different values at each iteration.)
and calculating a validation data quality measure for the current values of the trainable internal model parameters during the learning process, ([Feurer, Chapter 1, Section 1.4.1, p. 14] “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration”, wherein the examiner interprets “the performance of an iterative algorithm measured for each iteration” to be the same as “calculating a validation data quality measure for the current values of the trainable internal model parameters values during the learning process” because they are both directed to measuring model performance at successive iterations during training when the trainable internal model parameters have different values at each iteration)
wherein the validation data is not used in the learning process to adjust the values of the trainable internal model parameters and instead the validation data is only used for calculating the validation data quality measure; ([Feurer, Chapter 4, sec 4.2.1, page 83] “Generalization performance is estimated by splitting D into disjoint training and validation sets Dtrain and Dvalid, learning functions fi by applying A to Dtrain, and evaluating the predictive performance of these functions on Dvalid”, wherein the examiner interprets “splitting D into disjoint training and validation sets” and “learning functions fi by applying A to Dtrain” to be the same as the validation data is not used in the learning process to adjust the values of the trainable internal model parameters values because they are both directed to using only the training set for learning model parameters while keeping validation data separate, and wherein the examiner interprets “evaluating the predictive performance of these functions on Dvalid” to be the same as instead the validation data is only used for calculating the validation data quality measure because they are both directed to using validation data solely for performance evaluation rather than for learning or adjusting model parameters)
repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process ([Feurer, Chapter 1, sec 1.4.1, p. 14] “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration (or every i-th iteration if the calculation of the performance is expensive.”, wherein the examiner interprets “measured for each iteration” to be the same as “repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process” because they are both directed to repeatedly evaluating performance/quality values at successive stages of training.)
which uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs; ([Feurer, Chapter 2, sec 2.4.2, p. 49], “During training, they use their own weights as additional input data and observe their own errors to learn how to modify these weights in response to the new task at hand. The updating of the weights is defined in a parametric form that is differentiable end-to-end and can jointly optimize both the network and training algorithm using gradient descent.”, wherein the examiner interprets “their own weights” and “learn how to modify these weights” and “The updating of the weights” to be the same as “the same set of trainable internal model parameters” and “different values of the same set of trainable internal model parameters across the epochs” because they are both directed to repeatedly updating the same internal model weights during training such that the weights take on different values at successive stages of the learning process.)
and forming a training learning progression from successively determined training data quality measures and forming a validation learning progression from successively determined validation data quality measures; ([Feurer, Chapter 1, sec 1.4.1, p. 14] “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration (or every i-th iteration if the calculation of the performance is expensive.”, wherein the examiner interprets “learning curves” and “measured for each iteration” to be the same as “forming a training learning progression from successively determined training data quality measures and forming a validation learning progression from successively determined validation data quality measures” because they are both directed to building a progression/curve from successive performance values measured during training.)
wherein the training system comprises a verification model, which is fed with the training learning progression and the validation learning progression during the learning process; and ([Feurer, Chapter 1, sec 1.4, p. 14] “First, we review methods which model an algorithm’s learning curve during training and can stop the training procedure if adding further resources is predicted to not help.”, wherein the examiner interprets “methods which model an algorithm’s learning curve during training” to be the same as “a verification model, which is fed with the training learning progression and the validation learning progression during the learning process” because they are both directed to a model/method operating during training that uses observed learning curve information as input.)
wherein the verification model is configured to generate a quality assessment of the learning process of the machine learning model depending on the training learning progression and the validation learning progression. ([Feurer, Chapter 1, sec 1.4.1, p. 14] “Learning curve extrapolation is used in the context of predictive termination [26], where a learning curve model is used to extrapolate a partially observed learning curve for a configuration, and the training process is stopped if the configuration is predicted to not reach the performance of the best model trained so far in the optimization process.”, wherein the examiner interprets “training process is stopped if the configuration is predicted to not reach the performance of the best model trained so far” to be the same as “generate a quality assessment of the learning process of the machine learning model” because they are both directed to evaluating training quality using learning curve behavior and making a determination based on that evaluation, and wherein the examiner interprets “learning curve model … extrapolate a partially observed learning curve” to be the same as “depending on the training learning progression and the validation learning progression” because they are both directed to basing the assessment on the observed progression of performance during training.)
Feurer does not teach and a training system for carrying out a learning process of the machine learning model, … wherein the training system is configured for: [adjusting values of the trainable internal model parameters values] of the machine learning model successively in the learning process depending on training data processed with the trainable internal model parameters; … the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data; … the validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from validation data during the learning process
Wang teaches, and a training system for carrying out a learning process of the machine learning model, [Wang, [0029]], “The server computer 102, via the HOM 122 at block 220, iteratively runs a training portion of the preprocessed GTD 106 through each of the pipelines 402”, and [Wang, p. 3, [0019] “The server computer 102 is in communication with a source of Ground Truth Data (GTD) 106 useful for training and validating the models”, wherein the examiner interprets “server computer 102” with its various modules (HOM 122, APCM 124, etc.) to be the same as a “training system for carrying out a learning process of the machine learning model.” wherein the examiner interprets “server computer 102” operating via the HOM 122 to iteratively run training data through pipelines to be the same as “a training system for carrying out a learning process of the machine learning model” because both are directed to a computer system that executes the training process of machine learning models using training data.)
wherein the training system is configured for: [adjusting values of the trainable internal model parameters of the machine learning model] successively in the learning process depending on training data processed with the trainable internal model parameters: ([Wang, [0029]], “The server computer 102, via the HOM 122 at block 220, iteratively runs a training portion of the preprocessed GTD 106 through each of the pipelines 402”, wherein the examiner interprets “iteratively runs a training portion of the preprocessed GTD 106 through each of the pipelines 402” to be the same as “successively in the learning process depending on training data processed” because they are both directed to repeatedly processing training data through a machine learning model during the training process.)
Wang does not teach, the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data; … the validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from validation data during the learning process.
Guyon teaches the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data; ([Guyon, col. 32, lines 10-23] “Four metrics of classifier quality were used. Each value is computed both on the training set with the leave-one-out method and on the test set”, wherein the examiner interprets “Four metrics of classifier quality” and “computed both on the training set” to be the same as the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters values from the training data because they are both directed to quality metrics computed specifically on training data that evaluate how well the model performs with its current learned parameter values)
the validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from validation data during the learning process, ([Guyon, col. 32, lines 10-23], “Four metrics of classifier quality were used. Each value is computed both on the training set with the leave-one-out method and on the test set”, wherein the examiner interprets “Four metrics of classifier quality” and “computed on the test set” to be the same as the “validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters values from validation data” because they are both directed to quality metrics computed specifically on held-out data separate from training data that evaluate how well the model performs with its current learned parameter values on data not used to adjust those parameters)
Moore, Feurer, Wang, Guyon, and the instant application are analogous art because they are all directed to machine learning systems that train models using quality metrics computed on both training and validation data to assess learning progression and determine training effectiveness.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the microscopy system of claim 1 disclosed by Moore to include the learning process disclosed by Feurer. One would be motivated to do so to effectively monitor and optimize the training of machine learning models used in microscopy image processing, as suggested by Feurer ([Feurer, Chapter 1, sec 1.4, p. 14], “First, we review methods which model an algorithm's learning curve during training and can stop the training procedure if adding further resources is predicted to not help.”). It would have been further obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the microscopy system of claim 1 disclosed by Moore to include the iterative training pipelines disclosed by Wang. One would be motivated to do so to efficiently execute parallel training of multiple machine learning model configurations, as suggested by Wang ([Wang, [0029]], “The server computer 102, via the HOM 122 at block 220, iteratively runs a training portion of the preprocessed GTD 106 through each of the pipelines 402”). It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the microscopy system of claim 1 disclosed by Moore to include the “quality metrics computed on both training and test sets” disclosed by Guyon. One would be motivated to do so to accurately evaluate model performance on both datasets used for training and separate validation data, as suggested by Guyon ([Guyon, col. 32, lines 10-23], “Four metrics of classifier quality were used. Each value is computed both on the training set with the leave-one-out method and on the test set”).
Claim(s) 2-7, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feurer in view of Wang.
Regarding claim 2, Feurer teaches A method for monitoring a learning process of a machine learning model, wherein the machine learning model includes a set of trainable internal model parameters for processing input training data to calculate a model output, the trainable internal model parameters being distinct from hyperparameters that define training settings or model architecture, the method comprising: ([Feurer, Chapter 4, sec 4.2, page 83] “A learning algorithm A maps a set of training data points di to such a function, which is often expressed via a vector of model parameters … Most learning algorithms A further expose hyperparameters, which change the way the learning algorithm itself works. For example, hyperparameters are used to describe a description-length penalty, the number of neurons in a hidden layer, the number of data points that a leaf in a decision tree must contain to be eligible for splitting”, wherein the examiner interprets “A learning algorithm A maps a set of training data points” and “expressed via a vector of model parameters” to be the same as “a set of trainable internal model parameters for processing input training data to calculate a model output” because they are both directed to internal parameters of a machine learning model that are derived from processing training data to produce an output function. The examiner further interprets “hyperparameters, which change the way the learning algorithm itself works” and “the number of neurons in a hidden layer” to be the same as “hyperparameters that define training settings or model architecture” because they are both directed to external configuration values that control how the learning algorithm operates and determine the structural design of the model, and wherein the examiner interprets “model parameters” as described in the preceding sentence to be the same as trainable internal model parameters because they are both directed to values learned from data that are distinct from the externally-set hyperparameters.)
adjusting values of the trainable internal model parameters ([Feurer, Chapter 2, Sec 2.4.2, page 49], “During training, they use their own weights as additional input data and observe their own errors to learn how to modify these weights in response to the new task at hand. The updating of the weights is defined in a parametric form that is differentiable end-to-end and can jointly optimize both the network and training algorithm using gradient descent”, wherein the examiner interprets “During training, they use their own weights” and “learn how to modify these weights” and “The updating of the weights” and “using gradient descent” to be the same as adjusting values of the trainable internal model parameters values of the machine learning model successively in the learning process because they are both directed to modifying learned model weights iteratively during the training procedure through optimization algorithms such as gradient descent)
calculating a training data quality measure for current values of the trainable internal model parameters during the learning process, the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data; and ([Feurer, Chapter 1, Section 1.4.1, p. 14], “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration”, wherein the examiner interprets “the performance of an iterative algorithm measured for each iteration” to be the same as “calculating a training data quality measure for current values of the trainable internal model parameters during the learning process” because they are both directed to measuring a performance metric at successive iterations during training where the model parameters have different values at each iteration, combined with [Guyon, col. 32, lines 10-23] “Four metrics of classifier quality were used. Each value is computed both on the training set with the leave-one-out method and on the test set”, wherein the examiner interprets “Four metrics of classifier quality” and “computed both on the training set” to be the same as “the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data” because they are both directed to quality metrics computed specifically on training data that evaluate how well the model performs with its current learned parameter values.)
calculating a validation data quality measure for the current values of the trainable internal model parameters during the learning process, ([Feurer, Chapter 1, Section 1.4.1, p. 14], “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration”, wherein the examiner interprets “the performance of an iterative algorithm measured for each iteration” to be the same as “calculating a validation data quality measure for the current values of the trainable internal model parameters during the learning process” because they are both directed to measuring model performance at successive iterations during training when the trainable internal model parameters have different values at each iteration.)
the validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from validation data during the learning process, ([Guyon, col. 32, lines 10-23] “Four metrics of classifier quality were used. Each value is computed both on the training set with the leave-one-out method and on the test set”, wherein the examiner interprets “Four metrics of classifier quality” and “computed on the test set” to be the same as the “validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from validation data” because they are both directed to quality metrics computed specifically on held-out data separate from training data that evaluate how well the model performs with its current learned parameter values on data not used to adjust those parameters.)
wherein the validation data is not used in the learning process to adjust the values of the trainable internal model parameters values and instead the validation data is only used for calculating the validation data quality measure; ([Feurer, Chapter 4, page 83] “Generalization performance is estimated by splitting D into disjoint training and validation sets Dtrain and Dvalid, learning functions fi by applying A to Dtrain, and evaluating the predictive performance of these functions on Dvalid”, wherein the examiner interprets “splitting D into disjoint training and validation sets” and “learning functions fi by applying A to Dtrain” to be the same as “the validation data is not used in the learning process to adjust the values of the trainable internal model parameters” because they are both directed to using only the training set for learning model parameters while keeping validation data separate, and wherein the examiner further interprets “evaluating the predictive performance of these functions on Dvalid” to be the same as “instead the validation data is only used for calculating the validation data quality measure” because they are both directed to using validation data solely for performance evaluation rather than for learning or adjusting model parameters.)
repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process ([Feurer, Chapter 1, Section 1.4.1, p. 14], “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration (or every i-th iteration if the calculation of the performance is expensive.”, wherein the examiner interprets “measured for each iteration” to be the same as “repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process” because they are both directed to repeatedly evaluating performance/quality values at successive stages of training.)
which uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs; ([Feurer, Chapter 2, PDF p. 17] “Model training typically starts with parameters being initialized to some values. As training/learning progresses the initial values are updated using an optimization algorithm (e.g. gradient descent). The learning algorithm is continuously updating the parameter values as learning progresses.”, wherein the examiner interprets “parameters being initialized to some values” and “updated using an optimization algorithm” and “continuously updating the parameter values as learning progresses” to be the same as “uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs” because they are both directed to the same internal model parameters being repeatedly updated to different values over successive stages of the learning process (epochs), while performance/quality measures are evaluated during those stages.)
forming a training learning progression from successively determined training data quality measures, and ([Feurer, Chapter 1, Section 1.4.1, p. 14], “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration (or every i-th iteration if the calculation of the performance is expensive)”, wherein the examiner interprets “learning curves” and “the performance of an iterative algorithm measured for each iteration” to be the same as “forming a training learning progression from successively determined training data quality measures” because they are both directed to building a progression or curve from performance values that are measured successively at each iteration during the training process.)
forming a validation learning progression from successively determined validation data quality measures, ([Feurer, Chapter 1, Section 1.4.1, p. 14], “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration (or every i-th iteration if the calculation of the performance is expensive)”, wherein the examiner interprets “learning curves” and “the performance of an iterative algorithm measured for each iteration” to be the same as “forming a validation learning progression from successively determined validation data quality measures” because they are both directed to building a progression/curve from successive performance values measured at different iterations during training, and wherein the examiner interprets this in combination with [Feurer, Chapter 4, page 83] “Generalization performance is estimated by splitting D into disjoint training and validation sets Dtrain and Dvalid, learning functions fi by applying A to Dtrain, and evaluating the predictive performance of these functions on Dvalid” to confirm that these learning curves include validation performance measurements, as both are directed to tracking performance metrics evaluated on validation data across successive stages of the learning process.)
wherein the training learning progression and the validation learning progression are fed to a verification model during the learning process; ([Feurer, Chapter 1, sec 1.4, p. 14], “First, we review methods which model an algorithm's learning curve during training and can stop the training procedure if adding further resources is predicted to not help”, wherein the examiner interprets “methods which model an algorithm's learning curve during training” to be the same as “a verification model” that is “fed” with “the training learning progression and the validation learning progression” “during the learning process” because they are both directed to a model/method operating during training that uses observed learning curve information as input.)
and wherein the verification model generates a quality assessment of the learning process of the machine learning model depending on the training learning progression and the validation learning progression. ([Feurer, Chapter 1, sec 1.4.1, p. 14], “Learning curve extrapolation is used in the context of predictive termination [26], where a learning curve model is used to extrapolate a partially observed learning curve for a configuration, and the training process is stopped if the configuration is predicted to not reach the performance of the best model trained so far in the optimization process”, wherein the examiner interprets “training process is stopped if the configuration is predicted to not reach the performance of the best model trained so far” to be the same as “generates a quality assessment of the learning process of the machine learning model” because they are both directed to evaluating training quality using learning curve behavior and making a determination based on that evaluation, and wherein the examiner interprets “learning curve model … extrapolate a partially observed learning curve” to be the same as “depending on the training learning progression and the validation learning progression” because they are both directed to basing the assessment on the observed progression of performance during training.)
Feurer does not teach launching a learning process including … successively during the learning process depending on training data processed with the trainable internal model parameters,
Wang teaches launching a learning process including [adjusting values of the trainable internal model parameters] successively during the learning process depending on training data processed with the trainable internal model parameters, ([Wang, 0029], “The server computer 102, via the HOM 122 at block 220, iteratively runs a training portion of the preprocessed GTD 106 through each of the pipelines 402 with the hyperparameter sets generated by the PGM 116. The HOM 122 assess performance of each pipeline 402 iteratively, comparing performance for each of the associated hyperparameter sets.”, wherein the examiner interprets “iteratively runs a training portion… assess performance… iteratively” to be the same as “adjusting values… successively during the learning process,” because both are directed to repeated training iterations where performance is assessed and parameter selections are updated based on training data results.)
Feurer, Wang, and the instant application are analogous art because they are all directed to methods for monitoring a learning process of a machine learning model by evaluating training and validation performance iteratively.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify learning process monitoring disclosed by Feurer to include the iterative training process disclosed by Wang. One would be motivated to do so to develop a more efficient training execution framework that repeatedly processes training data and evaluates performance during learning, as suggested by Wang ([Wang, [0029]], “iteratively runs a training portion of the preprocessed GTD 106 through each of the pipelines … assess performance … iteratively”).
Regarding claim 3, Feurer and Wang teaches The method as defined in claim 2, (see rejection of claim 2).
Wang further teaches wherein the training learning progression and the validation learning progression are fed to the verification model during an ongoing training of the machine learning model before a predetermined stopping criterion of the training is reached. ([Wang, 0026] “It is preferred that pipeline generation occur iteratively, in conjunction with decision block 214, with the server computer 102 iteratively deciding after generating each pipeline 402, whether more pipelines are needed (e.g. , pipeline target quantity has been met or a user has indicated that a current set of pipelines is deemed sufficient).” AND [Wang, 0005] “…ranking pipeline performance in accordance therewith ; select a candidate pipe line in accordance, at least in part , with said pipeline performance ranking”, wherein the examiner interprets “iteratively deciding after generating each pipeline” and “ranking pipeline performance in accordance therewith” to be the same as “training learning progression and validation learning progression being fed to the verification model during an ongoing training,” as both are directed to evaluating intermediate results of a process and determining subsequent actions during the iterative development phase prior to a stopping criterion being met (like the target quantity, performance, or user satisfaction.)
Regarding claim 4, Feurer and Wang teaches The method as defined in claim 3, (see rejection of claim 3).
Wang further teaches wherein a decision is made based on the quality assessment whether to continue or abort the ongoing training of the machine learning model. ([Wang, 0020] “…includes a Pipeline Validation User Interface ( PVUI ) 128 that allows a user to examine pipeline execution results to correct, remove selected pipelines , and otherwise give input regarding pipeline performance to increase result interpretability and user confidence.”, wherein the examiner interprets “examine pipeline execution results to correct, remove selected pipelines, and otherwise give input regarding pipeline performance” to be the same as “making a decision based on the quality assessment whether to continue or abort the ongoing training,” as both are directed to assessing the quality of intermediate results and deciding the continuation or termination of the process based on those results.)
Regarding claim 5, Feurer and Wang teaches The method as defined in claim 2, (see rejection of claim 2).
Wang further teaches wherein the verification model comprises a verification machine learning model, which is trained to generate the quality assessment as output from the training learning progression and the validation learning progression as input data. ([Wang, 0031-0032] “The server computer 102 presents to a user for feedback, via the Pipeline Validation User Interface (PVUI) 128...results of applying the pipelines 402...Pipeline performance details are included to provide a high degree of interpretability” AND [Wang, Abstract], “A computer automatically selects a machine learning model pipeline using a meta-learning machine learning model.”, wherein the examiner interprets “results of applying the pipelines” and “pipeline performance details” to be the same as “the training learning progression and validation learning progression” and “meta-learning machine learning model” to be the same as “verification model,” as both are directed to evaluating and interpreting model performance during training to guide adjustments before completion of the process. Meta-learning refers to “learning to learn” and involves using ML to optimize or guide the training of another ML model.)
Regarding claim 6, Feurer and Wang teaches The method as defined in claim 5, (see rejection of claim 5).
Wang further teaches wherein the verification machine learning model is trained by an unsupervised learning process, in which a plurality of training learning progressions and associated validation learning progressions are used as verification machine learning model training data, or wherein the verification machine learning model is trained by a supervised learning process, in which a plurality of training learning progressions and associated validation learning progressions with a predetermined quality assessment are used as verification machine learning model training data. ([Wang, 0004] “The computer applies each of the pipelines with the associated preferred set of hyperparameters to score the favored data features of an appropriately preprocessed set of ground truth data and ranks the pipeline performance accordingly . The computer selects a candidate pipeline in accordance , at least in part , with the pipeline performance ranking .”, wherein the examiner interprets “applying each pipeline with preferred hyperparameters to score favored data features” and “ranking the pipeline performance” to be the same as “training a supervised learning verification model using training learning progressions and validation learning progressions with a predetermined quality assessment,” as both are directed to utilizing predefined criteria (ground truth data and hyperparameters) for guiding model selection and performance assessment during training.)
Regarding claim 7, Feurer and Wang teaches The method as defined in claim 5, (see rejection of claim 5).
Wang further teaches wherein the training learning progression and the validation learning progression are respectively fed to the verification machine learning model as a sequence of quality measure values; and wherein the verification machine learning model comprises a recurrent neural network. ([Wang, Abstract] “A computer automatically selects a machine learning model pipeline using a meta-learning machine learning model ... The computer generates hyper parameter sets for the pipelines. The computer applies preprocessing routines to ground truth data to generate a group of preprocessed sets of said ground truth data and ranks hyperparameter set performance for each pipeline to establish a preferred set of hyperparameters for each of pipeline . The computer selects favored data features and applies each of the pipelines , with associated sets of preferred hyperparameters , to score the favored data features of the preprocessed ground truth data. The computer ranks pipeline performance and selects a candidate pipeline according to the ranking .”, wherein the examiner interprets “ranking hyperparameter set performance for each pipeline” and “scoring favored data features of preprocessed ground truth data” to be the same as “feeding training learning progression and validation learning progression as a sequence of quality measure values to the verification machine learning model,” as both are directed to the iterative process of assessing performance through sequential evaluation. The machine learning model used for ranking and evaluation in Wang et al. could include a recurrent neural network (RNN), as such models are well-suited for processing sequential data and quality measure values in iterative processes.)
Regarding claim 18, the majority of the claim is analogous to claim 2, so it will inherit the rejection set forth for claim 2 above. Below, From Wang, is where it maps to A non-transitory computer-readable medium storing a computer program commands. ([0035] “The method 100, for example, may be embodied in a program 1060, including program instructions, embodied on a computer readable storage device, or computer readable storage medium, for example, generally referred to as memory 1030 and more specifically, computer readable storage medium 1050.”, where in the examiner interprets “program instructions, embodied on a computer readable storage device, or computer readable storage medium, for example, generally referred to as memory 1030 and more specifically, computer readable storage medium” to be the same as computer-readable medium, not transitory as clarified in [0045], that has stored computer programs and commands.). So the combination of Feurer and Wang embody claim 2.
Claim(s) 8-10 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Feurer in view of Wang further in view of Moore.
Regarding claim 8, Feurer and Wang teaches the method as defined in claim 5 (see rejection of claim 5).Feurer and Wang do not teach wherein the training learning progression and the validation learning progression are fed to the verification machine learning model as graphs in the form of image data; and wherein the verification machine learning model comprises a convolutional neural network;.
Moore teaches wherein the training learning progression and the validation learning progression are fed to the verification machine learning model as graphs in the form of image data; and wherein the verification machine learning model comprises a convolutional neural network; ([Moore, col 15, lines 3-10] “In other embodiments, the message may include an image of the sample for user verification of the computer-assisted evaluation. In some embodiments, one or more images of the sample may be digitally stained using pre-trained deep learning models and displayed to the viewer.” AND [Moore, col 26, lines 60-64] “In one embodiment of image stitching, a deep convolutional neural network (CNN) may be used to generate homographic estimates of images”), wherein the examiner interprets “pre-trained deep learning models for digitally staining and displaying images for user verification” and “a deep convolutional neural network (CNN) used for homographic estimates of images” to be the same as “a verification machine learning model” and the same as feeding “to the verification machine learning model … image data,” as both are directed to using machine learning processes (via CNNs) to analyze and output transformed image data for evaluation and further processing.)
Feurer, Wang, Moore, and the instant application are analogous art because they are all directed to monitoring and evaluating machine learning processes via verification models to improve system performance.It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the verification machine learning model disclosed by Feurer and Wang to include the “use of a deep convolutional neural network (CNN) for generating homographic estimates of images” taught by Moore. One would be motivated to do so to efficiently enhance the verification and validation process using images during training, as suggested by Moore (see Moore, [col 26, lines 60-64] quote above.)
Regarding claim 9, Feurer and Wang teaches the method as defined in claim 2, (see rejection of claim 2).
Wang further teaches wherein the verification model takes one or more of the following factors into account for the quality assessment: jumps in the training learning progression or validation learning progression; number of epochs after which the training learning progression or validation learning progression saturates, and a value of the quality measures during saturation; ([Wang, page 4] “The server computer 102, via the HOM 122 at block 220, iteratively runs a training portion of the preprocessed GTD 106 through each of the pipelines 402 with the hyperparameter sets generated by the PGM 116. The HOM 122 assess performance of each pipeline 402 iteratively, comparing performance for each of the associated hyperparameter sets. The HOM 122 determines favored hyperparameter sets for each pipeline 402.”, wherein the examiner interprets “iteratively running training data through pipelines and assessing performance for different hyperparameter sets” and “determining favored hyperparameter sets” to be the same as “taking into account jumps in the training learning progression or validation learning progression, the number of epochs before saturation, and the value of quality measures during saturation,” as both are directed to evaluating the progression of training performance and determining optimal conditions for improved quality.)
difference between the training learning progression and the validation learning progression; divergence of the training learning progression or validation learning progression; ([Wang, [0020], “The server computer 102 includes a Pipeline Validation User Interface ( PVUI ) 128 that allows a user to examine pipeline execution results to correct , remove selected pipelines, and otherwise give input regarding pipeline performance to increase result interpretability and user confidence. The server computer 102 includes an Ensemble Assembly Module (EAM) 130 that combines multiple pipelines into a cooperative bundle. The server computer 102 also includes an Ensemble Pipeline Application Module 132 applies the pipelines in the ensemble to provided data 106 which can indicate whether multiple pipelines provide results that agree.”, wherein the examiner interprets PVUI “allows a user to examine pipeline execution results to correct” to be the same as examining the “difference between the training learning progression and the validation learning progression” as both require examining differences between actual and validation results which is subsequently used to make changes to the machine learning models.)initial fluctuations in the training learning progression and validation learning progression and subsequent monotonous training learning progression and validation learning progression; (Wang, [0019], “The server computer 102 is in communication with a source of Ground Truth Data (GTD) 106 useful for training and validating the models to be selected by the system 100. According to aspects of the present invention, the GTD 106 is text - based and can reflect many different kind of a of information.”, wherein the examiner interprets “in communication with a source of GTD” to be the same as initial and subsequent progression comparison of training versus validating results for information between their fluctuations.)Wang does not teach whether an optimum of the values of trainable internal model parameter values at which a quality measure is below a predetermined limit value is reached early.Moore teaches whether an optimum of the values of trainable internal model parameter values at which a quality measure is below a predetermined limit value is reached early. ([Moore, col 24, lines 5-23] “The iterative algorithm first finds a translational shift S1131 and corrects the translational shift of the object or speckled pattern that was projected onto the sensor S1132...This may then be used to update the wavefunction for a high-resolution magnitude projection S1135...The system may continue to shift the diffuser or object until the defined convergence criteria is met at block S1138.”, wherein the examiner interprets “iterative adjustments to find and correct translational shifts and updating projections until convergence criteria are met” to be the same as detecting the optimum state of the model parameter values once the “convergence criteria” are met, where “convergence criteria” is the same as “a predetermined limit value is reached”.)Feurer, Wang, Moore, and the instant application are analogous art because they are all directed to methods for monitoring and optimizing machine learning processes based on quality measures derived from training and validation progressions.It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model disclosed by Feurer and Wang to include the “iterative adjustments to correct translational shifts and updating projections until convergence criteria are met” disclosed by Moore. One would be motivated to do so to effectively enhance the accuracy and robustness of training evaluation processes, as suggested by Moore (see Moore, [col 24, lines 5-23] quote above.)
Regarding claim 10, Feurer and Wang teaches the method as defined in claim 2 (see rejection of claim 2).Wang does not teach wherein the quality assessment comprises a suggestion for a modification of training parameters during the ongoing learning process or for a new learning process to be initiated;.Moore teaches wherein the quality assessment comprises a suggestion for a modification of training parameters during the ongoing learning process or for a new learning process to be initiated. ([Moore, col 32, lines 31-35] “In some embodiments, the trained models may be static and therefore unchanged with the addition of new data. In some embodiments, the models may continue to evolve to improve accuracy through additional data and user feedback.”, wherein the examiner interprets “models continuing to evolve to improve accuracy through additional data and user feedback” to be the same as “a suggestion for a modification of training parameters during the ongoing learning process or for a new learning process to be initiated,” as both are directed to adapting or modifying the training process based on ongoing input to achieve improved model performance.)Feurer, Wang, Moore, and the instant application are analogous art because they are all directed to methods for monitoring and improving the learning process of a machine learning model based on evaluation metrics derived during training.It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model as disclosed by Feurer and Wang to include the process to “improve accuracy” disclosed by Moore. One would be motivated to do so to effectively enhance the adaptability and accuracy of machine learning model training processes, as suggested by Moore (see Moore, [col 32, lines 31-35] quote above.)
Regarding claim 13, Feurer, Wang, and Moore teaches the method as defined in claim 10 (see rejection of claim 10).Moore further teaches wherein the modification of training parameters comprises a modification of the model parameter values; wherein the verification machine learning model also receives to this end, in addition to the training learning progression and the validation learning progression, associated model parameter values of the machine learning model as inputs; ([Moore, col 20, lines 15-20], “The method 200 may include one or more preprocessing methods S210, followed by image loading S220, and receiving as an input one or more system parameters S230”, wherein the examiner interprets “receiving as an input one or more system parameters” to be the same as “the verification machine learning model receiving associated model parameter values as inputs,” as both are directed to utilizing parameter inputs to enhance the processing or evaluation of a system.)Feurer, Wang, Moore, and the instant application are analogous art because they are all directed to methods for modifying training parameters and incorporating associated data to improve the learning process of a machine learning model.It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model disclosed by Feurer and Wang to include the “receiving as an input one or more system parameters” disclosed by Moore. One would be motivated to do so to efficiently improve the adaptability and precision of the training process by utilizing associated parameters to refine evaluations, as suggested by Moore (see Moore, [col 20, lines 15-20] quote above.)
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Feurer in view of Wang in view of Moore further in view of Colley et. al, US20210090694A1 (referred herein as Colley).
Regarding claim 11, Feurer, Wang, and Moore teaches the method as defined in claim 10 (see rejection of claim 10).Feurer, Wang, and Moore do not teach wherein the modification of training parameters comprises at least one of: a modification of a learning rate and a modification of a set number of epochs;
Colley teaches wherein the modification of training parameters comprises at least one of: a modification of a learning rate and a modification of a set number of epochs; ([Colley, [1861] “We continued training until we observed diminishing improvement in validation accuracy. We found that Inception-v3 and ResNet-50 performed competitively, achieving max accuracy of 92.2% and 91.44% respectively over 40 epochs.”, wherein the examiner interprets “continuing training until diminishing improvement in validation accuracy is observed and achieving maximum accuracy over a set number of epochs” to be the same as “a modification of a set number of epochs,” as both are directed to adjusting the training process to optimize performance by considering changes in validation metrics over defined training intervals.)Feurer, Wang, Moore, Colley, and the instant application are analogous art because they are all directed to methods for monitoring, evaluating, and optimizing the training process of machine learning models by modifying training parameters to improve performance.It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model disclosed by Feurer, Wang, and Moore to include the method for “continuation of training” to maximize validation accuracy taught by Colley. One would be motivated to do so to effectively improve the optimization of training processes by identifying and addressing diminishing returns in accuracy, as suggested by Colley (see Colley, [1861] quote above.)
Claims 12 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Feurer in view of Wang in view of Moore in further view of NPL reference “Hyperparameter optimization” by Hutter et. al (referred herein as Hutter).
Regarding claim 12, Feurer, Wang, and Moore teaches the method as defined in claim 10 (see rejection of claim 10).Feurer, Wang, and Moore do not teach wherein, in the event that the quality assessment assumes a local optimum of the values of the trainable internal model parameters, the modification of training parameters comprises a one-time or repeated increase of a learning rate in order to escape the local optimum of the values of the trainable internal model parameters;.Hutter teaches wherein, in the event that the quality assessment assumes a local optimum of the values of the trainable internal model parameters, the modification of training parameters comprises a one-time or repeated increase of a learning rate in order to escape the local optimum of the values of the trainable internal model parameters;. ([Hutter, page 85, sec 4.3] “In order to select its next hyperparameter configuration λ using model ML, SMBO uses a so-called acquisition function a ML : → R, which uses the predictive distribution of model ML at arbitrary hyperparameter configurations λ ∈ to quantify (in closed form) how useful knowledge about λ would be. SMBO then simply maximizes this function over to select the most useful configuration λ to evaluate next. Several well-studied acquisition functions exist [18, 27, 29]; all aim to automatically trade off exploitation (locally optimizing hyperparameters in regions known to perform well)”, wherein the examiner interprets “automatically trading off exploitation by locally optimizing hyperparameters in regions known to perform well” to be the same as “identifying and escaping a local optimum of the model parameter values,” as both are directed to refining hyperparameter configurations to improve performance by overcoming suboptimal parameter values.)Fuerer, Wang, Moore, Hutter, and the instant application are analogous art because they are all directed to methods for monitoring, assessing, and optimizing training processes in machine learning models by analyzing and improving model parameter configurations during training.It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 10 disclosed by Feurer, Wang, and Moore to include the process of “locally optimizing hyperparameters” taught by Hutter. One would be motivated to do so to effectively improve the adaptability and optimization of the training process, as suggested by Hutter (see Hutter, [page 85, sec 4.3] quote above.)
Regarding claim 14, Feurer, Wang, and Moore teach the method as defined in claim 13, (see rejection of claim 13, 10 and/or 2).Moore further teaches wherein the verification machine learning model comprises a neural network trained by a supervised learning process, in which training data comprises a plurality of training learning progressions, ([Moore, col 32, lines 7-10], “In some embodiments, these features may be used as criteria to train classification machine learning or deep learning models in supervised learning.”, wherein the examiner interprets “learning models in supervised learning” to be the same as “model comprises a neural network trained by a supervised learning process”.)Wang further teaches validation learning progressions and associated values of the trainable model parameters, as well as modifications of the values of the trainable internal model parameters as target data; ([Wang, [0020], “The server computer 102 includes a Pipeline Validation User Interface (PVUI) 128 that allows a user to examine pipeline execution results to correct, remove selected pipelines, and otherwise give input regarding pipeline performance to increase result interpretability and user confidence.”, wherein the examiner interprets “Validation User Interface…to examine pipeline execution results to correct .. pipelines” to be the same as “validation learning progressions … as well as modifications of the model parameter values”).
Feurer, Wang, Moore, and the instant application are analogous art, because they are all directed to a model comprising a trained network/ training data comprising learning progressions and trainable parameters.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 13 disclosed by Feurer, Wang, and Moore to include “examine pipeline execution results to correct, remove selected pipelines, and otherwise give input regarding pipeline performance to increase result interpretability and user confidence” disclosed by Wang. One would be motivated to edit/manipulate parameters of specific data to be used as validating learning progressions and modified/parameter values as suggested by Wang ([Wang, [0020], “The server computer 102 includes a Pipeline Validation User Interface (PVUI) 128 that allows a user to examine pipeline execution results to correct, remove selected pipelines, and otherwise give input regarding pipeline performance to increase result interpretability and user confidence.”)Feurer, Wang, and Moore do not teach or wherein the verification machine learning model comprises a neural network trained by a reinforcement learning method, in which a reinforcement learning agent learns using a predefined training environment, for an input comprising a training learning progression, a validation learning progression and associated model parameter values, how to modify the model parameter values in order to optimize the quality assessment;.
Hutter teaches or wherein the verification machine learning model comprises a neural network trained by a reinforcement learning method, in which a reinforcement learning agent learns using a predefined training environment, for an input comprising a training learning progression, a validation learning progression and associated model parameter values, how to modify the model parameter values in order to optimize the quality assessment; ([Hutter, page 51, Sec 2.4.4] “Meta-learning is certainly not limited to (semi-)supervised tasks, and has been successfully applied to solve tasks as varied as reinforcement learning, active learning, density estimation and item recommendation. The base-learner may be unsupervised while the meta-learner is supervised, but other combinations are certainly possible as well.” AND “Duan et al. [39] propose an end-to-end reinforcement learning (RL) approach consisting of a task-specific fast RL algorithm which is guided by a general-purpose slow meta-RL algorithm. The tasks are interrelated Markov Decision Processes (MDPs). The meta-RL algorithm is modeled as an RNN, which receives the observations, actions, rewards and termination flags. The activations of the RNN store the state of the fast RL learner, and the RNN”s weights are learned by observing the performance of fast learners across tasks.”, wherein the examiner interprets “a reinforcement learning approach using a predefined training environment and inputs such as observations, actions, and rewards” for verification to be the same as “a reinforcement learning agent learning from inputs of training and validation learning progressions and associated model parameter values to modify model parameters to optimize quality assessment,” as both are directed to methods for training neural networks to optimize learning processes based on associated inputs, validation, and environments.)Feurer, Wang, Moore, Hutter, and the instant application are analogous art because they are all directed to methods for training machine learning models using various learning paradigms (supervised and reinforcement) to optimize model performance.It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model disclosed by Feurer, Wang, and Moore to include the process of meta-learning applied to supervised and reinforcement learning tasks taught by Hutter. One would be motivated to do so to effectively enhance the adaptability and optimization of the learning process, as suggested by Hutter (see Hutter, [page 51, Sec 2.4.4] quote above.)
Claim(s) 15-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feurer in view of Wang further in view of Hutter.
Regarding claim 15, Feurer and Wang teaches the method as defined in claim 2 (see rejection of claim 2).Feurer and Wang do not teach wherein the training learning progression and the validation learning progression are input into a prediction machine learning model that is trained to predict future progressions of an input training learning progression and validation learning progression from the input training learning progression and validation learning progression and to add the predicted future progressions onto the input training learning progression and validation learning progression, wherein the training learning progression and validation learning progression supplemented by the prediction are output to a user or to the verification model;.
Hutter teaches wherein the training learning progression and the validation learning progression are input into a prediction machine learning model that is trained to predict future progressions of an input training learning progression and validation learning progression from the input training learning progression and validation learning progression and to add the predicted future progressions onto the input training learning progression and validation learning progression, wherein the training learning progression and validation learning progression supplemented by the prediction are output to a user or to the verification model; ([Hutter, page 18, sec 1.4.3] “Multi-task Bayesian optimization [147] uses a multi-task Gaussian process to model the performance of related tasks and to automatically learn the tasks” correlation 1 during the optimization process.”, wherein the examiner interprets “modeling the performance of related tasks and automatically learning their correlations during the optimization process” to be the same as “training a prediction machine learning model to predict future progressions of training and validation learning progressions based on their input values and adding predicted progressions to the original inputs,” as both are directed to leveraging machine learning to predict and enhance progressions of performance data based on correlations derived from prior tasks.)
Feurer, Wang, Hutter, and the instant application are analogous art because they are all directed to methods for predicting and optimizing the performance of machine learning models by analyzing and extending learning progressions using predictive methods.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model disclosed by Feurer and Wang to include the processes to “model the performance of related tasks and automatically learn correlations during the optimization process” taught by Hutter. One would be motivated to do so to effectively enhance the prediction and extension of learning progressions by incorporating inter-task correlations, as suggested by Hutter (see Hutter, [page 18, sec 1.4.3] quote above.)
Regarding claim 16, Feurer and Wang teaches the method as defined in claim 2 (see rejection of claim 2).Feurer and Wang do not teach wherein the verification machine learning model is configured to conduct an anomaly detection in order to determine deviations from typical training progressions, wherein the verification machine learning model for the anomaly detection is designed as an autoencoder trained with training learning progressions and validation learning progressions that do not contain any anomalies;.Hutter teaches wherein the verification machine learning model is configured to conduct an anomaly detection in order to determine deviations from typical training progressions, wherein the verification machine learning model for the anomaly detection is designed as an autoencoder trained with training learning progressions and validation learning progressions that do not contain any anomalies; ([Hutter, page 49, sec 2.4.2] “An early meta-learning approach is to create recurrent neural networks (RNNs) able to modify their own weights. During training, they use their own weights as additional input data and observe their own errors to learn how to modify these weights in response to the new task at hand.”, wherein the examiner interprets “training recurrent neural networks with their own weights as additional input data and observing their own errors” to be the same as “training a verification machine learning model with training learning progressions and validation learning progressions that do not contain any anomalies,” as both are directed to training machine learning models to detect deviations by learning from typical, non-anomalous training data.)Feurer, Wang, Hutter, and the instant application are analogous art because they are all directed to methods for training machine learning models to analyze and optimize training progressions.It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model disclosed by Feurer and Wang to include the use of training data that observes its own errors to “modify weights during training” taught by Hutter. One would be motivated to do so to efficiently enhance the ability of machine learning models to detect and address anomalies, as suggested by Hutter (see Hutter, [page 49, sec 2.4.2] quote above.)
Regarding claim 17, Feurer and Wang teaches the method as defined in claim 2 (see rejection of claim 2).Feurer and Wang do not teach wherein the verification model is designed to define assessment criteria for the generation of the quality assessment depending on contextual data, wherein the contextual data relate to the machine learning model or the training of the machine learning model, wherein the contextual data comprise information regarding one or more of the following aspects: - type of a learning method, wherein a distinction is made at least between supervised training, unsupervised training, reinforcement learning and a use of adversarial networks; - type of a task of the machine learning model, wherein a distinction is made at least between a classification, segmentation, detection or regression; - architecture of the machine learning model; - values of hyperparameters of the machine learning model; - a type of the training/validation data;Hutter teaches:wherein the verification model is designed to define assessment criteria for the generation of the quality assessment depending on contextual data, wherein the contextual data relate to the machine learning model or the training of the machine learning model, wherein the contextual data comprise information regarding one or more of the following aspects: (Hutter, page 106, “We performed optimization runs of up to 300 function evaluations searching the subset of the space depicted in Fig.5.1, and compared the quality of solution with specialized searches of specific classifier types (including best known classifiers).” wherein the examiner interprets “optimization runs of up to 300 function evaluations” and “compared the quality of solution” to be the same as “contextual data” that is “designed to define assessment”, respectively)type of a learning method, wherein a distinction is made at least between supervised training, unsupervised training, reinforcement learning and a use of adversarial networks; ([Hutter, page 179] “matching algorithms to problems (which may include supervised, unsupervised, or reinforcement learning, or other settings), acquisition of new data (active learning, query learning, reinforcement learning, causal experimentation), management of large volumes of data including the creation of appropriately-sized and stratified training, validation, and test sets.”, wherein the examiner interprets “matching algorithms to problems (which may include supervised, unsupervised, or reinforcement learning, or other settings) … and managing training, validation, and test data sets” to be the same as type of a learning method .. supervised, unsupervised, reinforcement learning, and use of adversarial networks.)type of a task of the machine learning model, wherein a distinction is made at least between a classification, segmentation, detection or regression; - architecture of the machine learning model; ([Hutter, page 73], “Notable first steps in this direction are applying Neural Architecture Search (NAS) to language modeling [74], music modeling [48], image restoration [58] and network compression [3]; applications to reinforcement learning, generative adversarial networks, semantic segmentation, or sensor fusion.”, wherein the examiner interprets “applications to reinforcement learning, generative adversarial networks, semantic segmentation” to be the same as “distinction is made at least between a classification, segmentation, detection, or regression” and Neural Architecture Search (NAS) to be the same as “architecture of the machine learning model”.)values of hyperparameters of the machine learning model; - a type of the training/validation data; ([Hutter, page 5, sec 1.2] “the Combined Algorithm Selection and Hyperparameter Optimization (CASH) problem, in which the choice of a classification algorithm is modeled as a categorical variable, the algorithm hyperparameters are modeled as conditional hyperparameters.”, wherein the examiner interprets modeling classification algorithms as categorical variables and hyperparameters as conditional hyperparameters to be the same as “define assessment criteria for the generation of the quality assessment depending on contextual data” such as hyperparameters and learning methods, as both are directed to creating assessment criteria tailored to specific machine learning contexts to optimize quality assessment generation.)Feurer, Wang, Hutter, and the instant application are analogous art because they are all directed to methods for optimizing machine learning models through contextual data and tailored assessment criteria.It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model disclosed by Feurer and Wang to include the process of “matching algorithms to problems (which may include supervised, unsupervised, or reinforcement learning, or other settings)” taught by Hutter. One would be motivated to do so to effectively improve the adaptability and accuracy of quality assessments by incorporating diverse machine learning contexts and task types, as suggested by Hutter (see Hutter, [page 179] quote.)
Claim(s) 19 is rejected under 35 U.S.C. 103 as being unpatentable over Feurer in view of Guyon.
Regarding claim 19, Feurer teaches:
A method for monitoring a learning process of a machine learning model including a generative adversarial network, ([Feurer, Chapter 1, Section 1.4, p. 14] “First, we review methods which model an algorithm's learning curve during training and can stop the training procedure if adding further resources is predicted to not help” and “Learning curve extrapolation is used in the context of predictive termination [26], where a learning curve model is used to extrapolate a partially observed learning curve for a configuration, and the training process is stopped if the configuration is predicted to not reach the performance of the best model trained so far in the optimization process” and [Feurer, Chapter 2, Section 2.4.2, page 48] “Generative Adversarial Networks (GANs; Goodfellow et al., 2014) are generative models that learn to produce realistic samples. They consist of two neural networks: a generator and a discriminator”, wherein the examiner interprets “methods which model an algorithm's learning curve during training” and “extrapolate a partially observed learning curve” and “training process is stopped if the configuration is predicted to not reach the performance” to be the same as “a method for monitoring a learning process of a machine learning model” because they are both directed to a process that observes and evaluates the progression of model training to make determinations about the quality of the learning process. The examiner further interprets wherein the examiner interprets “Generative Adversarial Networks (GANs)” and “a generator and a discriminator” to be the same as “a machine learning model including a generative adversarial network” because they are both directed to the same type of machine learning architecture consisting of adversarial neural network components.)
wherein the machine learning model includes trainable internal model parameters for processing input training data to calculate a model output, the method comprising ([Feurer, Chapter 4, sec 4.2, page 83] “A learning algorithm A maps a set of training data points di to such a function, which is often expressed via a vector of model parameters”, wherein the examiner interprets “A learning algorithm A maps a set of training data points” and “expressed via a vector of model parameters” to be the same as “trainable internal model parameters for processing input training data to calculate a model output” because they are both directed to internal parameters of a machine learning model that are derived from processing training data to produce an output function.)
launching a learning process using training data to adjust values of the trainableinternal model parameters of the machine learning model; ([Feurer, Chapter 2, Sec 2.4.2, page 49] “During training, they use their own weights as additional input data and observe their own errors to learn how to modify these weights in response to the new task at hand. The updating of the weights is defined in a parametric form that is differentiable end-to-end and can jointly optimize both the network and training algorithm using gradient descent”, wherein the examiner interprets “During training, they use their own weights” and “learn how to modify these weights” and “The updating of the weights” and “using gradient descent” to be the same as “to adjust values of the trainable internal model parameters of the machine learning model” because they are both directed to modifying learned model weights iteratively during the training procedure through optimization algorithms such as gradient descent.)
calculating, during the learning process, at least one quality measure for respectively current values of the trainable internal model parameters, [Feurer, Chapter 1, Section 1.4.1, p. 14] “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration”, wherein the examiner interprets “the performance of an iterative algorithm measured for each iteration” to be the same as “calculating, during the learning process, at least one quality measure for respectively current values of the trainable internal model parameters” because they are both directed to measuring a performance metric at successive iterations during training where the model parameters have different values at each iteration.)
repeating calculating the quality measure in different epochs of the learning process: ([Feurer, Chapter 1, Section 1.4.1, p. 14] “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration (or every i-th iteration if the calculation of the performance is expensive)”, wherein the examiner interprets “measured for each iteration” to be the same as “repeating calculating the quality measure in different epochs of the learning process” because they are both directed to repeatedly evaluating performance/quality values at successive stages of training.)
forming at least one training learning progression from the quality measures of different epochs; ([Feurer, Chapter 1, Section 1.4.1, p. 14] “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration (or every i-th iteration if the calculation of the performance is expensive)”, wherein the examiner interprets “learning curves” and “the performance of an iterative algorithm measured for each iteration” to be the same as “forming at least one training learning progression from the quality measures of different epochs” because they are both directed to building a progression or curve from performance/quality values that are measured successively at different stages during the training process.)
inputting the at least one training learning progression into a verification model during the learning process; ([Feurer, Chapter 1, sec 1.4, p. 14] “First, we review methods which model an algorithm's learning curve during training and can stop the training procedure if adding further resources is predicted to not help”, wherein the examiner interprets “methods which model an algorithm's learning curve during training” to be the same as “inputting the at least one training learning progression into a verification model during the learning process” because they are both directed to a model/method operating during training that receives and processes observed learning curve information as input.)
wherein the verification model comprises a verification machine learning model trained to generate a quality assessment of the learning process of the machine learning model depending on the at least one training learning progression; ([Feurer, Chapter 1, Section 1.4.1, p. 14] “Learning curve extrapolation is used in the context of predictive termination [26], where a learning curve model is used to extrapolate a partially observed learning curve for a configuration, and the training process is stopped if the configuration is predicted to not reach the performance of the best model trained so far in the optimization process”, wherein the examiner interprets “learning curve model” to be the same as “a verification machine learning model” because they are both directed to a model that processes learning curve data, and wherein the examiner interprets “used to extrapolate a partially observed learning curve” and “training process is stopped if the configuration is predicted to not reach the performance of the best model trained so far” to be the same as “trained to generate a quality assessment of the learning process of the machine learning model depending on the at least one training learning progression” because they are both directed to a model that has been configured/trained to evaluate training quality based on observed learning progression data and produce a determination about that training process.)
wherein the verification machine learning model is trained by an unsupervised learning process, in which a plurality of training learning progressions are input into the verification machine learning model as verification machine learning model training data, or wherein the verification machine learning model is trained by a supervised learning process, in which a plurality of training learning progressions labelled with predetermined quality assessments are used as verification machine learning model training data such that the training learning progressions are input into the verification machine learning model and ([Feurer, Chapter 1, Section 1.4.1, p. 15] “We train parametric learning curve models on learning curves observed in previous HPO runs. These models are then used to predict the performance of new configurations based on their partially observed learning curves”, wherein the examiner interprets “We train parametric learning curve models on learning curves observed in previous HPO runs” to be the same as “the verification machine learning model is trained” using “a plurality of training learning progressions” as “verification machine learning model training data” because they are both directed to training a model using previously observed learning curve data from multiple training runs.)
deviations of the verification machine learning model's outputs to the predetermined quality assessments are used to update parameters of the verification machine learning model. ([Feurer, Chapter 1, Section 1.4.1, p. 15] “These models are then used to predict the performance of new configurations based on their partially observed learning curves”, wherein the examiner interprets “These models are then used to predict the performance of new configurations based on their partially observed learning curves” to be the same as the verification model generating quality assessments for new learning processes because they are both directed to applying a trained model to evaluate new training progressions, combined with [Feurer, Chapter 2, Section 2.4.2, page 49] “During training, they use their own weights as additional input data and observe their own errors to learn how to modify these weights in response to the new task at hand. The updating of the weights is defined in a parametric form that is differentiable end-to-end and can jointly optimize both the network and training algorithm using gradient descent”, wherein the examiner interprets “observe their own errors” and “learn how to modify these weights” and “The updating of the weights” to be the same as “deviations of the verification machine learning model's outputs” being “used to update parameters of the verification machine learning model” because they are both directed to adjusting model parameters based on observed errors during the training process.)
Feurer does not teach the quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from an input to the machine learning model;
Guyon teaches the quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from an input to the machine learning model; ([Guyon, col. 32, lines 10-23] “Four metrics of classifier quality were used. Each value is computed both on the training set with the leave-one-out method and on the test set”, wherein the examiner interprets “Four metrics of classifier quality” and “Each value is computed” to be the same as “the quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from an input to the machine learning model” because they are both directed to quality metrics that evaluate how well the model performs with its current learned parameter values when processing input data.)
Feurer, Guyon, and the instant application are analogous art because they are all directed to monitoring a learning process of a machine learning model.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method claim 19 disclosed by Feurer to include the quality metrics disclosed by Guyon. One would be motivated to do so to effectively quantify and compare learning-process quality using explicit classifier-quality metrics computed on training and held-out test data for assessment of performance during learning, as suggested by Guyon (Guyon, [col. 32, lines 10-23] “Four metrics of classifier quality were used. Each value is computed both on the training set with the leave-one-out method and on the test set.”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DEVAN KAPOOR whose telephone number is (703)756-1434. The examiner can normally be reached Monday - Friday: 9:00AM - 5:00 PM EST (times may vary).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEVAN KAPOOR/Examiner, Art Unit 2126
/LUIS A SITIRICHE/Primary Examiner, Art Unit 2126