DETAILED ACTION
This action is responsive to the application filed on 04/27/2026. Claims 1-20 are pending and have been examined.
This action is Final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C.
120, 121, 365(c), or 386(c) is acknowledged.
Response to Arguments
Argument 1: The applicant argues that the amended claims are not merely abstract data analysis because the claimed training system is allegedly restructured in a specific algorithmic way: it forms both a training learning progression and a validation learning progression, feeds both into a verification model during the learning process, and uses the verification model to generate a quality assessment. The applicant frames this as an “algorithmic restructuring” of the training technology itself, not just a desired result. They further argue that the quality assessment improves the machine-learning training process by accelerating training and reducing expert intervention, which in turn improves the microscopy system that processes microscope images using the trained model. For claims 4 and new claim 20, the applicant herein emphasizes that the claims now recite automatically deciding whether to continue, abort, or modify training before the stopping criterion is reached, which they argue further supports a technical improvement rather than an abstract idea.
Response to Argument 1: The applicant’s arguments have been considered but are not persuasive. The amendments do not change the character of the claims as being directed to abstract evaluation, organization, and decision-making using generic machine-learning/computing components. Although the applicant characterizes the amendments as an “algorithmic restructuring,” the claim language still recites, at a high level of generality, calculating a training data quality measure, calculating a validation data quality measure, organizing those values into two learning progressions, feeding the progressions to a verification model, and generating a quality assessment based on the progressions. These limitations are directed to evaluating calculated results, organizing values into tracked sequences, and rendering an assessment or judgment regarding the quality of a learning process, which are mental processes and/or mathematical concepts as set forth in the rejection. The claims also do not recite a particular improvement to the microscope, camera, image sensor, processor architecture, memory structure, image-processing technique, loss function, parameter-update rule, or verification-model architecture. Rather, the microscope/camera/computing-device language merely provides the technological environment in which the abstract training-evaluation process is applied, and the “for processing a microscope image” language identifies an intended field of use without specifying a particular technical improvement to microscope-image processing. The asserted benefits of accelerating training and reducing expert intervention are not equal with the claim scope because the claims recite the desired result of producing a quality assessment and, in claims 4 and 20, deciding whether to continue, abort, or modify training, without reciting a specific technical mechanism that improves the functioning of the computer, microscope, or machine-learning model itself. The automatic decision limitation of claims 4 and 20 likewise remains an evaluative judgment as to whether to continue or abort a process based on quality information, and automating that judgment using a generic model or computer does not supply significantly more. Accordingly, the amendments do not integrate the abstract idea into a practical application or add an inventive concept, and the rejection under 35 U.S.C. 101 is maintained.
Argument 2: The applicant’s main art argument is that the Office has not established a prima facie case because the cited references allegedly do not teach the amended “two progression” structure. The applicant first attacks the previous office action’s reliance on “Feurer” by arguing that the office action improperly cites other chapters/papers from the same book, Vanschoren and Kotthoff, as if they were Feurer, and that the Office provides no motivation to combine those separate papers. The applicant also argues that Feurer’s cited learning-curve disclosure describes, at most, one learning curve and does not teach separately calculating both a training data quality measure and an additional validation data quality measure, repeating both across epochs, forming both a training learning progression and a validation learning progression, or feeding both progressions to a verification model. Applicant says Wang and Barnhill/Guyon do not cure this deficiency, and the same defect carries through the dependent rejections. For claim 19, Applicant separately argues that Feurer’s learning-curve model merely predicts a future learning curve and is not a trained verification machine-learning model that generates a quality assessment; Applicant also challenges the location/accuracy of certain cited Feurer sentences and argues that Feurer does not teach using deviations from predetermined quality assessments to update the verification model.
Response to Argument 2: The applicant’s arguments have been considered but are not persuasive. The applicant’s art arguments are directed largely to the prior reliance on Feurer for the amended “two progression” structure, however, the present rejection has been updated and does not rely on Feurer alone for that structure. For claim 1, the rejection now relies on Moore for the microscopy system and microscope-image processing environment, new reference Nair is for neural-network training, trainable internal parameters, hyperparameters, epoch-based loss/time-series information, feeding such information to a predictive/learning-curve model, and generating a model-based quality assessment, and Prechelt for the separate training and validation quality measures, validation data not used for weight adjustment, training error curve, validation error curve, and assessment depending on both training-progress behavior and validation/generalization behavior. Thus, the argument that Feurer teaches, at most, one learning curve does not overcome the updated rejection because Prechelt expressly teaches both “Etr(t), the training set error” and “Eva(t), the validation error,” teaches that the validation set is not used for weight adjustment, and teaches the training error curve and validation error curve as separate progressions. Also, the argument that the art does not teach feeding both progressions to a verification model is also not persuasive because Nair teaches time-series features used as input to a model, including loss values, loss functions, or validation loss functions from training or experiments, with the loss/time-series data organized by epoch, and further teaches a predictive/learning-curve model incorporated into an early-stopping algorithm that generates a quality assessment and may terminate training based on predicted validation-loss behavior. The updated claim 2 rejection similarly addresses the alleged deficiency by relying on Prechelt for the training/validation quality-measure and progression limitations and Nair for epoch-organized loss/progression information fed to a verification model and used to generate a predictive quality assessment. Criticism regarding treating separate AutoML chapters as Feurer is therefore not dispositive of the present rejection because the Office has supplied additional art and articulated reasons to combine, including applying training monitoring to microscope-image machine-learning models, distinguishing training-set performance from validation-set performance, detecting overfitting, improving generalization, and reducing unnecessary training through predictive early stopping. With respect to claim 19, The applicant’s arguments are also not persuasive because Feurer is mapped to monitoring a learning process, GANs, learning-curve extrapolation, predictive termination, and training parametric learning-curve models on learning curves observed in previous HPO runs, which are then used to predict performance of new configurations based on partially observed learning curves; the Office interprets this as a trained verification machine-learning model generating a quality assessment from learning progression information. Feurer is further mapped to observing errors and modifying weights to address updating model parameters based on deviations, and Guyon is relied upon for explicit classifier-quality metrics computed on training and test data. Accordingly, the updated mappings cure the alleged Feurer-only deficiency and establish that the cited references teach or suggest the amended two-progression structure, feeding progression information to a verification model during training, and generating a quality assessment depending on training and validation behavior; therefore, the rejection under 103 is maintained.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition
of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the
conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1,
Step 1: This claim is directed to a system, which falls within the machine category, one of the four statutory categories. Therefore, claim 1 satisfies Step 1.
Step 2A Prong 1:
“calculating a training data quality measure for current values of the trainable internal model parameters during the learning process, the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data” - This limitation is directed to evaluating a calculated result to determine a quality measure, which can be performed in the human mind using observation, evaluation, and judgment, and/or as a mathematical calculation. This falls under the mental process and mathematical concept groupings of abstract ideas.
“calculating a validation data quality measure, in addition to the training data quality measure, for the current values of the trainable internal model parameters during the learning process...wherein the validation data is not used in the learning process to adjust the values of the trainable internal model parameters and instead the validation data is only used for calculating the validation data quality measure” - This limitation is directed to a second evaluative judgment performed against a different set of data, which amounts to observation and evaluation performable in the human mind, and thus is a mental process.
“forming two learning progressions, including forming a training learning progression from successively determined training data quality measures and forming, in addition to the training learning progression, a validation learning progression from successively determined validation data quality measures” - This limitation is directed to organizing successively determined values into two tracked sequences, which is a process capable of being performed using observation and evaluation with the aid of pen and paper, and is thus a mental process.
“the verification model generating a quality assessment of the learning process of the machine learning model, the quality assessment depending on both the training learning progression and the validation learning progression” -- This limitation is directed to rendering a judgment as to the quality of a process based on two sets of organized values, which is an evaluative, observational mental process.
Step 2A Prong 2 and Step 2B:
“A microscopy system comprising: at least one microscope, which comprises at least one camera for capturing a microscope image and a computing device, wherein the computing device comprises an imaging processing program for processing the microscope image using a machine learning model” - This limitation recites a microscope system where a camera captures an image and a computing device executes an imaging processing program to process the image using a machine learning model. This limitation recites mere instructions to apply a camera and an image processing program to be generically executed on a computer (computing device), and the microscope being used in its ordinary capacity/high generality, for which cannot be integrated into a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
“wherein the machine learning model includes a set of trainable internal model parameters for processing input training data to calculate a model output, the trainable internal model parameters being distinct from hyperparameters that define training settings or model architecture; and a training system for carrying out a learning process of the machine learning model, wherein the training system is configured for: adjusting values of the trainable internal model parameters of the machine learning model successively in the learning process depending on training data processed with the trainable internal model parameters” - This limitation recites conventional data gathering and manipulation using a generic training system, which is insignificant extra-solution activity that cannot be integrated into a practical application (see MPEP 2106.05(g)); further, under Step 2B, using a machine learning model with parameters to process input data and produce an output through iterative adjustment is a well-understood, routine, and conventional (WURC) activity that cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“feeding the two learning progressions, including the training learning progression and additionally the validation learning progression, to a verification model during the learning process” - This limitation amounts to no more than transmitting data to a generic model within a particular field of use/environment (the recited "verification model" during the "learning process"), and does not integrate the exception into a practical application, nor does it provide significantly more than the judicial exception (see MPEP 2106.05(h)).
“repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process which uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs;” - This limitation is directed to repetitive calculation across epochs, which is an insignificant, extra-solution activity that cannot be integrated into a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, the act of repetitive calculation is also a well-understood, routine, and conventional activity (WURC), and cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Thus, claim 1 is non-patent eligible.
Regarding claim 2,
Step 1:
The claim is directed to a method, which is a process, one of the four statutory categories. Therefore, the claim satisfies Step 1.
Step 2A Prong 1:
"calculating a training data quality measure... for current values of the trainable internal model parameters during the learning process, the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data; and calculating a validation data quality measure for the current values of the trainable internal model parameters during the learning process" - This limitation is directed to evaluating calculated results against training and validation data respectively, which can be performed in the human mind using observation and judgment, and thus is a mental process.
“forming two learning progressions, including forming a training learning progression from successively determined training data quality measures; and forming, in addition to the training learning progression, a validation learning progression from successively determined validation data quality measures” - This limitation is directed to organizing successively determined values into two tracked sequences, performable using observation and evaluation with the aid of pen and paper, and thus is a mental process.
“the verification model generating a quality assessment of the learning process of the machine learning model, the quality assessment depending on both the training learning progression and the validation learning progression” - This limitation is directed to an evaluative judgment (the quality assessment) as to process quality based on two sets of organized values, and thus is a mental process.
Step 2A Prong 2 and Step 2B:
“A method for monitoring a learning process of a machine learning model for processing a microscope image, wherein the machine learning model includes a set of trainable internal model parameters for processing input training data to calculate a model output, the trainable internal model parameters being distinct from hyperparameters that define training settings or model architecture" - The recitation of "for processing a microscope image" merely identifies an intended field of use/technological environment in which the abstract idea is applied, without specifying any particular technical means for processing that image; this cannot integrate the exception into a practical application, nor does it add significantly more (see MPEP 2106.05(h)).
“the method comprising: launching a learning process including adjusting values of the trainable internal model parameters-values successively during the learning process depending on training data processed with the trainable internal model parameters;” -- The limitation recites a method that comprises launching a learning process that includes adjusting parameters depending on training data with the internal model parameters, and it is considered mere data observation. The limitation is considered to be an insignificant, extra-solution activity, and cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under step 2B, the act of storing/and retrieving information in memory (training system), is a well-understood, routine, and conventional activity (WURC), and cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)(iv).
“repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process which uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs;” -- Analogous to claim 1 limitations, it is seen as an insignificant, extra-solution activity that cannot be integrated to a practical application (see MPEP 2106.05(g)). Further, under Step 2B, the limitation is also a well-understood, routine, and conventional activity (WURC) that cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“feeding the two learning progressions, including the training learning progression and additionally the validation learning progression, to a verification model during the learning process” - This is mere field-of-use limitation reciting transmission of data to a generic model, and does not integrate the exception into a practical application, nor add significantly more (see MPEP 2106.05(h)).
Thus, claim 2 is non-patent eligible.
Regarding claim 3,
Step 1: Claim 3 depends on claim 2, which is deemed as a process. Claim 3 satisfies Step 1.
Step 2A Prong 1
“during an ongoing training of the machine learning model before a predetermined stopping criterion of the training is reached.” -- This limitation is directed to ongoing training of a ML model before a pre-set threshold of training has been reached, which can be done by observation, evaluation, and judgement of the human mind, and therefore is considered a mental process.
Step 2A Prong 2 and step 2B:
“The method as defined in claim 2, wherein the training learning progression and the validation learning progression are fed to the verification model” -- This limitation recites a training system that further comprises a verification model that will be fed into the training of the learning and validation progression. This limitation is considered to be mere limiting to a particular field of use/particular environment (type of model, progression taking place within technology), and therefore cannot be integrated to a practical application, not provide significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 3 is non-patent eligible.
Regarding claim 4,
Step 1: Claim 4 depends from claim 3, which depends from claim 2, a process. Therefore, claim 4 satisfies Step 1.
Step 2A Prong 1:
“The method as defined in claim 3, further comprising automatically deciding whether to continue the ongoing training of the machine learning model or abort the ongoing training of the machine learning model before the predetermined stopping criterion of the training is reached based on the quality assessment of the learning process of the machine learning model” - This limitation is directed to making a judgment as to whether to continue or abort a process based on an evaluative assessment, which is a decision capable of being made in the human mind using observation, evaluation, and judgment, and thus is a mental process.
There are no steps to be evaluated Step 2A Prong 2 and Step 2B.
Thus, claim 4 is non-patent eligible. Claim 20 is analogous to claim 4, and thus will face the same rejection as above.
Regarding claim 5,
Step 1: Claim 5 is dependent on claim 2, which is dependent on claim 1, which is deemed as a process. Claim 4 satisfies Step 1.
Step 2A Prong 1:
“which is trained to generate the quality assessment as output from the training learning progression and the validation learning progression as input data.” -- This limitation is directed to training that will involve doing an evaluation on performance to be the output of the learning progression that is done on input data, which can be performed using evaluation as judgement, and therefore is seen as a mental process.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 2, wherein the verification model comprises a verification machine learning model” -- This limitation recites a method where a verification model will further comprise a verification machine learning model. This is recited in a high level of generality and is seen as mere instructions to apply generically onto a computer (ML model) , therefore the imitation cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 5 is non-patent eligible.
Regarding claim 6,
Step 1: Claim 6 is dependent on claim 5, which is dependent on claim 2, which is deemed as a process. Claim 6 satisfies Step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 5, wherein the verification machine learning model is trained by an unsupervised learning process, in which a plurality of training learning progressions and associated validation learning progressions are used as verification machine learning model training data, or wherein the verification machine learning model is trained by a supervised learning process, in which a plurality of training learning progressions and associated validation learning progressions with a predetermined quality assessment are used as verification machine learning model training data.” -- This limitation recites unsupervised and supervised learning at a high level of generality; therefore it amounts to mere instructions to apply a judicial exception (which is the quality assessment= evaluation and judgment) on a computer, and cannot be integrated to a practical application, not provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 6 is non-patent eligible.
Regarding claim 7,
Step 1: Claim 7 is dependent on claim 5, which is dependent on claim 2, which is deemed as a process. Claim 7 satisfies Step 1.
Step 2A Prong 1:
There are no elements to be evaluated under step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 5, wherein the training learning progression and the validation learning progression are respectively fed to the verification machine learning model as a sequence of quality measure values;” -- This limitation recites a training system that further comprises a verification model that will be fed into the training of the learning and validation progression. This limitation is considered to be mere limiting to a particular field of use/particular environment (type of model, progression taking place within technology), and therefore cannot be integrated to a practical application, not provide significantly more than the judicial exception (see MPEP 2106.05(h)).
“and wherein the verification machine learning model comprises a recurrent neural network.” -- This limitation recites mere instructions that a verification ML model (computer) will comprise (instruct to apply) a recurrent NN, which is considered to a generic statement of application to a computer, and is not a practical application nor able to provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 7 is non-patent eligible.
Regarding claim 8,
Step 1: Claim 8 is dependent on claim 5, which is dependent on claim 2, which is deemed as a process. Claim 8 satisfies Step 1.
Step 2A Prong 1:
There are no elements to be evaluated under step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 5, wherein the training learning progression and the validation learning progression are respectively fed to the verification machine learning model as a sequence of quality measure values;” -- This limitation recites a training system that further comprises a verification model that will be fed into the training of the learning and validation progression. This limitation is considered to be mere limiting to a particular field of use/particular environment (type of model, progression taking place within technology), and therefore cannot be integrated to a practical application, not provide significantly more than the judicial exception (see MPEP 2106.05(h)).
“and wherein the verification machine learning model comprises a convolutional neural network.” -- This limitation recites mere instructions that a verification ML model (computer) will comprise (instruct to apply) a convolutional NN, which is considered to a generic statement of application to a computer, and is not a practical application nor able to provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 8 is non-patent eligible.
Regarding claim 9,
Step 1: Claim 9 is dependent on claim 2, which is deemed as a process. Claim 9 satisfies Step 1.
Step 2A Prong 1:
“The method as defined in claim 2, wherein the verification model takes one or more of the following factors into account for the quality assessment: …number of epochs after which the training learning progression or validation learning progression saturates, and a value of the quality measures during saturation; - divergence of the training learning progression or validation learning progression; - initial fluctuations in the training learning progression and validation learning progression and subsequent monotonous training learning progression and validation learning progression” -- This limitation is directed to a method for which multiple factors are listed for the quality assessment, which involves jumps in the learning and validation progression, epochs after progression saturation, and a value of quality that is made after saturation, which are all directed to the use of a mathematical operation and involve known mathematical concepts known in the art (calculating differences, divergence in progression). All these factors are ultimately done to judge the quality, which amounts to evaluation and judgment/mental.
“-jumps in the training learning progression or validation learning progression; - whether an optimum of the values of the trainable internal model parameters at which a quality measure is below a predetermined limit value is reached early.” -- This limitation is directed to jumps in trained data and assessing if trainable, internal parameter values at quality measure are below a predetermined value, which can be performed with observation, evaluation, and judgement of the human mind or given with pen and paper, therefore, it is a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 9 is non-patent eligible.
Regarding claim 10,
Step 1: Claim 9 is dependent on claim 2, which is deemed as a process. Claim 9 satisfies Step 1.
Step 2A Prong 1:
“The method as defined in claim 2, wherein the quality assessment comprises a suggestion for a modification of training parameters during the ongoing learning process or for a new learning process to be initiated.” -- This limitation is directed to a suggestion of modifying parameters during a current or learning process. This limitation can be performed using observation, evaluation, and judgement of the human mind to suggest a modification to a learning process, therefore it is considered to be a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 10 is non-patent eligible.
Regarding claim 11,
Step 1: Claim 11 is dependent on claim 10, and claim 10 is dependent on claim 2, which is deemed as a process. Claim 11 satisfies Step 1.
Step 2A Prong 1:
“The method as defined in claim 10, wherein the modification of training parameters comprises at least one of: a modification of a learning rate and a modification of a set number of epochs” -- This limitation is directed to a modification of learning rate and a set number of epochs that are done. Modifying learning rates and number of epochs involves the use of a mathematical concept, and therefore is seen as a mathematical concept/calculation. (see [0076] of instant application).
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 11 is non-patent eligible.
Regarding claim 12,
Step 1: Claim 12 is dependent on claim 10, and claim 10 is dependent on claim 2, which is deemed as a process. Claim 12 satisfies Step 1.
Step 2A Prong 1:
There are no elements to be evaluated under step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 10, wherein, in the event that the quality assessment assumes a local optimum of the values of the trainable internal model parameters, the modification of training parameters comprises a one-time or repeated increase of a learning rate in order to escape the local optimum of the values of the trainable internal model parameters.” -- This limitation recites a method where local optimum of parameter values is assumed and modifying parameter values consists of a repetitive or non-repetitive increasing of learning rate as an attempt to resolve local optimum. Merely changing data values is considered to be an insignificant, extra solution activity, and therefore cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, the limitation, especially with the performance of repetitive calculations, is a well-understood, routine and conventional activity in the field of machine learning, and therefore cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Thus, claim 12 is non-patent eligible.
Regarding claim 13,
Step 1: Claim 13 is dependent on claim 10, and claim 10 is dependent on claim 2, which is deemed as a process. Claim 13 satisfies Step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 10, wherein the modification of training parameters comprises a modification of the values of the trainable internal model parameters;” -- This limitation recites a method of mere instructions (generically explained) to comprise (apply) a modification of parameter values, which cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
“wherein the verification machine learning model also receives to this end, in addition to the training learning progression and the validation learning progression, associated values of the trainable internal model parameters of the machine learning model as inputs.” -- This limitation recites receiving data (associated model parameters) as inputs for the verification machine learning model, which would involve mere data gathering, and is seen to be an insignificant, extra-solution activity, and cannot be integrated to a practical application. Furthermore, under step 2B, receiving data over memory/computer is considered to be a well-understood, routine and conventional activity (WURC), and cannot be significantly more than the judicial exception (see MPEP 2106.05(d)(II).
Thus, claim 13 is non-patent eligible.
Regarding claim 14,
Step 1: Claim 14 is dependent on claim 13, and claim 13 is dependent on claim 10, and claim 10 is dependent on claim 2, which is deemed as a process. Claim 14 satisfies Step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 13, wherein the verification machine learning model comprises a neural network trained by a supervised learning process, in which training data comprises a plurality of training learning progressions, validation learning progressions and associated values of the trainable internal model parameters, as well as modifications of the values of the trainable internal model parameters as target data or wherein the verification machine learning model comprises a neural network trained by a reinforcement learning method,” -- This limitation recites a verification machine model that comprises mere instructions to apply a trained NN by a learning process, for which the trained data will further comprise a group of learning and validation learning progressions as well as parameter values, and modified model parameter values to be target data. The limitation is mere instructions to apply the concept of training and modifying different values and trained data onto a computer, which cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
“in which a reinforcement learning agent learns using a predefined training environment” -- This limitation recites reinforcement learning using a predefined training environment, which is seen as mere limiting to a particular field of use (ML) and particular environment, which cannot be integrated to a practical application, nor provide significantly more than the judicial exception, (see MPEP 2106.05(h)).
“for an input comprising a training learning progression, a validation learning progression and associated values of the trainable internal model parameters, how to modify the model parameter values of the trainable internal model parameters in order to optimize the quality assessment.” -- This limitation recites inputs to learning progression and parameter values and how to modify the values, which is directed to mere data gathering, and is considered an insignificant, extra-solution activity that cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under step 2B, the act of receiving data to be manipulated is a well-understood, routine and conventional activity (WURC), and cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II).
Thus, claim 14 is non-patent eligible.
Regarding claim 15,
Step 1: Claim 15 is dependent on claim 2, and claim 2 is deemed a process. Claim 15 satisfies step 1.
There are no elements to be evaluated under step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 2, wherein the training learning progression and the validation learning progression are input into a prediction machine learning model that is trained to predict future progressions of an input training learning progression and validation learning progression from the input training learning progression and validation learning progression and to add the predicted future progressions onto the input training learning progression and validation learning progression, wherein the training learning progression and validation learning progression supplemented by the prediction are output to a user or to the verification model.” -- This limitation recites a method of inputting progressions to a prediction ML model that will be trained to output predict future progressions with a prediction to be outputted to either the user or the model itself, and then add them to the trained learning and validation progressions. This limitation is directed to mere data gathering and manipulation of data, for which is considered to be an insignificant, extra-solution activity, and cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under step 2B, the act of transmitting data over a network (ML model) is a well-understood, routine, and conventional activity (WURC) in the field of ML, and therefore does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II).
Thus, claim 15 is non-patent eligible.
Regarding claim 16,
Step 1: Claim 16 is dependent on claim 2, and claim 2 is deemed a process. Claim 16 satisfies step 1.
Step 2A Prong 1:
“The method as defined in claim 2, wherein the verification machine learning model is configured to conduct an anomaly detection in order to determine deviations from typical training progression” -- This limitation is directed to a method of conduction anomaly detections to determine deviations of the training progression, which is considered to be a mental process that can be performed in the human mind using observation, evaluation, and judgement.
Step 2A Prong 2 and Step 2B:
“wherein the verification machine learning model for the anomaly detection is designed as an autoencoder trained with training learning progressions and validation learning progressions that do not contain any anomalies.” -- This limitation recites a verification ML model will be instructed to apply trained progressions that don’t have anomaly detections, which is recited in a high level of generality, and thus cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 16 is non-patent eligible.
Regarding claim 17,
Step 1: Claim 17 is dependent on claim 2, and claim 2 is deemed a process. Claim 17 satisfies step 1.
Step 2A Prong 1:
“define assessment criteria for the generation of the quality assessment depending on contextual data, wherein the contextual data relate to the machine learning model or the training of the machine learning model, wherein the contextual data comprise information regarding one or more of the following aspects: - type of a learning method, wherein a distinction is made at least between supervised training, unsupervised training, reinforcement learning and a use of adversarial networks; - type of a task of the machine learning model, wherein a distinction is made at least between a classification, segmentation, detection or regression; - architecture of the machine learning model; - values of hyperparameters of the machine learning model; - a type of the training/validation data.” -- This limitation is directed to defining assessment criteria for the ML model based on data (contextual data), which can all be performed using evaluation and observation with the human mind, and therefore is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“The method as defined in claim 2, wherein the verification model is designed to… wherein the contextual data comprise information regarding one or more of the following aspects: - type of a learning method, wherein a distinction is made at least between supervised training, unsupervised training, reinforcement learning and a use of adversarial networks; - type of a task of the machine learning model, wherein a distinction is made at least between a classification, segmentation, detection or regression; - architecture of the machine learning model; - values of hyperparameters of the machine learning model; - a type of the training/validation data.” -- This limitation recites a verification model and its mere instructions to be applied onto the computer at a high level of generality, and cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 17 is non-patent eligible.
Regarding claim 18,
Step 1: Claim 18 is analogous to method claim 2, and thus satisfy step 1 as claim 2 did under the category of process.
There are no elements to be evaluated under step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“A non-transitory computer-readable medium storing a computer program with commands that, when executed by a computer, cause the execution of the method defined in claim 2.” -- This limitation recites that the non-transitory CRM that stores a computer with commands that will cause the execution of the method defined in claim 2. The limitation does not amount to no more than mere instructions to apply onto a computer, and thus it cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 18 is non-patent eligible.
Regarding claim 19,
Step 1: The claim is directed to a method, which falls under the category of a process. The claim satisfies step 1.
Step 2A Prong 1:
“calculating, during the learning process, at least one quality measure for respectively current values of the trainable internal model parameters” -- The limitation is directed to calculating a quality measure for current model parameter values. Under broadest reasonable interpretation, the limitation recites a process that can be completed in the human mind using evaluation, observation, and judgment (with the aid of pen and paper) to complete, and thus it is directed to a mental process.
“forming at least one training learning progression from the quality measures of different epochs” -- The limitation is directed to forming training progression from quality measures of different epochs (period of time). The limitation is directed to a process that can be performed in the human mind using evaluation, observation, and judgement, and thus the limitation is directed to a mental process.
“wherein the verification machine learning model is trained by a supervised learning process, in which a plurality of training learning progressions with predetermined quality assessments of the training learning progressions are used as verification machine learning model training data.” -- The limitation is directed to training a machine learning model through a plurality of training learning progressions with predetermined quality assessments as training data. Training a model using predetermined quality assessment is a process that can be done using evaluation, and judgment, therefore it is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“A method for monitoring a learning process of a machine learning model including a generative adversarial network, wherein the machine learning model includes trainable internal model parameters for processing input training data to calculate a model output,” -- The limitation recites a method of monitoring learning of a machine learning model that will further include a GAN, and includes model parameters for input processing training data to calculate the model output. The limitation does not amount to no more than mere further limits to a field of use/environment, and it cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(h)).
“the method comprising launching a learning process using training data to adjust values of the trainable internal model parameters of the machine learning model;”-- This limitation recites a method to launch a learning process that involves adjusting trainable parameter values based/depending on gathered and manipulated data. Mere data manipulation is considered to be an insignificant, extra-solution activity, and cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under step 2B, the act of storing/and retrieving information in memory (training system), is a well-understood, routine, and conventional activity (WURC), and cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)(iv)).
“the quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from an input to the machine learning model;” -- The limitation is directed to the quality measure indicated a quality for a calculated result with the current model parameters, wherein it will be further limited to not be utilized in the learning process to adjust the model parameter values, but instead using the validation data quality measure. The limitation amounts to no more than further limiting to a field of use/environment, and does not integrate to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(h)).
“repeating calculating the quality measure in different epochs of the learning process;” -- The limitation is directed to repetitive calculation of the training and validation data quality measure in different epochs. Repetitive calculation is an insignificant, extra-solution activity that cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, the act of repetitive calculation is also a well-understood, routine, and conventional activity (WURC), and cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“inputting the at least one training learning progression into a verification model during the learning process” -- The limitation is directed to inputting training progression into a verification model during the learning process of the model. Inputting/outputting gathered data is an insignificant, extra-solution activity, and it cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, the limitation is also considered to be a well-understood, routine, and conventional activity, that cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“wherein the verification machine learning model is trained by an unsupervised learning process, in which a plurality of training learning progressions are used as verification machine learning model training data.” -- The limitation recites to training a model by unsupervised learning process, which entails using a plurality of training learning progressions as the training data. The limitation is directed to mere instructions to apply onto a computer, and it cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
“such that the training learning progressions are input into the verification machine learning model and deviations of the verification machine learning model's outputs to the predetermined quality assessments of the training learning progressions are used to update parameters of the verification machine learning model.” -- The limitation recites that the learning progressions are input to a model and that deviations of the outputs to the quality assessments are used to update parameters of the model. The limitation is directed to an insignificant, extra-solution activity that cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, the act of updating parameters and input/output relations over a model is a well-understood, routine, and conventional activity (WURC) that cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Thus, claim 19 is non-patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this
Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not
identically disclosed as set forth in section 102, if the differences between the claimed invention and the
prior art are such that the claimed invention as a whole would have been obvious before the effective filing
date of the claimed invention to a person having ordinary skill in the art to which the claimed invention
pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are
summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1 is rejected under 35 U.S.C. 103 as being unpatentable over US11927738B2, by Moore et. al. (referred herein as Moore in view of US20200372342A1, by Nair et. al. (referred herein as Nair) in view of NPL reference “Early Stopping | but when?”, by Perchelt et. al. (referred herein as Perchelt).
Regarding claim 1, Moore teaches:
A microscopy system comprising: at least one microscope, which comprises at least one camera for capturing a microscope image and a computing device, wherein the computing device comprises an imaging processing program for processing the microscope image using a machine learning model, ([Moore, col 12, lines 5-20, col 26, lines 15-17], “initiating movement of the camera and lighting apparatus for sample illumination and image capture… a plurality of images may be captured…the captured image from the main camera includes two copies of the sample…assessing the sample using feature extraction and/or one or more machine and/or deep learning models… the RANSAC method is an iterative method to estimate parameters of a mathematical model”, wherein the examiner interprets “the camera…image capture” to be the same as at least one camera capturing a microscope image, and “assessing the sample using…machine and/or deep learning models” to be the same as a computing device comprising an imaging processing program for processing the microscope image using a machine learning model because they are both directed to capturing microscope images with an imaging device and processing those images using machine learning–based analysis executed by a computing device.)
Moore does not teach wherein the machine learning model includes a set of trainable internal model parameters for processing input training data to calculate a model output, the trainable internal model parameters being distinct from hyperparameters that define training settings or model architecture; and a training system for carrying out a learning process of the machine learning model, wherein the training system is configured for: adjusting values of the trainable internal model parameters values of the machine learning model successively in the learning process depending on training data processed with the trainable internal model parameters; calculating a training data quality measure for current values of the trainable internal model parameters values during the learning process, the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data; and calculating a validation data quality measure, in addition to the training data quality measure, in addition to the training data quality measure, for the current values of the trainable internal model parameters during the learning process, the validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from validation data during the learning process, wherein the validation data is not used in the learning process to adjust the values of the trainable internal model parameters and instead the validation data is only used for calculating the validation data quality measure; repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process which uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs; and forming two learning progressions, including forming a training learning progression from successively determined training data quality measures and forming, in addition to the training learning progression, a validation learning progression from successively determined validation data quality measures, feeding the two learning progressions, including the training learning progression and additionally the validation learning progression, to a verification model during the learning process; and the verification model generating a quality assessment of the learning process of the machine learning model, the quality assessment depending on both the training learning progression and the validation learning progression.
Nair teaches:
wherein the machine learning model includes a set of trainable internal model parameters for processing input training data to calculate a model output, ([Nair, [0003] “the output of each neuron may be computed by function of the (typically weighted) sum of its inputs. NN links or edges typically have a weight that is adjusted as learning proceeds,” wherein the examiner interprets “weight that is adjusted as learning proceeds” to be the same as “trainable internal model parameters” because they are both directed to internal neural-network values that are adjusted during learning and affect the model output calculated from input data).
the trainable internal model parameters being distinct from hyperparameters that define training settings or model architecture, ([Nair, [0005-0006] “Hyperparameters may be parameters that cannot be directly optimized as a function of the model's loss, and may be determined using a part of the data that has not been used for training (e.g. a validation or hold out set). NN hyperparameters may describe an overall architecture of the NN such as number and structure of layers…A NN may also be described by non-hyperparameter parameters such as weights of links or functions that link layers of a NN, which may be adjusted during learning or training,” wherein the examiner interprets hyperparameters describing the number and structure of layers to be the same as “hyperparameters that define training settings or model architecture” because they are both directed to configuration information for the neural-network architecture or training process, and wherein the examiner interprets non-hyperparameter weights adjusted during learning or training to be the same as “trainable internal model parameters” because they are both directed to learned internal values distinct from hyperparameters).
and a training system for carrying out a learning process of the machine learning model, wherein the training system is configured for: adjusting values of the trainable internal model parameters of the machine learning model successively in the learning process depending on training data processed with the trainable internal model parameters, ([Nair, [0008]] “in the forward pass the NN is presented with an input and produces an output…generates update gradients which are used to alter the weights at the links or edges”, AND [Nair, 0004] “Machine learning algorithms, using e.g. neural network (NN) or connectionist systems, are trained or learn by iteratively adjusting model parameters to optimize an objective function, also known as a loss function. The goal of training a model may be to determine a set of hyperparameters, and/or weights of links or functions connecting neurons, and an optimization procedure that minimizes the training loss and allows the model to achieve generalization on unseen data.”, wherein the examiner interprets update gradients used to alter weights during forward/backward passes to be the same as adjusting values of trainable internal model parameters successively in the learning process because they are both directed to repeatedly modifying neural-network weights based on processed training input and resulting error/loss).
repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process which uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs, ([Nair, [0012] “An epoch may be a set of batches representing the entire training set; thus an epoch may represent a number of forward/backward passes, each resulting in weight adjustment…A NN may be trained in multiple epochs, each epoch using the same training data as the last,” AND ([Nair, [0016] “over a series of NN training epochs, where in each epoch the NN undergoes training and a loss is computed,” AND [0047] “Other of different ways of calculating a series of loss data, and defining which loss values to use, may be used. In some embodiments, learning may not be organized by batch or epoch” AND [0048] “In some embodiments, for any model in the set of models used, responsible for making a prediction based on the nth epoch of data”, wherein the examiner interprets multiple epochs with forward/backward passes and weight adjustment to be the same as using the same set of trainable internal model parameters across epochs because they are both directed to repeatedly training the same neural network over epochs while changing the internal weight values. Furthermore, the examiner interprets loss computed in each epoch to be the same as calculating a quality measure in different epochs because they are both directed to determining model-performance or error information at successive epoch-based stages of the learning process).
feeding the two learning progressions, including the training learning progression and additionally the validation learning progression, to a verification model during the learning process, ([Nair, [0047] “Time series features used as input to a model may be loss values or loss functions, or validation loss functions from training or experiments. Typically, loss data or time series data is the loss recorded for the last loss in the last batch in each epoch over a series of epochs, and thus such data is organized by epoch.”, wherein the examiner interprets “Time series features used as input to a model” to be the same as feeding learning progression information to a verification model because they are both directed to providing epoch-ordered training or validation performance/loss data to a model during the learning process).
and the verification model generating a quality assessment of the learning process of the machine learning model, ([Nair, [0016] “determining, using a model (or one or more models; models may include sub models), a probability of improvement in the loss of the NN” AND [0009] “Instead, during learning the NN automatically generates identifying characteristics from the learning material that it processes.” wherein the examiner interprets determining a probability of improvement in loss to be the same as generating a quality assessment of the learning process because they are both directed to evaluating whether the ongoing neural-network training process is likely to improve sufficiently, and wherein the examiner interprets stopping training based on that probability and generating identifying characteristics by automatically from past processed material to be the same as using the quality assessment to determine the status of the learning process because both are directed to model-based assessment of whether training should continue).
and the verification model generating a quality assessment of the learning process of the machine learning model, ([Nair, 0036] “Embodiments may incorporate a predictive model into an early stopping algorithm that is responsible for stopping a NN training process at the point at or as soon as the minimum predicted validation loss is achieved. An embodiment may terminate training based on a model, such as a prediction provided by a learning curve model,” wherein the examiner interprets “predictive model” and “learning curve model” to be the same as “verification model” because they are both directed to a model that evaluates training-progress information, and wherein the examiner interprets minimum predicted validation loss and terminating training based on the model to be the same as generating a quality assessment of the learning process because they are both directed to assessing the quality or future usefulness of continued training based on predicted loss behavior).
Nair does not teach calculating a training data quality measure for current values of the trainable internal model parameters during the learning process, the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data; and calculating a validation data quality measure, in addition to the training data quality measure, for the current values of the trainable internal model parameters during the learning process, the validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from validation data during the learning process, wherein the validation data is not used in the learning process to adjust the values of the trainable internal model parameters and instead the validation data is only used for calculating the validation data quality measure; forming two learning progressions, including forming a training learning progression from successively determined training data quality measures and forming, in addition to the training learning progression, a validation learning progression from successively determined validation data quality measures; and the quality assessment depending on both the training learning progression and the validation learning progression.
Prechelt teaches:
calculating a training data quality measure for current values of the trainable internal model parameters during the learning process, the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data; and calculating a validation data quality measure, in addition to the training data quality measure, for the current values of the trainable internal model parameters during the learning process, the validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from validation data during the learning process, ([Prechelt, page 3] “Etr(t), the training set error” and “Eva(t), the validation error,” AND [Prechelt, page 6] “The training data is further subdivided into a training set of examples used to adjust the network weights and a validation set of examples used to estimate network performance during training as required by the stopping criteria. The validation set is never used for weight adjustment.”, wherein the examiner interprets “training set error” to be the same as “training data quality measure” because they are both directed to a performance/error measure calculated on training data, and wherein the examiner interprets “validation error” to be the same as “validation data quality measure” because they are both directed to a performance/error measure calculated on validation data).
wherein the validation data is not used in the learning process to adjust the values of the trainable internal model parameters and instead the validation data is only used for calculating the validation data quality measure, ([Prechelt, page 2] “It is claimed to show the evolution over time of the per-example error on the training set and on a validation set not used for training (the training error curve and the validation error curve)…Train only on the training set and evaluate the per-example error on the validation set once in a while,” wherein the examiner interprets a validation set not used for training to be the same as validation data not used to adjust trainable internal model parameters because both are directed to excluding validation data from the model-parameter/weight updating process, and wherein the examiner interprets evaluating error on the validation set to be the same as using validation data only for calculating the validation data quality measure because both are directed to using validation data for performance evaluation rather than model-parameter adjustment).
forming two learning progressions, including forming a training learning progression from successively determined training data quality measures and forming, in addition to the training learning progression, a validation learning progression from successively determined validation data quality measures; ([Prechelt, page 2] “the training error curve and the validation error curve,” AND [Prechelt, page 13] “The jagged form of the validation error curve during phase II arises because neither bias nor variance change monotonically, let alone smoothly. The bias error component may change abruptly because training algorithms never per form gradient descent, but take nite steps in parameter space that sometimes have severe results. The observed variance error component may change abruptly because, first, the validation set error is only an estimate of the actual generalization error and, second, the effect of a parameter change maybe very different in different parts of parameter space.” wherein the examiner interprets “training error curve…the validation set error is only an estimate of the actual generalization error and, second, the effect of a parameter change maybe very different in different parts of parameter space” to be the same as “training learning progression from successively determined training data quality measures” because they are both directed to a curve/progression of training-set error or performance values over time, and wherein the examiner interprets “validation error curve” to be the same as “validation learning progression from successively determined validation data quality measures” because they are both directed to a curve/progression of validation-set error or performance values over time).
the quality assessment depending on both the training learning progression and the validation learning progression, ([Prechelt, page 3, 5] “when the training error still decreases quickly, generalization losses have higher chance to be ‘repaired’” and “use the quotient of generalization loss and progress,” wherein the examiner interprets using training-error progress together with generalization/validation loss in a stopping criterion to be the same as the quality assessment depending on both the training learning progression and the validation learning progression because both are directed to assessing whether continued training is useful based on both training-progress behavior and validation/generalization-error behavior).
Moore, Nair, Prechelt, and the instant application are analogous art because they are all directed to machine-learning model training or machine-learning model assessment systems in which model performance is evaluated using training data, validation data, quality measures, and/or learning-progress information to assess or improve the learning process.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the microscopy system disclosed by Moore to include the neural-network training and predictive early-stopping model disclosed by Nair. One would be motivated to do so to efficiently train the machine-learning model used in Moore’s microscopy-image analysis system and to reduce unnecessary training by stopping when further improvement is unlikely, as suggested by Nair (Nair, [0005-0006] “Hyperparameters may be parameters that cannot be directly optimized as a function of the model's loss, and may be determined using a part of the data that has not been used for training (e.g. a validation or hold out set). NN hyperparameters may describe an overall architecture of the NN such as number and structure of layers…A NN may also be described by non-hyperparameter parameters such as weights of links or functions that link layers of a NN, which may be adjusted during learning or training,”)
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the microscopy system disclosed by Moore, as modified by Nair, to include the training-error and validation-error monitoring disclosed by Prechelt. One would be motivated to do so to effectively detect overfitting and improve generalization during training of the machine-learning image-processing model, as suggested by Prechelt ([Prechelt, page 2] “It is claimed to show the evolution over time of the per-example error on the training set and on a validation set not used for training (the training error curve and the validation error curve)…Train only on the training set and evaluate the per-example error on the validation set once in a while,”).
Claims 2-10, 13, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over NPL reference “Hyperparameter optimization” by Feurer et. al. (referred herein as Feurer) in view of Moore in view of US20220051049A1, by Wang et. al. (referred herein as Wang) in view of Prechelt further in view of Nair.
Regarding claim 2, Feurer teaches:
A method for monitoring a learning process of a machine learning model, wherein the machine learning model includes a set of trainable internal model parameters for processing input training data to calculate a model output, the trainable internal model parameters being distinct from hyperparameters that define training settings or model architecture, the method comprising: ([Feurer, Chapter 4, sec. 4.2, page 83] “A learning algorithm A maps a set of training data points di to such a function, which is often expressed via a vector of model parameters … Most learning algorithms A further expose hyperparameters, which change the way the learning algorithm itself works. For example, hyperparameters are used to describe a description-length penalty, the number of neurons in a hidden layer, the number of data points that a leaf in a decision tree must contain to be eligible for splitting”, wherein the examiner interprets “A learning algorithm A maps a set of training data points” and “expressed via a vector of model parameters” to be the same as “a set of trainable internal model parameters for processing input training data to calculate a model output” because they are both directed to internal parameters of a machine learning model derived from processing training data to produce an output function. The examiner further interprets “hyperparameters, which change the way the learning algorithm itself works” and “the number of neurons in a hidden layer” to be the same as “hyperparameters that define training settings or model architecture” because they are both directed to external configuration values that control how the learning algorithm operates and determine model architecture).
adjusting values of the trainable internal model parameters, ([Feurer, Chapter 2, sec. 2.4.2, page 49] “During training, they use their own weights as additional input data and observe their own errors to learn how to modify these weights in response to the new task at hand. The updating of the weights is defined in a parametric form that is differentiable end-to-end and can jointly optimize both the network and training algorithm using gradient descent”, wherein the examiner interprets “use their own weights,”, “learn how to modify these weights,” and “The updating of the weights” to be the same as adjusting values of the trainable internal model parameters because they are both directed to modifying learned model weights during training).
Feurer does not teach for processing a microscope image.
Moore teaches for processing a microscope image, ([Moore, col. 12, lines 5-20; col. 26, lines 15-17] “initiating movement of the camera and lighting apparatus for sample illumination and image capture” and “assessing the sample using feature extraction and/or one or more machine and/or deep learning models”, wherein the examiner interprets “sample illumination and image capture” to be the same as processing a microscope image because they are both directed to capturing image data of a biological/microscopy sample, and wherein the examiner interprets “assessing the sample using feature extraction and/or one or more machine and/or deep learning models” to be the same as a machine learning model for processing the microscope image because they are both directed to machine-learning-based analysis of captured sample image data).
Feurer and Moore do not teach launching a learning process including adjusting values of the trainable internal model parameters successively during the learning process depending on training data processed with the trainable internal model parameters.
Wang teaches launching a learning process including adjusting values of the trainable internal model parameters successively during the learning process depending on training data processed with the trainable internal model parameters, ([Wang, 0029] “The server computer 102, via the HOM 122 at block 220, iteratively runs a training portion of the preprocessed GTD 106 through each of the pipelines 402 with the hyperparameter sets generated by the PGM 116. The HOM 122 assess performance of each pipeline 402 iteratively, comparing performance for each of the associated hyperparameter sets.”, wherein the examiner interprets “iteratively runs a training portion” and “assess performance of each pipeline 402 iteratively” to be the same as launching a learning process including adjusting values successively during the learning process because they are both directed to repeatedly processing training data through machine-learning pipelines and evaluating performance during the learning process).
Wang does not teach calculating a training data quality measure, in addition to the training data quality measure, for current values of the trainable internal model parameters during the learning process, the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data; and calculating a validation data quality measure for the current values of the trainable internal model parameters during the learning process, the validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from validation data during the learning process, wherein the validation data is not used in the learning process to adjust the values of the trainable internal model parameters and instead the validation data is only used for calculating the validation data quality measure; repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process which uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs; forming two learning progressions, including forming a training learning progression from successively determined training data quality measures; and forming, in addition to the training learning progression, a validation learning progression from successively determined validation data quality measures; and the quality assessment depending on both the training learning progression and the validation learning progression.
Prechelt teaches:
calculating a training data quality measure, in addition to the training data quality measure, for current values of the trainable internal model parameters during the learning process, the training data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from the training data; and calculating a validation data quality measure for the current values of the trainable internal model parameters during the learning process, the validation data quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from validation data during the learning process, ([Prechelt, page 3] “Etr(t), the training set error” and “Eva(t), the validation error,” AND [Prechelt, page 6] “The training data is further subdivided into a training set of examples used to adjust the network weights and a validation set of examples used to estimate network performance during training as required by the stopping criteria. The validation set is never used for weight adjustment.”, wherein the examiner interprets “training set error” to be the same as “training data quality measure” because they are both directed to a performance/error measure calculated on training data, and wherein the examiner interprets “validation error” to be the same as “validation data quality measure” because they are both directed to a performance/error measure calculated on validation data).
wherein the validation data is not used in the learning process to adjust the values of the trainable internal model parameters and instead the validation data is only used for calculating the validation data quality measure, ([Prechelt, page 2] “It is claimed to show the evolution over time of the per-example error on the training set and on a validation set not used for training (the training error curve and the validation error curve)…Train only on the training set and evaluate the per-example error on the validation set once in a while,” wherein the examiner interprets a validation set not used for training to be the same as validation data not used to adjust trainable internal model parameters because both are directed to excluding validation data from the model-parameter/weight updating process, and wherein the examiner interprets evaluating error on the validation set to be the same as using validation data only for calculating the validation data quality measure because both are directed to using validation data for performance evaluation rather than model-parameter adjustment).
the quality assessment depending on both the training learning progression and the validation learning progression, ([Prechelt, pages 3,5] “when the training error still decreases quickly, generalization losses have higher chance to be ‘repaired’” and “use the quotient of generalization loss and progress”, wherein the examiner interprets using training-error progress together with generalization/validation loss in a stopping criterion to be the same as the quality assessment depending on both the training learning progression and the validation learning progression because they are both directed to assessing whether continued training remains useful based on both training-progress behavior and validation/generalization-error behavior).
Prechelt does not teach repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process which uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs,
forming two learning progressions, including forming a training learning progression from successively determined training data quality measures; and forming, in addition to the training learning progression, a validation learning progression from successively determined validation data quality measures,
feeding the two learning progressions, including the training learning progression and additionally the validation learning progression, to a verification model during the learning process; and the verification model generating a quality assessment of the learning process of the machine learning model.
Nair teaches:
repeating calculating the training data quality measure and the validation data quality measure in different epochs of the learning process which uses the same set of trainable internal model parameters such that training data quality measures and validation data quality measures are determined for different values of the same set of trainable internal model parameters across the epochs ([Nair, [0012] “An epoch may be a set of batches representing the entire training set; thus an epoch may represent a number of forward/backward passes, each resulting in weight adjustment…A NN may be trained in multiple epochs, each epoch using the same training data as the last,” AND ([Nair, [0016] “over a series of NN training epochs, where in each epoch the NN undergoes training and a loss is computed,” AND [0047] “Other of different ways of calculating a series of loss data, and defining which loss values to use, may be used. In some embodiments, learning may not be organized by batch or epoch” AND [0048] “In some embodiments, for any model in the set of models used, responsible for making a prediction based on the nth epoch of data”, wherein the examiner interprets multiple epochs with forward/backward passes and weight adjustment to be the same as using the same set of trainable internal model parameters across epochs because they are both directed to repeatedly training the same neural network over epochs while changing the internal weight values. Furthermore, the examiner interprets loss computed in each epoch to be the same as calculating a quality measure in different epochs because they are both directed to determining model-performance or error information at successive epoch-based stages of the learning process).
feeding the two learning progressions, including the training learning progression and additionally the validation learning progression, to a verification model during the learning process, ([Nair, [0047] “Time series features used as input to a model may be loss values or loss functions, or validation loss functions from training or experiments. Typically, loss data or time series data is the loss recorded for the last loss in the last batch in each epoch over a series of epochs, and thus such data is organized by epoch.”, wherein the examiner interprets “Time series features used as input to a model” to be the same as feeding learning progression information to a verification model because they are both directed to providing epoch-ordered training or validation performance/loss data to a model during the learning process).
and the verification model generating a quality assessment of the learning process of the machine learning model, ([Nair, 0036] “Embodiments may incorporate a predictive model into an early stopping algorithm that is responsible for stopping a NN training process at the point at or as soon as the minimum predicted validation loss is achieved. An embodiment may terminate training based on a model, such as a prediction provided by a learning curve model,” wherein the examiner interprets “predictive model” and “learning curve model” to be the same as “verification model” because they are both directed to a model that evaluates training-progress information, and wherein the examiner interprets minimum predicted validation loss and terminating training based on the model to be the same as generating a quality assessment of the learning process because they are both directed to assessing the quality or future usefulness of continued training based on predicted loss behavior).
Feurer, Moore, Wang, Prechelt, Nair, and the instant application are analogous art because they are all directed to machine-learning model training or machine-learning model assessment systems in which model performance is evaluated using training data, validation data, quality measures, and/or learning-progress information to assess or improve a learning process.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process disclosed by Feurer to include the iterative training process disclosed by Wang. One would be motivated to do so to develop a more efficient training execution framework that repeatedly processes training data and evaluates performance during learning, as suggested by Wang ([Wang, [0029] “iteratively runs a training portion of the preprocessed GTD 106 through each of the pipelines 402” and “assess performance of each pipeline 402 iteratively”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method disclosed by Feurer and Wang to include the microscope-image processing environment disclosed by Moore. One would be motivated to do so to effectively apply machine-learning training monitoring to biological/microscopy image-processing models, as suggested by Moore ([Moore, col. 12, lines 5-20; col. 26, lines 15-17] “initiating movement of the camera and lighting apparatus for sample illumination and image capture…assessing the sample using feature extraction and/or one or more machine and/or deep learning models”)
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method disclosed by Feurer Wang, and Moore, to include the training-error and validation-error monitoring disclosed by Prechelt. One would be motivated to do so to effectively distinguish training-set performance from validation-set performance during model training, thereby detecting overfitting and improving generalization, as suggested by Prechelt ([Pretchelt, page 2] “It is claimed to show the evolution over time of the per-example error on the training set and on a validation set not used for training (the training error curve and the validation error curve)…Train only on the training set and evaluate the per-example error on the validation set once in a while,”)
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method disclosed by Feurer and Wang, as modified by Moore and Prechelt, to include the predictive early-stopping model disclosed by Nair. One would be motivated to do so to efficiently determine during training whether further training is likely to improve the model and to reduce unnecessary training, as suggested by Nair’s teaching that predictive early stopping may improve neural-network training efficiency and save compute or processor time. Claim 20 is analogous to claim 2/3, and shares elements of previous claims, however mostly is analogous to claim 2, thus the rejection above can apply to both claims.
Regarding claim 3, Feurer, Moore, Wang, Prechelt, and Nair teaches The method as defined in claim 2, (see rejection of claim 2).
Wang further teaches wherein the training learning progression and the validation learning progression are fed to the verification model during an ongoing training of the machine learning model before a predetermined stopping criterion of the training is reached. ([Wang, 0026] “It is preferred that pipeline generation occur iteratively, in conjunction with decision block 214, with the server computer 102 iteratively deciding after generating each pipeline 402, whether more pipelines are needed (e.g. , pipeline target quantity has been met or a user has indicated that a current set of pipelines is deemed sufficient).” AND [Wang, 0005] “…ranking pipeline performance in accordance therewith ; select a candidate pipe line in accordance, at least in part , with said pipeline performance ranking”, wherein the examiner interprets “iteratively deciding after generating each pipeline” and “ranking pipeline performance in accordance therewith” to be the same as “training learning progression and validation learning progression being fed to the verification model during an ongoing training,” as both are directed to evaluating intermediate results of a process and determining subsequent actions during the iterative development phase prior to a stopping criterion being met (like the target quantity, performance, or user satisfaction.)
Feurer, Moore, Wang, Prechelt, Nair, and the instant application are analogous art because they are all directed to monitoring, evaluating, and controlling machine-learning training processes based on training-progress information, validation information, performance rankings, and/or stopping determinations during model development.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 2 disclosed by Feurer, Moore, Prechelt, and Nair to include the iterative pipeline-decision process disclosed by Wang. One would be motivated to do so to effectively evaluate intermediate training/pipeline results during ongoing model development and determine whether further training or pipeline generation remains necessary before a predetermined stopping or sufficiency condition is reached, as suggested by Wang ([Wang, 0026] “the server computer 102 iteratively deciding after generating each pipeline 402, whether more pipelines are needed” and [Wang, 0005] “ranking pipeline performance in accordance therewith; select a candidate pipeline in accordance, at least in part, with said pipeline performance ranking”).
Regarding claim 4, Feurer, Moore, Wang, Prechelt, and Nair teaches The method as defined in claim 3, (see rejection of claim 3).
Nair further teaches automatically deciding whether to continue the ongoing training of the machine learning model or abort the ongoing training of the machine learning model before the predetermined stopping criterion of the training is reached based on the quality assessment of the learning process of the machine learning model, ([Nair, [0016] “over a series of NN training epochs, where in each epoch the NN undergoes training and a loss is computed, determining, using a model...a probability of improvement in the loss of the NN…If the probability is less than a threshold, or a wait value is greater than a wait threshold, training may be stopped” AND [Nair, [0032] “a training module or unit may call an early stopping module or unit...and receive instructions to stop training or not stop training” AND [Nair, [0036] “Embodiments may incorporate a predictive model into an early stopping algorithm that is responsible for stopping a NN training process at the point at or as soon as the minimum predicted validation loss is achieved. An embodiment may terminate training based on a model, such as a prediction provided by a learning curve model” AND [Nair, [0080] “return a stop / True value or a continue training / False value”, wherein the examiner interprets determining a probability of improvement in loss using a model to be the same as basing the decision on the quality assessment of the learning process because both are directed to evaluating whether ongoing training is likely to improve, wherein the examiner interprets the early stopping module returning stop/True or continue-training/False to be the same as automatically deciding whether to abort or continue ongoing training because both are directed to a computer-implemented model-based decision controlling whether neural-network training stops or proceeds, and wherein the examiner interprets stopping based on a predictive/learning-curve model at or near the predicted minimum validation loss to be the same as aborting the ongoing training before a predetermined stopping criterion is reached because both are directed to stopping training early based on a model-based quality assessment rather than waiting for the ordinary completion of training).
Feurer, Moore, Wang, Prechelt, Nair, and the instant application are analogous art because they are all directed to machine-learning model training, monitoring, performance assessment, learning-progress evaluation, and/or stopping decisions during a learning process.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 3 disclosed by Feurer, Moore, Wang, Prechelt, and Nair to include automatic early-stopping decision disclosed by Nair. One would be motivated to do so to automatically determine during training whether the learning process is likely to improve, and to avoid unnecessary further training when the quality assessment indicates that additional training is unlikely to provide sufficient improvement, as suggested by Nair ([Nair, [0016] “over a series of NN training epochs, where in each epoch the NN undergoes training and a loss is computed, determining, using a model...a probability of improvement in the loss of the NN…If the probability is less than a threshold, or a wait value is greater than a wait threshold, training may be stopped” AND [Nair, [0032] “a training module or unit may call an early stopping module or unit...and receive instructions to stop training or not stop training” AND [Nair, [0036] “Embodiments may incorporate a predictive model into an early stopping algorithm that is responsible for stopping a NN training process at the point at or as soon as the minimum predicted validation loss is achieved. An embodiment may terminate training based on a model, such as a prediction provided by a learning curve model” AND [Nair, [0080] “return a stop / True value or a continue training / False value”). Claim 20 is analogous to claim 4, and thus faces the same rejection.
Regarding claim 5, Feurer, Moore, Wang, Prechelt, and Nair teaches The method as defined in claim 2, (see rejection of claim 2).
Wang further teaches wherein the verification model comprises a verification machine learning model, which is trained to generate the quality assessment as output from the training learning progression and the validation learning progression as input data. ([Wang, 0031-0032] “The server computer 102 presents to a user for feedback, via the Pipeline Validation User Interface (PVUI) 128...results of applying the pipelines 402...Pipeline performance details are included to provide a high degree of interpretability” AND [Wang, Abstract], “A computer automatically selects a machine learning model pipeline using a meta-learning machine learning model.”, wherein the examiner interprets “results of applying the pipelines” and “pipeline performance details” to be the same as “the training learning progression and validation learning progression” and “meta-learning machine learning model” to be the same as “verification model,” as both are directed to evaluating and interpreting model performance during training to guide adjustments before completion of the process. Meta-learning refers to “learning to learn” and involves using ML to optimize or guide the training of another ML model.)
Feurer, Moore, Wang, Prechelt, Nair, and the instant application are analogous art because they are all directed to machine-learning model training and evaluation systems that use performance data, learning-progress information, validation information, or meta-learning models to assess and improve machine-learning model selection, training, or verification.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 2 disclosed by Feurer, Moore, Prechelt, and Nair to include the meta-learning machine-learning model and pipeline-performance evaluation disclosed by Wang. One would be motivated to do so to effectively generate a model-based assessment of training or pipeline quality from performance information and improve interpretability and confidence in the selected machine-learning model, as suggested by Wang ([Wang, Abstract] “A computer automatically selects a machine learning model pipeline using a meta-learning machine learning model” and [Wang, 0031-0032] “Pipeline performance details are included to provide a high degree of interpretability”).
Regarding claim 6, Feurer, Moore, Wang, Prechelt, and Nair teaches The method as defined in claim 5, (see rejection of claim 5).
Wang further teaches wherein the verification machine learning model is trained by an unsupervised learning process, in which a plurality of training learning progressions and associated validation learning progressions are used as verification machine learning model training data, or wherein the verification machine learning model is trained by a supervised learning process, in which a plurality of training learning progressions and associated validation learning progressions with a predetermined quality assessment are used as verification machine learning model training data. ([Wang, 0004] “The computer applies each of the pipelines with the associated preferred set of hyperparameters to score the favored data features of an appropriately preprocessed set of ground truth data and ranks the pipeline performance accordingly . The computer selects a candidate pipeline in accordance , at least in part , with the pipeline performance ranking .”, wherein the examiner interprets “applying each pipeline with preferred hyperparameters to score favored data features” and “ranking the pipeline performance” to be the same as “training a supervised learning verification model using training learning progressions and validation learning progressions with a predetermined quality assessment,” as both are directed to utilizing predefined criteria (ground truth data and hyperparameters) for guiding model selection and performance assessment during training.)
Feurer, Moore, Wang, Prechelt, Nair, and the instant application are analogous art because they are all directed to training, evaluating, and selecting machine-learning models using training data, validation data, ground-truth data, performance information, and/or learned model-selection criteria.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 5 disclosed by Feurer, Moore, Prechelt, Nair, and Wang to include the scoring and ranking of pipeline performance using ground-truth data and preferred hyperparameters disclosed by Wang. One would be motivated to do so to effectively train or configure a verification machine-learning model to assess model quality using known performance criteria and to select a candidate model/pipeline based on ranked performance, as suggested by Wang ([Wang, 0004] “applies each of the pipelines with the associated preferred set of hyperparameters to score the favored data features” and “ranks the pipeline performance accordingly”).
Regarding claim 7, Feurer, Moore, Wang, Prechelt, and Nair teaches The method as defined in claim 5, (see rejection of claim 5).
Wang further teaches wherein the training learning progression and the validation learning progression are respectively fed to the verification machine learning model as a sequence of quality measure values; and wherein the verification machine learning model comprises a recurrent neural network. ([Wang, Abstract] “A computer automatically selects a machine learning model pipeline using a meta-learning machine learning model ... The computer generates hyper parameter sets for the pipelines. The computer applies preprocessing routines to ground truth data to generate a group of preprocessed sets of said ground truth data and ranks hyperparameter set performance for each pipeline to establish a preferred set of hyperparameters for each of pipeline . The computer selects favored data features and applies each of the pipelines , with associated sets of preferred hyperparameters , to score the favored data features of the preprocessed ground truth data. The computer ranks pipeline performance and selects a candidate pipeline according to the ranking .”, wherein the examiner interprets “ranking hyperparameter set performance for each pipeline” and “scoring favored data features of preprocessed ground truth data” to be the same as “feeding training learning progression and validation learning progression as a sequence of quality measure values to the verification machine learning model,” as both are directed to the iterative process of assessing performance through sequential evaluation. The machine learning model used for ranking and evaluation in Wang et al. could include a recurrent neural network (RNN), as such models are well-suited for processing sequential data and quality measure values in iterative processes.)
Feurer, Moore, Wang, Prechelt, Nair, and the instant application are analogous art because they are all directed to machine-learning training and evaluation systems that process sequential training-performance information, validation-performance information, and/or iterative model-performance values to assess or optimize a learning process.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 5 disclosed by Feurer, Moore, Prechelt, Nair, and Wang to feed training and validation learning progressions as a sequence of quality measure values into the verification machine-learning model. One would be motivated to do so to effectively process iterative performance values generated during training and validation and use those sequential values to assess model quality over time, as suggested by Wang ([Wang, Abstract] “ranks hyperparameter set performance for each pipeline” and “The computer ranks pipeline performance and selects a candidate pipeline according to the ranking”).
Regarding claim 8, Feurer, Moore, Wang, Prechelt, and Nair teaches The method as defined in claim 5, (see rejection of claim 5).
Moore further teaches wherein the training learning progression and the validation learning progression are fed to the verification machine learning model as graphs in the form of image data; and wherein the verification machine learning model comprises a convolutional neural network; ([Moore, col 15, lines 3-10] “In other embodiments, the message may include an image of the sample for user verification of the computer-assisted evaluation. In some embodiments, one or more images of the sample may be digitally stained using pre-trained deep learning models and displayed to the viewer.” AND [Moore, col 26, lines 60-64] “In one embodiment of image stitching, a deep convolutional neural network (CNN) may be used to generate homographic estimates of images”), wherein the examiner interprets “pre-trained deep learning models for digitally staining and displaying images for user verification” and “a deep convolutional neural network (CNN) used for homographic estimates of images” to be the same as “a verification machine learning model” and the same as feeding “to the verification machine learning model … image data,” as both are directed to using machine learning processes (via CNNs) to analyze and output transformed image data for evaluation and further processing.)
Feurer, Moore, Wang, Prechelt, Nair, and the instant application are analogous art because they are all directed to monitoring and evaluating machine learning processes via verification models to improve system performance.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 5 disclosed by Feurer, Moore, Wang, Prechelt, and Nair to include the “use of a deep convolutional neural network (CNN) for generating homographic estimates of images” taught by Moore. One would be motivated to do so to efficiently enhance the verification and validation process using images during training, as suggested by Moore (see Moore, [col 26, lines 60-64] quote above.)
Regarding claim 9, Feurer, Moore, Wang, Prechelt, and Nair teaches The method as defined in claim 2, (see rejection of claim 2).Wang further teaches wherein the verification model takes one or more of the following factors into account for the quality assessment: jumps in the training learning progression or validation learning progression; number of epochs after which the training learning progression or validation learning progression saturates, and a value of the quality measures during saturation; ([Wang, page 4] “The server computer 102, via the HOM 122 at block 220, iteratively runs a training portion of the preprocessed GTD 106 through each of the pipelines 402 with the hyperparameter sets generated by the PGM 116. The HOM 122 assess performance of each pipeline 402 iteratively, comparing performance for each of the associated hyperparameter sets. The HOM 122 determines favored hyperparameter sets for each pipeline 402.”, wherein the examiner interprets “iteratively running training data through pipelines and assessing performance for different hyperparameter sets” and “determining favored hyperparameter sets” to be the same as “taking into account jumps in the training learning progression or validation learning progression, the number of epochs before saturation, and the value of quality measures during saturation,” as both are directed to evaluating the progression of training performance and determining optimal conditions for improved quality.)
difference between the training learning progression and the validation learning progression; divergence of the training learning progression or validation learning progression; ([Wang, [0020], “The server computer 102 includes a Pipeline Validation User Interface ( PVUI ) 128 that allows a user to examine pipeline execution results to correct , remove selected pipelines, and otherwise give input regarding pipeline performance to increase result interpretability and user confidence. The server computer 102 includes an Ensemble Assembly Module (EAM) 130 that combines multiple pipelines into a cooperative bundle. The server computer 102 also includes an Ensemble Pipeline Application Module 132 applies the pipelines in the ensemble to provided data 106 which can indicate whether multiple pipelines provide results that agree.”, wherein the examiner interprets PVUI “allows a user to examine pipeline execution results to correct” to be the same as examining the “difference between the training learning progression and the validation learning progression” as both require examining differences between actual and validation results which is subsequently used to make changes to the machine learning models.)initial fluctuations in the training learning progression and validation learning progression and subsequent monotonous training learning progression and validation learning progression; (Wang, [0019], “The server computer 102 is in communication with a source of Ground Truth Data (GTD) 106 useful for training and validating the models to be selected by the system 100. According to aspects of the present invention, the GTD 106 is text - based and can reflect many different kind of a of information.”, wherein the examiner interprets “in communication with a source of GTD” to be the same as initial and subsequent progression comparison of training versus validating results for information between their fluctuations.)Moore further teaches whether an optimum of the values of trainable internal model parameter values at which a quality measure is below a predetermined limit value is reached early. ([Moore, col 24, lines 5-23] “The iterative algorithm first finds a translational shift S1131 and corrects the translational shift of the object or speckled pattern that was projected onto the sensor S1132...This may then be used to update the wavefunction for a high-resolution magnitude projection S1135...The system may continue to shift the diffuser or object until the defined convergence criteria is met at block S1138.”, wherein the examiner interprets “iterative adjustments to find and correct translational shifts and updating projections until convergence criteria are met” to be the same as detecting the optimum state of the model parameter values once the “convergence criteria” are met, where “convergence criteria” is the same as “a predetermined limit value is reached”.)Feurer, Moore, Wang, Prechelt, Nair, and the instant application are analogous art because they are all directed to methods for monitoring and optimizing machine learning processes based on quality measures derived from training and validation progressions.It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model disclosed by Feurer, Moore, Wang, Prechelt, and Nair to include the “iterative adjustments to correct translational shifts and updating projections until convergence criteria are met” disclosed by Moore. One would be motivated to do so to effectively enhance the accuracy and robustness of training evaluation processes, as suggested by Moore (see Moore, [col 24, lines 5-23] quote above.)
Regarding claim 10, Feurer, Moore, Wang, Prechelt, and Nair teaches the method as defined in claim 2 (see rejection of claim 2).
Moore teaches wherein the quality assessment comprises a suggestion for a modification of training parameters during the ongoing learning process or for a new learning process to be initiated. ([Moore, col 32, lines 31-35] “In some embodiments, the trained models may be static and therefore unchanged with the addition of new data. In some embodiments, the models may continue to evolve to improve accuracy through additional data and user feedback.”, wherein the examiner interprets “models continuing to evolve to improve accuracy through additional data and user feedback” to be the same as “a suggestion for a modification of training parameters during the ongoing learning process or for a new learning process to be initiated,” as both are directed to adapting or modifying the training process based on ongoing input to achieve improved model performance.)
Feurer, Moore, Wang, Prechelt, Nair, and the instant application are analogous art because they are all directed to methods for monitoring and improving the learning process of a machine learning model based on evaluation metrics derived during training.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 2 disclosed by Feurer, Moore, Wang, Prechelt, and Nair to include the process to “improve accuracy” disclosed by Moore. One would be motivated to do so to effectively enhance the adaptability and accuracy of machine learning model training processes, as suggested by Moore (see Moore, [col 32, lines 31-35] quote above.)
Regarding claim 13, Feurer, Moore, Wang, Prechelt, and Nair teaches the method as defined in claim 10 (see rejection of claim 10).Moore further teaches wherein the modification of training parameters comprises a modification of the model parameter values; wherein the verification machine learning model also receives to this end, in addition to the training learning progression and the validation learning progression, associated model parameter values of the machine learning model as inputs; ([Moore, col 20, lines 15-20], “The method 200 may include one or more preprocessing methods S210, followed by image loading S220, and receiving as an input one or more system parameters S230”, wherein the examiner interprets “receiving as an input one or more system parameters” to be the same as “the verification machine learning model receiving associated model parameter values as inputs,” as both are directed to utilizing parameter inputs to enhance the processing or evaluation of a system.)Feurer, Moore, Wang, Prechelt, Nair, and the instant application are analogous art because they are all directed to methods for modifying training parameters and incorporating associated data to improve the learning process of a machine learning model.It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 10 disclosed by Feurer, Moore, Wang, Prechelt, and Nair to include the “receiving as an input one or more system parameters” disclosed by Moore. One would be motivated to do so to efficiently improve the adaptability and precision of the training process by utilizing associated parameters to refine evaluations, as suggested by Moore (see Moore, [col 20, lines 15-20] quote above.)
Regarding claim 18, the majority of the claim is analogous to claim 2, so it will inherit the rejection set forth for claim 2 above. Below, From Wang, is where it maps to A non-transitory computer-readable medium storing a computer program commands. ([0035] “The method 100, for example, may be embodied in a program 1060, including program instructions, embodied on a computer readable storage device, or computer readable storage medium, for example, generally referred to as memory 1030 and more specifically, computer readable storage medium 1050.”, where in the examiner interprets “program instructions, embodied on a computer readable storage device, or computer readable storage medium, for example, generally referred to as memory 1030 and more specifically, computer readable storage medium” to be the same as computer-readable medium, not transitory as clarified in [0045], that has stored computer programs and commands.). So the combination of Feurer and Wang embody claim 2.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Feurer in view of Moore in view of Wang in view of Prechelt in view of Nair further in view of Colley et. al, US20210090694A1 (referred herein as Colley).
Regarding claim 11, Feurer, Moore, Wang, Prechelt, and Nair teaches the method as defined in claim 10 (see rejection of claim 10).Feurer, Moore, Wang, Prechelt, and Nair do not teach wherein the modification of training parameters comprises at least one of: a modification of a learning rate and a modification of a set number of epochs;
Colley teaches wherein the modification of training parameters comprises at least one of: a modification of a learning rate and a modification of a set number of epochs; ([Colley, [1861] “We continued training until we observed diminishing improvement in validation accuracy. We found that Inception-v3 and ResNet-50 performed competitively, achieving max accuracy of 92.2% and 91.44% respectively over 40 epochs.”, wherein the examiner interprets “continuing training until diminishing improvement in validation accuracy is observed and achieving maximum accuracy over a set number of epochs” to be the same as “a modification of a set number of epochs,” as both are directed to adjusting the training process to optimize performance by considering changes in validation metrics over defined training intervals.)Feurer, Moore, Wang, Prechelt, Nair, Colley, and the instant application are analogous art because they are all directed to methods for monitoring, evaluating, and optimizing the training process of machine learning models by modifying training parameters to improve performance.It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model disclosed by Feurer, Moore, Wang, Prechelt, and Nair to include the method for “continuation of training” to maximize validation accuracy disclosed by Colley. One would be motivated to do so to effectively improve the optimization of training processes by identifying and addressing diminishing returns in accuracy, as suggested by Colley (see Colley, [1861] quote above.)
Claims 12 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Feurer in view of Moore in view of Wang in view Prechelt in view of Nair in further view of NPL reference “Hyperparameter optimization” by Hutter et. al (referred herein as Hutter).
Regarding claim 12, Feurer, Moore, Wang, Prechelt, and Nair teaches the method as defined in claim 10 (see rejection of claim 10).
Feurer, Moore, Wang, Prechelt, and Nair do not teach wherein, in the event that the quality assessment assumes a local optimum of the values of the trainable internal model parameters, the modification of training parameters comprises a one-time or repeated increase of a learning rate in order to escape the local optimum of the values of the trainable internal model parameters;.
Hutter teaches wherein, in the event that the quality assessment assumes a local optimum of the values of the trainable internal model parameters, the modification of training parameters comprises a one-time or repeated increase of a learning rate in order to escape the local optimum of the values of the trainable internal model parameters;. ([Hutter, page 85, sec 4.3] “In order to select its next hyperparameter configuration λ using model ML, SMBO uses a so-called acquisition function a ML : → R, which uses the predictive distribution of model ML at arbitrary hyperparameter configurations λ ∈ to quantify (in closed form) how useful knowledge about λ would be. SMBO then simply maximizes this function over to select the most useful configuration λ to evaluate next. Several well-studied acquisition functions exist [18, 27, 29]; all aim to automatically trade off exploitation (locally optimizing hyperparameters in regions known to perform well)”, wherein the examiner interprets “automatically trading off exploitation by locally optimizing hyperparameters in regions known to perform well” to be the same as “identifying and escaping a local optimum of the model parameter values,” as both are directed to refining hyperparameter configurations to improve performance by overcoming suboptimal parameter values.)
Feurer, Moore, Wang, Prechelt, Nair, Hutter, and the instant application are analogous art because they are all directed to methods for monitoring, assessing, and optimizing training processes in machine learning models by analyzing and improving model parameter configurations during training.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 10 disclosed by Feurer, Moore, Wang, Prechelt, and Nair to include the process of “locally optimizing hyperparameters” taught by Hutter. One would be motivated to do so to effectively improve the adaptability and optimization of the training process, as suggested by Hutter (see Hutter, [page 85, sec 4.3] quote above.)
Regarding claim 14, Feurer, Moore, Wang, Prechelt, and Nair teach the method as defined in claim 13, (see rejection of claim 13, 10 and/or 2).
Moore further teaches wherein the verification machine learning model comprises a neural network trained by a supervised learning process, in which training data comprises a plurality of training learning progressions, ([Moore, col 32, lines 7-10], “In some embodiments, these features may be used as criteria to train classification machine learning or deep learning models in supervised learning.”, wherein the examiner interprets “learning models in supervised learning” to be the same as “model comprises a neural network trained by a supervised learning process”.)
Wang further teaches validation learning progressions and associated values of the trainable model parameters, as well as modifications of the values of the trainable internal model parameters as target data; ([Wang, [0020], “The server computer 102 includes a Pipeline Validation User Interface (PVUI) 128 that allows a user to examine pipeline execution results to correct, remove selected pipelines, and otherwise give input regarding pipeline performance to increase result interpretability and user confidence.”, wherein the examiner interprets “Validation User Interface…to examine pipeline execution results to correct .. pipelines” to be the same as “validation learning progressions … as well as modifications of the model parameter values”).
Feurer, Moore, Wang, Prechelt, Nair, and the instant application are analogous art, because they are all directed to a model comprising a trained network/ training data comprising learning progressions and trainable parameters.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 13 disclosed by Feurer, Moore, Wang, Prechelt, and Nair to include “examine pipeline execution results to correct, remove selected pipelines, and otherwise give input regarding pipeline performance to increase result interpretability and user confidence” disclosed by Wang. One would be motivated to edit/manipulate parameters of specific data to be used as validating learning progressions and modified/parameter values as suggested by Wang ([Wang, [0020], “The server computer 102 includes a Pipeline Validation User Interface (PVUI) 128 that allows a user to examine pipeline execution results to correct, remove selected pipelines, and otherwise give input regarding pipeline performance to increase result interpretability and user confidence.”)
Feurer, Moore, Wang, Prechelt, and Nair do not teach or wherein the verification machine learning model comprises a neural network trained by a reinforcement learning method, in which a reinforcement learning agent learns using a predefined training environment, for an input comprising a training learning progression, a validation learning progression and associated model parameter values, how to modify the model parameter values in order to optimize the quality assessment;.
Hutter teaches or wherein the verification machine learning model comprises a neural network trained by a reinforcement learning method, in which a reinforcement learning agent learns using a predefined training environment, for an input comprising a training learning progression, a validation learning progression and associated model parameter values, how to modify the model parameter values in order to optimize the quality assessment; ([Hutter, page 51, Sec 2.4.4] “Meta-learning is certainly not limited to (semi-)supervised tasks, and has been successfully applied to solve tasks as varied as reinforcement learning, active learning, density estimation and item recommendation. The base-learner may be unsupervised while the meta-learner is supervised, but other combinations are certainly possible as well.” AND “Duan et al. [39] propose an end-to-end reinforcement learning (RL) approach consisting of a task-specific fast RL algorithm which is guided by a general-purpose slow meta-RL algorithm. The tasks are interrelated Markov Decision Processes (MDPs). The meta-RL algorithm is modeled as an RNN, which receives the observations, actions, rewards and termination flags. The activations of the RNN store the state of the fast RL learner, and the RNN”s weights are learned by observing the performance of fast learners across tasks.”, wherein the examiner interprets “a reinforcement learning approach using a predefined training environment and inputs such as observations, actions, and rewards” for verification to be the same as “a reinforcement learning agent learning from inputs of training and validation learning progressions and associated model parameter values to modify model parameters to optimize quality assessment,” as both are directed to methods for training neural networks to optimize learning processes based on associated inputs, validation, and environments.)
Feurer, Moore, Wang, Prechelt, Nair, Hutter, and the instant application are analogous art because they are all directed to methods for training machine learning models using various learning paradigms (supervised and reinforcement) to optimize model performance.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model disclosed by Feurer, Moore, Wang, Prechelt, and Nair to include the process of meta-learning applied to supervised and reinforcement learning tasks taught by Hutter. One would be motivated to do so to effectively enhance the adaptability and optimization of the learning process, as suggested by Hutter (see Hutter, [page 51, Sec 2.4.4] quote above.)
Claim(s) 15-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feurer in view of Moore in view of Wang in view of Prechelt in view of Nair further in view of Hutter.
Regarding claim 15, Feurer, Moore, Wang, Prechelt, and Nair teaches the method as defined in claim 2 (see rejection of claim 2).
Feurer, Moore, Wang, Prechelt, and Nair do not teach wherein the training learning progression and the validation learning progression are input into a prediction machine learning model that is trained to predict future progressions of an input training learning progression and validation learning progression from the input training learning progression and validation learning progression and to add the predicted future progressions onto the input training learning progression and validation learning progression, wherein the training learning progression and validation learning progression supplemented by the prediction are output to a user or to the verification model;.
Hutter teaches wherein the training learning progression and the validation learning progression are input into a prediction machine learning model that is trained to predict future progressions of an input training learning progression and validation learning progression from the input training learning progression and validation learning progression and to add the predicted future progressions onto the input training learning progression and validation learning progression, wherein the training learning progression and validation learning progression supplemented by the prediction are output to a user or to the verification model; ([Hutter, page 18, sec 1.4.3] “Multi-task Bayesian optimization [147] uses a multi-task Gaussian process to model the performance of related tasks and to automatically learn the tasks” correlation 1 during the optimization process.”, wherein the examiner interprets “modeling the performance of related tasks and automatically learning their correlations during the optimization process” to be the same as “training a prediction machine learning model to predict future progressions of training and validation learning progressions based on their input values and adding predicted progressions to the original inputs,” as both are directed to leveraging machine learning to predict and enhance progressions of performance data based on correlations derived from prior tasks.)
Feurer, Moore, Wang, Prechelt, Nair, Hutter, and the instant application are analogous art because they are all directed to methods for predicting and optimizing the performance of machine learning models by analyzing and extending learning progressions using predictive methods.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model disclosed by Feurer, Moore, Wang, Prechelt, and Nair to include the processes to “model the performance of related tasks and automatically learn correlations during the optimization process” taught by Hutter. One would be motivated to do so to effectively enhance the prediction and extension of learning progressions by incorporating inter-task correlations, as suggested by Hutter (see Hutter, [page 18, sec 1.4.3] quote above.)
Regarding claim 16, Feurer, Moore, Wang, Prechelt, and Nair teaches the method as defined in claim 2 (see rejection of claim 2).
Feurer, Moore, Wang, Prechelt, and Nair do not teach wherein the verification machine learning model is configured to conduct an anomaly detection in order to determine deviations from typical training progressions, wherein the verification machine learning model for the anomaly detection is designed as an autoencoder trained with training learning progressions and validation learning progressions that do not contain any anomalies;.
Hutter teaches wherein the verification machine learning model is configured to conduct an anomaly detection in order to determine deviations from typical training progressions, wherein the verification machine learning model for the anomaly detection is designed as an autoencoder trained with training learning progressions and validation learning progressions that do not contain any anomalies; ([Hutter, page 49, sec 2.4.2] “An early meta-learning approach is to create recurrent neural networks (RNNs) able to modify their own weights. During training, they use their own weights as additional input data and observe their own errors to learn how to modify these weights in response to the new task at hand.”, wherein the examiner interprets “training recurrent neural networks with their own weights as additional input data and observing their own errors” to be the same as “training a verification machine learning model with training learning progressions and validation learning progressions that do not contain any anomalies,” as both are directed to training machine learning models to detect deviations by learning from typical, non-anomalous training data.)
Feurer, Moore, Wang, Prechelt, Nair, Hutter, and the instant application are analogous art because they are all directed to methods for training machine learning models to analyze and optimize training progressions.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model disclosed by Feurer, Moore, Wang, Prechelt, and Nair to include the use of training data that observes its own errors to “modify weights during training” disclosed by Hutter. One would be motivated to do so to efficiently enhance the ability of machine learning models to detect and address anomalies, as suggested by Hutter (see Hutter, [page 49, sec 2.4.2] quote above.)
Regarding claim 17, Feurer, Moore, Wang, Prechelt, and Nair teaches the method as defined in claim 2 (see rejection of claim 2).
Feurer, Moore, Wang, Prechelt, and Nair do not teach wherein the verification model is designed to define assessment criteria for the generation of the quality assessment depending on contextual data, wherein the contextual data relate to the machine learning model or the training of the machine learning model, wherein the contextual data comprise information regarding one or more of the following aspects: - type of a learning method, wherein a distinction is made at least between supervised training, unsupervised training, reinforcement learning and a use of adversarial networks; - type of a task of the machine learning model, wherein a distinction is made at least between a classification, segmentation, detection or regression; - architecture of the machine learning model; - values of hyperparameters of the machine learning model; - a type of the training/validation data;Hutter teaches:
wherein the verification model is designed to define assessment criteria for the generation of the quality assessment depending on contextual data, wherein the contextual data relate to the machine learning model or the training of the machine learning model, wherein the contextual data comprise information regarding one or more of the following aspects: (Hutter, page 106, “We performed optimization runs of up to 300 function evaluations searching the subset of the space depicted in Fig.5.1, and compared the quality of solution with specialized searches of specific classifier types (including best known classifiers).” wherein the examiner interprets “optimization runs of up to 300 function evaluations” and “compared the quality of solution” to be the same as “contextual data” that is “designed to define assessment”, respectively)
type of a learning method, wherein a distinction is made at least between supervised training, unsupervised training, reinforcement learning and a use of adversarial networks; ([Hutter, page 179] “matching algorithms to problems (which may include supervised, unsupervised, or reinforcement learning, or other settings), acquisition of new data (active learning, query learning, reinforcement learning, causal experimentation), management of large volumes of data including the creation of appropriately-sized and stratified training, validation, and test sets.”, wherein the examiner interprets “matching algorithms to problems (which may include supervised, unsupervised, or reinforcement learning, or other settings) … and managing training, validation, and test data sets” to be the same as type of a learning method .. supervised, unsupervised, reinforcement learning, and use of adversarial networks.)
type of a task of the machine learning model, wherein a distinction is made at least between a classification, segmentation, detection or regression; - architecture of the machine learning model; ([Hutter, page 73], “Notable first steps in this direction are applying Neural Architecture Search (NAS) to language modeling [74], music modeling [48], image restoration [58] and network compression [3]; applications to reinforcement learning, generative adversarial networks, semantic segmentation, or sensor fusion.”, wherein the examiner interprets “applications to reinforcement learning, generative adversarial networks, semantic segmentation” to be the same as “distinction is made at least between a classification, segmentation, detection, or regression” and Neural Architecture Search (NAS) to be the same as “architecture of the machine learning model”.)
values of hyperparameters of the machine learning model; - a type of the training/validation data; ([Hutter, page 5, sec 1.2] “the Combined Algorithm Selection and Hyperparameter Optimization (CASH) problem, in which the choice of a classification algorithm is modeled as a categorical variable, the algorithm hyperparameters are modeled as conditional hyperparameters.”, wherein the examiner interprets modeling classification algorithms as categorical variables and hyperparameters as conditional hyperparameters to be the same as “define assessment criteria for the generation of the quality assessment depending on contextual data” such as hyperparameters and learning methods, as both are directed to creating assessment criteria tailored to specific machine learning contexts to optimize quality assessment generation.)
Feurer, Moore, Wang, Prechelt, Nair, Hutter, and the instant application are analogous art because they are all directed to methods for optimizing machine learning models through contextual data and tailored assessment criteria.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for monitoring a learning process of a machine learning model disclosed by Feurer, Moore, Wang, Prechelt, and Nair to include the process of “matching algorithms to problems (which may include supervised, unsupervised, or reinforcement learning, or other settings)” taught by Hutter. One would be motivated to do so to effectively improve the adaptability and accuracy of quality assessments by incorporating diverse machine learning contexts and task types, as suggested by Hutter (see Hutter, [page 179] quote.)
Claim(s) 19 is rejected under 35 U.S.C. 103 as being unpatentable over Feurer in view of Guyon.
Regarding claim 19, Feurer teaches:
A method for monitoring a learning process of a machine learning model including a generative adversarial network, ([Feurer, Chapter 1, Section 1.4, p. 14] “First, we review methods which model an algorithm's learning curve during training and can stop the training procedure if adding further resources is predicted to not help” and “Learning curve extrapolation is used in the context of predictive termination [26], where a learning curve model is used to extrapolate a partially observed learning curve for a configuration, and the training process is stopped if the configuration is predicted to not reach the performance of the best model trained so far in the optimization process” and [Feurer, Chapter 2, Section 2.4.2, page 48] “Generative Adversarial Networks (GANs; Goodfellow et al., 2014) are generative models that learn to produce realistic samples. They consist of two neural networks: a generator and a discriminator”, wherein the examiner interprets “methods which model an algorithm's learning curve during training” and “extrapolate a partially observed learning curve” and “training process is stopped if the configuration is predicted to not reach the performance” to be the same as “a method for monitoring a learning process of a machine learning model” because they are both directed to a process that observes and evaluates the progression of model training to make determinations about the quality of the learning process. The examiner further interprets wherein the examiner interprets “Generative Adversarial Networks (GANs)” and “a generator and a discriminator” to be the same as “a machine learning model including a generative adversarial network” because they are both directed to the same type of machine learning architecture consisting of adversarial neural network components.)
wherein the machine learning model includes trainable internal model parameters for processing input training data to calculate a model output, the method comprising ([Feurer, Chapter 4, sec 4.2, page 83] “A learning algorithm A maps a set of training data points di to such a function, which is often expressed via a vector of model parameters”, wherein the examiner interprets “A learning algorithm A maps a set of training data points” and “expressed via a vector of model parameters” to be the same as “trainable internal model parameters for processing input training data to calculate a model output” because they are both directed to internal parameters of a machine learning model that are derived from processing training data to produce an output function.)
launching a learning process using training data to adjust values of the trainable internal model parameters of the machine learning model; ([Feurer, Chapter 2, Sec 2.4.2, page 49] “During training, they use their own weights as additional input data and observe their own errors to learn how to modify these weights in response to the new task at hand. The updating of the weights is defined in a parametric form that is differentiable end-to-end and can jointly optimize both the network and training algorithm using gradient descent”, wherein the examiner interprets “During training, they use their own weights” and “learn how to modify these weights” and “The updating of the weights” and “using gradient descent” to be the same as “to adjust values of the trainable internal model parameters of the machine learning model” because they are both directed to modifying learned model weights iteratively during the training procedure through optimization algorithms such as gradient descent.)
calculating, during the learning process, at least one quality measure for respectively current values of the trainable internal model parameters, [Feurer, Chapter 1, Section 1.4.1, p. 14] “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration”, wherein the examiner interprets “the performance of an iterative algorithm measured for each iteration” to be the same as “calculating, during the learning process, at least one quality measure for respectively current values of the trainable internal model parameters” because they are both directed to measuring a performance metric at successive iterations during training where the model parameters have different values at each iteration.)
repeating calculating the quality measure in different epochs of the learning process: ([Feurer, Chapter 1, Section 1.4.1, p. 14] “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration (or every i-th iteration if the calculation of the performance is expensive)”, wherein the examiner interprets “measured for each iteration” to be the same as “repeating calculating the quality measure in different epochs of the learning process” because they are both directed to repeatedly evaluating performance/quality values at successive stages of training.)
forming at least one training learning progression from the quality measures of different epochs; ([Feurer, Chapter 1, Section 1.4.1, p. 14] “Examples of learning curves are the performance of the same configuration trained on increasing dataset subsets, or the performance of an iterative algorithm measured for each iteration (or every i-th iteration if the calculation of the performance is expensive)”, wherein the examiner interprets “learning curves” and “the performance of an iterative algorithm measured for each iteration” to be the same as “forming at least one training learning progression from the quality measures of different epochs” because they are both directed to building a progression or curve from performance/quality values that are measured successively at different stages during the training process.)
inputting the at least one training learning progression into a verification model during the learning process; ([Feurer, Chapter 1, sec 1.4, p. 14] “First, we review methods which model an algorithm's learning curve during training and can stop the training procedure if adding further resources is predicted to not help”, wherein the examiner interprets “methods which model an algorithm's learning curve during training” to be the same as “inputting the at least one training learning progression into a verification model during the learning process” because they are both directed to a model/method operating during training that receives and processes observed learning curve information as input.)
wherein the verification model comprises a verification machine learning model trained to generate a quality assessment of the learning process of the machine learning model depending on the at least one training learning progression; ([Feurer, Chapter 1, Section 1.4.1, p. 14] “Learning curve extrapolation is used in the context of predictive termination [26], where a learning curve model is used to extrapolate a partially observed learning curve for a configuration, and the training process is stopped if the configuration is predicted to not reach the performance of the best model trained so far in the optimization process”, wherein the examiner interprets “learning curve model” to be the same as “a verification machine learning model” because they are both directed to a model that processes learning curve data, and wherein the examiner interprets “used to extrapolate a partially observed learning curve” and “training process is stopped if the configuration is predicted to not reach the performance of the best model trained so far” to be the same as “trained to generate a quality assessment of the learning process of the machine learning model depending on the at least one training learning progression” because they are both directed to a model that has been configured/trained to evaluate training quality based on observed learning progression data and produce a determination about that training process.)
wherein the verification machine learning model is trained by an unsupervised learning process, in which a plurality of training learning progressions are input into the verification machine learning model as verification machine learning model training data, or wherein the verification machine learning model is trained by a supervised learning process, in which a plurality of training learning progressions labelled with predetermined quality assessments of the training learning progressions are used as verification machine learning model training data such that the training learning progressions are input into the verification machine learning model and ([Feurer, Chapter 1, Section 1.4.1, page 15] “We train parametric learning curve models on learning curves observed in previous HPO runs. These models are then used to predict the performance of new configurations based on their partially observed learning curves”, wherein the examiner interprets “We train parametric learning curve models on learning curves observed in previous HPO runs” to be the same as “the verification machine learning model is trained” using “a plurality of training learning progressions” as “verification machine learning model training data” because they are both directed to training a model using previously observed learning curve data from multiple training runs.)
deviations of the verification machine learning model's outputs to the predetermined quality assessments of the training learning progressions are used to update parameters of the verification machine learning model. ([Feurer, Chapter 1, Section 1.4.1, page 15] “These models are then used to predict the performance of new configurations based on their partially observed learning curves”, wherein the examiner interprets “These models are then used to predict the performance of new configurations based on their partially observed learning curves” to be the same as the verification model generating quality assessments for new learning processes because they are both directed to applying a trained model to evaluate new training progressions, combined with [Feurer, Chapter 2, Section 2.4.2, page 49] “During training, they use their own weights as additional input data and observe their own errors to learn how to modify these weights in response to the new task at hand. The updating of the weights is defined in a parametric form that is differentiable end-to-end and can jointly optimize both the network and training algorithm using gradient descent”, wherein the examiner interprets “observe their own errors” and “learn how to modify these weights” and “The updating of the weights” to be the same as “deviations of the verification machine learning model's outputs” being “used to update parameters of the verification machine learning model” because they are both directed to adjusting model parameters based on observed errors during the training process.)
Feurer does not teach the quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from an input to the machine learning model;
Guyon teaches the quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from an input to the machine learning model; ([Guyon, col. 32, lines 10-23] “Four metrics of classifier quality were used. Each value is computed both on the training set with the leave-one-out method and on the test set”, wherein the examiner interprets “Four metrics of classifier quality” and “Each value is computed” to be the same as “the quality measure indicating a quality for a result calculated with the current values of the trainable internal model parameters from an input to the machine learning model” because they are both directed to quality metrics that evaluate how well the model performs with its current learned parameter values when processing input data.)
Feurer, Guyon, and the instant application are analogous art because they are all directed to monitoring a learning process of a machine learning model.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method claim 19 disclosed by Feurer to include the quality metrics disclosed by Guyon. One would be motivated to do so to effectively quantify and compare learning-process quality using explicit classifier-quality metrics computed on training and held-out test data for assessment of performance during learning, as suggested by Guyon (Guyon, [col. 32, lines 10-23] “Four metrics of classifier quality were used. Each value is computed both on the training set with the leave-one-out method and on the test set.”)
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DEVAN KAPOOR whose telephone number is (703)756-1434. The examiner can normally be reached Monday - Friday: 9:00AM - 5:00 PM EST (times may vary).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEVAN KAPOOR/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights