CTFR 17/469,075 CTFR 97128 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Response to Amendment (Submitted 3/5/2026) In regard to 103 rejections - On Page 10 , the applicant argues that the references fails to teach training two different models using the same training data and based on three different loss functions. Further within the context of amendment to the claim, the applicant argues that reference Brown does not disclose modifying the free parameters of inference model based on the plurality of differences generated via the output labels. Examiner’s Response Perhaps known to the POSITA that in a neural network inference model, free parameters refer to the learned weights and biases (or other internal variables) that the model uses to make predictions. These are the model parameters that are adjusted during training so that the model’s outputs match the expected results as closely as possible. The examiner first recognize that the reference “Brown” is a secondary reference”. The examiner submits that perhaps known to a POSITA that inference model uses the fixed values of parameters (the ones learned during training) to process new inputs and produce predictions. Modifying these free parameters in inference typically means changing these learned weights or biases in some way to alter the model’s behavior such as using adjust, modify, pruning weights, channels, or neurons to reduce model size and speed up inference The examiner rejects the applicant in regard to training different models and using same training data. The examiner notes that reference traches in [0074]” in at least one embodiment, each augmented audio datum from augmented content 306 has a different type of modification applied to a same audio datum from content 302. In at least one embodiment, modifications to video data (e.g., including both audio and images) from content input 302 includes all modifications listed above with respect to modifications to audio data while also including adjustments to frame rate, adding frames, removing frames, etc” and teaches in [0078] “ different type of model trained to perform a same task (e.g., classification) or otherwise another neural network that is different from a neural network being trained” The examiner submits that a model trained on different data or with a different architecture is generally considered a different model in ML Perhaps the training is performed on two different models (e.g., different architectures, algorithms, or parameterizations ) on the same audio dataset and the same task , then each training run is considered a separate training process, even if the data is identical. This is because the training process is defined by both the model structure and the training procedure and the training process shapes the model’s learned parameters and capabilities. The examiner submits that further reference “Brown” teaches in [0327]” in at least one embodiment, processing cluster array 2212 can include logic to execute processing tasks including filtering of video and/or audio data, performing modeling operations” and teaches in [ 0595] “ 21. A processor, comprising: [0596] one or more circuits to use one or more neural networks to generate data labels, wherein the one or more neural networks are trained based , at least in part, on a first version of training data and a modified version of the first version of training data” and teaches in [0597] “ 22. The processor of clause 21, wherein the one or more neural network is a convolutional neural network. ” In CONCLUSION , the examiner rejects claims 1-6, 8-13 and 1 5-19 under 103 as FINAL REJECTION . Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claims 1-4, 6, 8-11, 13, 15-17 an d 19 are rejected under 35 U.S.C. 103 as being unpatentable over Sina Mohseni et.al. (hereinafter Mohseni ) US 2021/0142160 A1, in view of Abel Karl Brown et.al. (hereinafter Brown ) US 2022/0101112 A1. in view of Maoqi Xu et.al. (hereinafter Xu ) US 11514515 B2. In regard to claim 1 : (Currently Amended) Mohsen i discloses: - train each of a plurality of models to output labels based on a first set of training data comprising a first set of plurality of features and a corresponding label; In [0106]: train one or more machine learning models or predict or infer information using one or more machine learning models according to one or more embodiments described herein in [0106]: in at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information in [0108]: Inference and/or train ing logic 915 are used to perform inferencing and/or train ing operations associated with one or more embodiments In [0096]: In at least one embodiment, semi-supervised learning may be used, which is a technique in which in training dataset 1002 includes a mix of label ed and un label ed data. In [0055]: In at least one embodiment, first set of output nodes 212 are classifier functions. In [0055]: In at least one embodiment, first set of output nodes 212 learn target label s from IND training set 206 during training. (BRI: target label is the “output label”) - input second sets of values of each of the plurality of features into each of the plurality of trained models to generate, for each of the second sets of values, a plurality of output labels; In [0106]: i n at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1100 by using weight parameters calculated through one or more training techniques described herein. in [ 0055]: In at least one embodiment, first set of output nodes 212 are classifier functions. In at least one embodiment, first set of output nodes 212 are output nodes for a different type of machine learning technique In [0055]: In at least one embodiment, first set of output nodes 212 learn target labels from IND training set 206 during training. in [0051]: In at least one embodiment, target label s are ground truth data that indicate what classifier functions are to infer. (BRI: A target label is an output label) - determine for each of the second set of values, pseudo-label based on the plurality of output labels generated for the second set of values In [0059]: In at least one embodiment, third set of data is unlabeled. In at least one embodiment, third set of data includes inputs that lack corresponding ground truth data (e.g., labels, annotations, or metadata) that indicate what second portion is to infer. In at least one embodiment, technique 300 includes assigning pseudo -labels to OOD training samples at block 306. In [0106]: i n at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1100 by using weight parameters calculated through one or more training techniques described herein. In [0062]: in at least one embodiment, second portion is trained to cluster unlabeled OOD samples using rejector functions in a manner such that first portion maintains in-distribution classification performance - determine a second set of training data comprising a second plurality of pairs, each of the second plurality of pairs comprising one of the second sets of values and a pseudo-label determined for the one of the second sets of values In [0060]: In at least one embodiment, first portion of neural network and second portion of neural network are trained with an automated technique. In at least one embodiment, first portion of neural network and second portion of neural network are trained using a machine learning (ML) approach to dynamically determine at least one of a ratio of IND samples to OOD samples, a learning rate, a dropout rate, and/or any other suitable parameters or hyperparameters in accordance with at least one performance metric. (BRI: the OOD sample has pseudo-labels. The mix of the OOD and IND is via ratio) In [0094]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, In [0094]: In at least one embodiment, untrained neural network 1006 is trained in a supervised manner processes inputs from training dataset 1002 and compares resulting outputs against a set of expected or desired outputs - train a first inference model and a second inference model having fewer free parameters than the first inference model to each output an inferred label based on the first set of training data and the second set of training data by: In [0054]: model architecture 200 includes a neural network 202 and training data sets 204. In at least one embodiment, training data sets 204 include an in- distribution (IND) training data set 206 and an out-of- distribution (OOD) training data set 208, In [0072]: performing other actions at block 510 includes leveraging an ensemble of multiple DNN blocks, In [0094]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, In [0051]: classifier functions learn target labels from a labeled in-distribution training set during a supervised learning technique, In [ 0094]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, In [0061]: a ratio of IND samples to OOD samples is progressively altered during training OOD detectors from a predetermined initial ratio to a predetermined final ratio, in [0060: dynamically determine at least one of a ratio of IND samples to OOD samples, a learning rate, a dropout rate, and/or any other suitable parameter s or hyper parameter s in accordance with at least one performance metric. (BRI: a setting that can be adjusted to fine-tune the model's predictions based on learning. The learning rate or a dropout rate is a hyperparameter learned by altering the ratio) reception of first labels output from the first inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; In [0106]: i n at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1100 by using weight parameters calculated through one or more training techniques described herein. in [0055]: at least one embodiment, first set of output nodes 212 learn target labels from IND training set 206 during training. In at least one embodiment, technique 500 reduces in-distribution classification error risk via selective classification for inputs near decision boundaries. In [0074]: FIG. 6 illustrates decision boundary scenarios 600, including a first scenario 602 and a second scenario 604, according to at least one embodiment . In [0074]: first dataset 610 includes data associated with a first category, second dataset 612 includes data associated with a second category, and third dataset 614 includes data associated with a third category. In [0058]: In at least one embodiment, second set of data is similar to first set of data within a first range. In [0058]: In at least one embodiment, within first range refers to second set of data includes elements labeled in same categories as elements in first set of data. reception of second labels output from the first inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; In [0106]: i n at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1100 by using weight parameters calculated through one or more training techniques described herein. Mohseni does not explicitly disclose: system comprising: a memory storing processor-executable program code; and a processing unit to execute the processor-executable program code to cause the system to: - input of a plurality of the first sets of values and a plurality of the second sets of values to the first inference model and to the second inference model; reception of first labels output from the first inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; reception of second labels output from the first inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; - determination of a first difference between the first labels output from the first inference model and the labels corresponding to the plurality of the first sets of values and the pseudo-labels determined for the plurality of the second sets of values - determination of a second difference between the first labels and the second labels and output from the first inference model based model a third difference between the second labels output from the second inference model and the labels corresponding to the plurality of the first sets of values and the pseudo-labels determined for the plurality of the second sets of values modification of the first inference model based on the first difference - and modification of the second inference model based on the second difference and the third difference. However, Brown discloses: A system comprising: a memory storing processor-executable program code; and a processing unit to execute the processor-executable program code to cause the system to: In [0020]; in [0045]; in [0050]; in [0079]; input of a plurality of the first sets of values and a plurality of the second sets of values to the first inference model and to the second inference model; in [0082]: In at least one embodiment, a system performing at least a part of process 500 includes executable code to update 508 a model being trained using training data, labels, augmented training data, and pseudo-labels. In at least one embodiment, an updated model is used to infer data labels from data input . in [0078]: FIG. 4 illustrates a diagram 400 of generating pseudo-labels 406 to train neural networks, according to at least one embodiment. In at least one embodiment, a model 408 is trained over one or more iterations 410, where at each round of training, weights of said model 408 are adjusted to cause said model 408 to make inference s that match labels and pseudo-labels of data used to train said model 408. In at least one embodiment, GPU determines, for a model 408 that is trained using training data, a set of weights. (BRI: model 408 is an inference model) reception of first labels output from the first inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; in [0060]: in at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ense mble consistency regularization in [0113]: machine learning models or predict or infer information using one or more machine learning models (BRI: Within the context of ensemble models, the first label and second label output may correspond to use of respective predicted labeled output ) in [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training data set 1002 includes an input paired with a desired output for an input in [0082]: In at least one embodiment, a system performing at least a part of process 500 includes executable code to update 508 a model being trained using training data, labels, augmented training data, and pseudo-labels. In at least one embodiment, an updated model is used to infer data labels from data input . (BRI: the desired output is the label output) - reception of second labels output from the first inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; in [0060]: in at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ense mble consistency regularization in [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training data set 1002 includes an input paired with a desired output for an input - determination of a first difference between the first labels output from the first inference model and the labels corresponding to the plurality of the first sets of values and the pseudo-labels determined for the plurality of the second sets of values In [0011]: FIG. 9A illustrates inference and/or training logic, according to at least one embodiment In [0060]: In at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ensemb le consistency regularization in [0113]: machine learning models or predict or infer information using one or more machine learning models (BRI: Within the context of ensemble models, the first label and second label output may correspond to use of respective predicted labeled output ) In [0006]: FIG. 4 illustrates a diagram of generating pseu do-labels to train neural networks, according to at least one embodiment; In [0067]: In at least one embodiment, GPU processes an image (e.g., image of an airplane) from content input 202 using an average model 204 to make a prediction on image 202 and generate an output 206. In at least one embodiment, output 206 comprises a pseudo-label. (BRI: output 206 is the label output say “X”) in[ 0067]: FIG. 2 illustrates another diagram 200 of determining a model and training scheme for image classification In [0070]: in at least one embodiment, techniques described herein combines a noise-robust task loss with augmentation, pseudo-labeling, and consistency regularization. In at least one embodiment, techniques described herein leverages generalized cross entropy GCE (which is a theoretically grounded noise-robust loss function that can be seen as a generalization of mean absolute error (MAE) (BRI: MAE is the loss considered as first difference between label output and labels associated with regularization application of set of inputs where first difference is within the context of first inference model using one or more models [See 0113]) in [0069]: a consistency regularization term is applied where output s from another model 208 are made to agree with output 206 in [0080]: In at least one embodiment, a training set comprises image data and a corresponding set of labels, which is also referred to as annotations. In at least one embodiment, a sub set of training data comprises an image and a label (where a label corresponds to text data classifying contents of image). In at least one embodiment, a system trains a model using an image and a corresponding label. In at least one embodiment, a label is text data, audio data, or metadata associated with an image in [0062]: a neural network classifies input data by inferring one or more labels for each datum of a set of input data, such as by inferring from an image an object appearing in said image. (BRI: with consistency regularization, particularly when combined with pseudo-labeling , the regularization technique forces the model's outputs to be consistent with its own predictions, effectively using its own outputs (pseudo-labels) as plurality of series of values for unlabeled data to guide the training process) In [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, or where training dataset 1002 includes input having a known output, - determination of a second difference between the first labels output from the first inference model and the second labels output from the second inference model In [0011]: FIG. 9A illustrates inference and/or training logic, according to at least one embodiment In [0060]: In at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ensemb le consistency regularization in [0113]: machine learning models or predict or infer information using one or more machine learning models (BRI: Within the context of ensemble models, the first label and second label output may correspond to use of respective predicted labeled output) In [0006]: FIG. 4 illustrates a diagram of generating pseu do-labels to train neural networks, according to at least one embodiment; In [0067]: In at least one embodiment, GPU processes an image (e.g., image of an airplane) from content input 202 using an average model 204 to make a prediction on image 202 and generate an output 206. In at least one embodiment, output 206 comprises a pseudo-label. (BRI: output 206 is the label output) in[ 0067]: FIG. 2 illustrates another diagram 200 of determining a model and training scheme for image classification In [0070]: in at least one embodiment, techniques described herein combines a noise-robust task loss with augmentation, pseudo-labeling, and consistency regularization. In at least one embodiment, techniques described herein leverages generalized cross entropy GCE (which is a theoretically grounded noise-robust loss function that can be seen as a generalization of mean absolute error (MAE) (BRI: MAE is the loss as a result of second difference between label output from the first inference model and second label output from the second inference model where first and second inference model are within the context of one or more models [See 0113]) in [0069]: a consistency regularization term is applied where output s from another model 208 are made to agree with output 206 in [0062]: a neural network classifies input data by inferring one or more labels for each datum of a set of input data, such as by inferring from an image an object appearing in said image. (BRI: with consistency regularization, particularly when combined with pseudo-labeling , the regularization technique forces the model's outputs to be consistent with its own predictions, effectively using its own outputs (pseudo-labels) as plurality of series of values for unlabeled data to guide the training process) In [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, or where training dataset 1002 includes input having a known output, and a third difference between the second labels output from the second inference model and the labels corresponding to the plurality of the first sets of values and the pseudo-labels determined for the plurality of the second sets of values In [0011]: FIG. 9A illustrates inference and/or training logic, according to at least one embodiment In [0060]: In at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ensemb le consistency regularization in [0113]: machine learning models or predict or infer information using one or more machine learning models In [0006]: FIG. 4 illustrates a diagram of generating pseu do-labels to train neural networks, according to at least one embodiment; In [0067]: In at least one embodiment, GPU processes an image (e.g., image of an airplane) from content input 202 using an average model 204 to make a prediction on image 202 and generate an output 206. In at least one embodiment, output 206 comprises a pseudo-label. (BRI: output 206 is the label output) in[ 0067]: FIG. 2 illustrates another diagram 200 of determining a model and training scheme for image classification In [0070]: in at least one embodiment, techniques described herein combines a noise-robust task loss with augmentation, pseudo-labeling, and consistency regularization. In at least one embodiment, techniques described herein leverages generalized cross entropy GCE (which is a theoretically grounded noise-robust loss function that can be seen as a generalization of mean absolute error (MAE) (BRI: MAE is the loss as a result of third difference between label output and labels associated with regularization application of set of inputs where the third difference associates to a loss and use a second inference model within the context of plurality of models [0113])) in [0069]: a consistency regularization term is applied where output s from another model 208 are made to agree with output 206 in [0062]: a neural network classifies input data by inferring one or more labels for each datum of a set of input data, such as by inferring from an image an object appearing in said image. (BRI: with consistency regularization, particularly when combined with pseudo-labeling , the regularization technique forces the model's outputs to be consistent with its own predictions, effectively using its own outputs (pseudo-labels) as plurality of series of values for unlabeled data to guide the training process) In [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, or where training dataset 1002 includes input having a known output, In [0070]: in at least one embodiment, techniques described herein combines a noise-robust task loss with augmentation, pseudo-labeling, and consistency regularization. In at least one embodiment, techniques described herein leverages generalized cross entropy GCE (which is a theoretically grounded noise-robust loss function that can be seen as a generalization of mean absolute error (MAE) (BRI: MAE is the loss as a result of third difference between label output and labels associated with regularization application of set of inputs where the third difference associates to a loss and use a second inference model within the context of plurality of models [0113])) In [0085]: in at least one embodiment, average model is used 604 to train a student model. In at least one embodiment, a student model is a task loss model that processes one or more augmented images. In [0102]: In at least one embodiment, training framework 1004 trains untrained neural network 1006 repeatedly while adjust weights to refine an output of untrained neural network 1006 using a loss function and adjustment algorithm, such as stochastic gradient descent. In at least one embodiment, training framework 1004 trains untrained neural network 1006 until untrained neural network 1006 achieves a desired accuracy. In at least one embodiment, trained neural network 1008 can then be deployed to implement any number of machine learning operations. In [0059]: pseudo-labels that are generated are assigned to modified training data, such as images that have been augmented in various ways. In at least one embodiment, training a neural network in this way simultaneously utilizes both task loss (updating weights of said neural network so that said neural network infers from datum a label that was provided with said datum) and consistency loss (updating weights of said neural network so that said neural network infers a corresponding pseudo-label consistently for multiple augmentations of said datum). In at least one embodiment, a number of augmentations used in training is increased as training progresses. (BRI: the above is weighted loss providing third difference from the first and second loss(differences) within the context of applying multiple augmentation) - modification of the first inference model based on the first difference in [0081]: generate 506 modified version of first version of training data comprises augmented training data. In at least one embodiment, a system causes a data processing service to perform a modification on training data to modify training data. (BRI: modifying the training data can modify the model through processes like retraining or fine-tuning and may influence model's learning) In [0502]: in at least one embodiment, once a model is trained—or partially trained—at one location, a machine learning model may be added to model registry 3824. In at least one embodiment, a machine learning model may then be retrain ed, or updated, at any number of other facilities, and a retrain ed or updated model may be made available in model registry 3824, in [0559]: In at least one embodiment, to retrain, or update, initial model 4204, output or loss layer(s) of initial model 4204 may be reset, or deleted, and/or replaced with an updated or new output or loss layer(s). (BRI: updating the loss layer may modify and retrain a model. The loss function's explicit purpose is to guide the optimization process) - and modification of the second inference model based on the second difference and the third difference. in [0081]: generate 506 modified version of first version of training data comprises augmented training data. In at least one embodiment, a system causes a data processing service to perform a modification on training data to modify training data. (BRI: modifying the training data can modify the model through processes like retraining or fine-tuning and may influence model's learning) In [0502]: in at least one embodiment, once a model is trained—or partially trained—at one location, a machine learning model may be added to model registry 3824. In at least one embodiment, a machine learning model may then be retrain ed, or updated, at any number of other facilities, and a retrain ed or updated model may be made available in model registry 3824, in [0559]: In at least one embodiment, to retrain, or update, initial model 4204, output or loss layer(s) of initial model 4204 may be reset, or deleted, and/or replaced with an updated or new output or loss layer(s). (BRI: updating the loss layer may modify and retrain a model. The loss function's explicit purpose is to guide the optimization process) It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, and Brown. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. One of ordinary skill would have motivation to combine Mohseni, and Brown that can provide a measure of confidence interpreted as a probability of detections of objects in an image (classification)(Brown [0164]). Mohseni and Brown do not explicitly disclose: - modification of free parameters of the first inference model based on the first difference and not based on the second difference and the third difference; - and modification of free parameters of the second inference model based on the second difference and the third difference, and not based on the first difference However, Xu discloses: - modification of free parameters of the first inference model based on the first difference and not based on the second difference and the third difference; [Col 11, lines 36-40]: in one or more embodiments, the lead management system 102 uses weights of a fuzzy augmentation model to update a previously generated loss function (e.g., for optimization-based algorithms like logistic regression) [Col 11, lines 45-54]: The lead management system 102 can then use the loss function 216 to generate a trained lead scoring model 218 . Specifically, the lead management system 102 uses the function(s) and/or value(s) of the loss function 216 to modify one or more parameters or algorithms of the lead scoring model 200. Modifying parameters or algorithms of the lead scoring model 200 results in the trained lead scoring model 218 that takes into account the synthetic outcomes 214 of the imputed dataset 212 in addition to the actual outcomes 208 of the original dataset 202. [Col 10, lines 9-18]: The lead management system 102 uses a selected reject inference model 210 to generate synthetic data based on the rejected leads 206 of the original dataset 202. As briefly mentioned previously, the reject inference model 210 can be a simple augmentation model or a fuzzy augmentation model. Each of the reject inference models generates synthetic data using different operations, and therefore result in different synthetic data. Furthermore, the reject inference models can also output different amounts and/or types of synthetic data. [Col 2, lines 54-60]: The lead management system selects a reject inference model for generating the imputed dataset based on a plurality of simulations on historical prospect data . The lead management system then uses the selected reject inference model to generate an imputed dataset including synthetic outcome data representing simulated outcomes for the reject data. [Col 3, lines 26-29]: The synthetic outcome data can include a label indicating the outcome of a given rejected prospect, and in some cases a weighting/score, depending on the reject inference model that the lead management system uses. [Col 10, lines 19-20]: In one or more embodiments, the reject inference model 210 builds upon a known good- bad (“KGB”) model. I n particular, a KGB models (used by some conventional models for training a scoring model, typically in credit scoring applications) uses only the known “good” labels and known “bad” labels , thereby only using known labels (while excluding unknown labels in the reject data) to train scoring models. Thus, the KGB model utilizes only leads with known outcomes , which may include only accepted leads 204 in at least some embodiments. For embodiments in which the known labels include at least some rejected leads with outcomes , the KGB model may also be trained on the reject data with outcomes (i.e., a subset of rejected leads that have engagement and outcome information). only the accepted leads are engaged and only labels for A are available, the KGB model is developed based on (X.sub.A, y.sub.A) by dividing A into pseudo -accepted leads and pseudo-rejected leads. (BRI : in the context of selected reject inference for lead scoring, generating synthetic data from rejected leads, and using this synthetic dataset to modify or retrain the lead scoring model by adjustig its parameter does represents modifying free parameters of the inference model) [Col 14, lines 47-56]: apply a KGB model that uses only the known labels. If only the accepted leads are engaged and only labels for A are available, the KGB model is developed based on (X.sub.A, y.sub.A) by dividing A into pseudo-accepted leads and pseudo-rejected leads . Because the KGB model utilizes only known labels, however the resulting performance metric is likely to be biased. Additionally, the KGB model may not split the accepted leads into pseudo -accepted/ pseudo -rejected leads (BRI using a model’s predictions to assign provisional labels to unlabeled data and then including those as part of the training set represents performing pseudo-labeling . The term “pseudo accepted leads” ( or “pseudo accepted labels ”) is essentially a rephrasing of “ pseudo-labels ,” where the labels are not ground truth but are generated by the model itself ) [Col 11, lines 25-40]: the process of generating synthetic data for updating a lead scoring model includes using the imputed dataset 212 to optionally generate a loss function 216 . The loss function 216 indicates a difference between the lead scoring model 200 output (e.g., the scores as predictions of successful outcomes) and the actual outcomes (e.g., outcomes 208 ). To generate the loss function 216, the lead management system 102 compares the resulting outcomes to the predicted outcomes based on the scores from the lead scoring model 200. The loss function 216 can include one or more functions or values representing the difference(s) between the predicted and actual values. In one or more embodiments, the lead management system 102 uses weights of a fuzzy augmentation model to update a previously generated loss function (e.g., for optimization-based algorithms like logistic regression). [Col 12, lines 21-34]: A process for determining whether to generate an imputed dataset includes comparing characteristics of a lead scoring model 300 to a plurality of thresholds. The process begins by the lead management system 102 determining whether a scoring split of the lead scoring model 300 meets a scoring split threshold 302. For instance, the lead management system 102 can analyze the original dataset to determine how accurately the lead scoring model 300 labeled the accepted leads and the rejected leads . To illustrate, the lead management system 102 can determine whether the lead scoring model 300 is accurately indicating that the leads that are most likely to result in a successful outcome are labeled as accepted lead s, while those that are not likely to result in a successful outcome are labeled as rejected leads. [Col 14, lines 61-67]: the lead management system 102 determines an estimate for the area under the receiver operating characteristic curve (i.e., area under ROC curve or simply “AUC”). In particular, the lead management system 102 first estimates the overall true positive rate (“TPR”) and overall false positive rate (“FPR”) for the original dataset. The TPR and FPR may be reasonably estimated via [Col 15, lines 1-2]: reweighting if the lead management system 102 obtains partial rejects via random sampling. [Col 15, lines 3-7]: The overall TPR is a weighted average of the TPR for accepted leads and the TPR for rejected leads. Similarly, the overall FPR is a weighted average of the FPR for accepted leads and the FPR for rejected leads. In particular, the overall TPR is represented as: PNG media_image1.png 32 202 media_image1.png Greyscale [Col 15, lines 14-19]: In which PNG media_image2.png 40 197 media_image2.png Greyscale (BRI: the approach is a form of threshold based model validation and not limited to first or second difference. In this context, the first difference is the change in metric TPR as moved from the threshold the next) - and modification of free parameters of the second inference model based on the second difference and the third difference, and not based on the first difference. [Col 13, lines 57-67]: the lead management system 102 can process the dataset for comparing to threshold s in a single operation or in multiple operations. For example, the lead management system 102 can determine the split effectiveness, success rate, and size simultaneously during a single group of simulations for the original dataset . Alternatively, the lead management system 102 can perform the operations of FIG. 3 in any order and may exclude one or more of the thresholds or include one or more additional thresholds based on the characteristics associated with the original dataset. [Col 15, lines 43-47]: If the split of the original dataset is effective, it is likely that r.sub.A≥r.sub.R. The observed number of “1” labels in the rejected leads, however, is at least the total number of “1” labels in the reje cted leads. Because sub.R.sub.sub.sup.+≤n.sub.R.sup.+=(r.sub.R)(n.sub.R)≤(r.sub.A)(n.sub.R), . a conserva tive estimate of TPR.sub.overall is TPR.sub.overall:=min (A, B) where [Col 15, lines 50-55]: PNG media_image3.png 56 581 media_image3.png Greyscale PNG media_image4.png 63 520 media_image4.png Greyscale (BRI: the approach is a form of threshold based model validation and not limited to first or second difference. In this context, the second difference is the change in the first difference indicating an improvement either TPR slows down or speed as the threshold increases. Changing threshold impact the TPR. Excluding thresholds increase TPR are overlay strict and adding thresholds decrease TPR if the threshold is too lenient (more false positive)) [Col 26, lines 36-44]: The lead management system 102 includes a lead scoring model manager 708 to facilitate the generation of scores for leads associated with an entity. For example, the lead scoring model manager 708 can utilize a lead scoring model to generate scores for leads by analyzing features of the leads and then ranking the leads on a scale. The lead scoring model manager 708 can also assign the leads labels indicating an accepted or rejected lead based on a predetermined threshold . [Col 12, lines 55-58]: the reject rate of the original dataset is small with a small mislabel rate . For example, a small reject rate indicates that a small number of unengaged rejected leads are incorrectly labeled. [Col 15, lines 36-41]: Information on whether a lead has been engaged may be missing for certain bases. For instance, failed engaged leads and unengaged rejected leads may be labeled as “0” because of a negative outcome or negative engagement. Accordingly, unengaged rejected leads that are labeled as “1” are mislabeled. (BRI: A small rejection rate means most leads are being accepted. If those accepted leads are mostly unengaged (i.e., they are actually negative cases but were mislabeled as positive), then the proportion of such misclassifications among the accepted set will be high — and that will directly increase the FPR. It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, Brown and Xu. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. Xu teaches updating the free parameters in an inference model. One of ordinary skill would have motivation to combine Mohseni, Brown and Xu that improve the performance of the lead scoring model for future datasets)( Xu [Col 3, lines 36-41]) In regard to claim 2 : (Previously Presented) Mohseni does not explicitly disclose: - wherein each of the plurality of models conforms to hyperparameters which are different from hyperparameters to which each other of the plurality of models conforms - and is trained using training parameters different from hyperparameters used to which each of the plurality of models conforms However, Brown discloses: - wherein each of the plurality of models conforms to hyperparameters which are different from hyperparameters to which each other of the plurality of models conforms In [0090]: In at least one embodiment, inference and/or training logic 915 may include, without limitation, code and/or data storage 901 to store forward and/or output weight and/or input/output data, and/or other parameters to configure neurons or layers of a neural network trained and/or used for inferencing in aspects of one or more embodiments (BRI: parameters that are used to configure a neural network that are set before the training process begins. They control the learning process and the architecture of the model) In [0090]: in at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on an architecture of a neural network to which such code corresponds. In at least one embodiment, code and/or data storage 901 stores weight parameters and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input/output data and/or weight parameter s during training and/or inferencing using aspects of one or more embodiments. (BRI: the weight is a model parameter which is different from hyperparameters) - and is trained using training parameters different from hyperparameters used to which each of the plurality of models conforms In [0102]: in at least one embodiment, untrained neural network 1006 is trained in a supervised manner and processes inputs from training dataset 1002 and compares resulting outputs against a set of expected or desired outputs . In at least one embodiment, errors are then propagated back through untrained neural network 1006. In at least one embodiment, training framework 1004 adjusts weights that control untrained neural network 1006 . In at least one embodiment, training framework 1004 includes tools to monitor how well untrained neural network 1006 is converging towards a model, such as trained neural network 1008, (BRI: Refers to training data , which is the dataset used to adjust a model's parameters to recognize patterns or make decisions) It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, and Brown. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. One of ordinary skill would have motivation to combine Mohseni, and Brown that can provide a measure of confidence interpreted as a probability of detections of objects in an image (classification)(Brown [0164]). In regard to claim 3 : (Previously Presented) Mohseni does not explicitly disclose: - wherein the first inference model and the second inference model conform to hyperparameters which are different from hyperparameters to which each of the plurality of models conforms. However, Brown discloses: - wherein the first inference model and the second inference model conform to hyperparameters which are different from hyperparameters to which each of the plurality of models conforms. In [0102]: n at least one embodiment, untrained neural network 1006 is trained in a supervised manner and processes inputs from training dataset 1002 and compares resulting outputs against a set of expected or desired outputs . In at least one embodiment, errors are then propagated back through untrained neural network 1006. In at least one embodiment, training framework 1004 adjusts weights that control untrained neural network 1006 . In at least one embodiment, training framework 1004 includes tools to monitor how well untrained neural network 1006 is converging towards a model, such as trained neural network 1008, (BRI: Refers to training data , which is the dataset used to adjust a model's parameters to recognize patterns or make decisions) In regard to claim 4 : (Original) Mohseni does not explicitly disclose: - wherein determination of a pseudo-label for a feature comprises determining an average of the plurality of output labels generated for the feature. However, Brown discloses: - wherein determination of a pseudo-label for a feature comprises determining an average of the plurality of output labels generated for the feature. In [0065]: In at least one embodiment, GPU 108 receives training data 102 and uses average model 110 to process training data 102 . In at least one embodiment, GPU 108 uses average model 110 to make a prediction on an image from a first version of training data 102. In at least one embodiment, an average model 110 is generated by averaging weights from one or more previous versions of one or more neural networks . In at least one embodiment, previous versions of one or more neural networks include weights from each training round (e.g., training iteration, training step) after said one or more neural networks processes training data 102. In [0086] : In at least one embodiment, a system determines that training will continue as an average model can still be improved. In at least one embodiment, as more modified training data is processed by student model, an average model is updated with more information from student model. In at least one embodiment, teacher model is continually made stronger and is trained to infer same information from multiple different modifications of same input. In at least one embodiment, as training continues, data processing service provides additional augmented images to student model for training. In [0066]: In at least one embodiment, each round of training updates weights so that a trained model matches an image to a label, said trained model then matches image to a pseudo-label that is generated from average model based on image input, and then further matches augmented images to pseudo-labels generated by average models based on one or more augmented images. (BRI: a model that uses an average of its weights (often called a "teacher" model) to generate pseudo-labels based on features of training data will produce a more stable and averaged output label than a single model trained from scratch ) It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, and Brown. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. One of ordinary skill would have motivation to combine Mohseni, and Brown that can provide a measure of confidence interpreted as a probability of detections of objects in an image (classification)(Brown [0164]). In regard to claim 6 : (Previously Presented) Mohseni discloses: - wherein the plurality of the first sets of values and the plurality of the second sets of values comprise a fixed ratio of pairs from the first set of training data and pairs from the second set of training data. In [0052]: neural network 100 includes a first portion 112 that includes first set of output nodes 104. In at least one embodiment, neural network 100 includes a second portion 114 that includes second set of output nodes 106. In [0051]: in at least one embodiment, output layer 102 includes a first set of output nodes 104 and a second set of output nodes 106. In at least one embodiment, first set of output nodes 104 includes at least one output node 108. In at least one embodiment, output nodes 108 are classification output nodes . In [0052]: In at least one embodiment, neural network 100 is used to identify out-of-distribution (OOD) input data by producing an output at second set of output nodes 106. In [0052]: In at least one embodiment, neural network 100 is used during inferencing to identify OOD input data as unknown objects In [0057]: In at least one embodiment, technique 300 includes, at a block 302, training a first portion of a neural network on a first set of data. In at least one embodiment, first set of data is an in-distribution training set such as IND training set 206 of FIG. 2. In at least one embodiment, training first portion includes training first portion to classify in-distribution input data to greater than a first predefined classification metric. In [0057]: In at least one embodiment, said first predefined classification metric is based on at least one of a false posive rate (FPR) and a true positive rate (TPR), such as a FPR at a TPR of 0.95, or any other suitable FPR and TPR relationship. In [ 0060]: In at least one embodiment, pseudo-labels assigned at block 306 are different than labels associated with an IND training set. In at least one embodiment, IND labels include numbers from 0 to 99 for a one hundred class classifier, and pseudo-labels associated an OOD training set include numbers from 100 to a predetermined number greater than 100. In at least one embodiment, technique 300 keeps some in-distribution samples in each mini-batch while training OOD samples at block 306 so neural network model does not forget in-distribution features learned at block 302 while training OOD detectors at block 306. In at least one embodiment, a predetermined ratio of IND samples to OOD samples is used during training OOD detectors, such as a ratio of one IND sample to five OOD samples, or a ratio of one IND sample to four OOD samples. In at least one embodiment, a ratio of IND samples to OOD samples is progressively altered during training OOD detectors from a predetermined initial ratio to a predetermined final ratio. (BRI: a predetermined ratio of in-distribution and out-of-distribution samples during training can influence how the training samples are split) In regard to claim 8 : (Currently Amended) Mohseni discloses: inputting second sets of values of each of plurality of features into each of the plurality of trained models to generate, for each of the second sets of values, a plurality of output labels; In [0106]: i n at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1100 by using weight parameters calculated through one or more training techniques described herein. in [0062]: in at least one embodiment, second portion is trained to cluster unlabeled OOD samples using rejector functions in a manner such that first portion maintains in-distribution classification performance in [ 0055]: In at least one embodiment, first set of output nodes 212 are classifier functions. In at least one embodiment, first set of output nodes 212 are output nodes for a different type of machine learning technique In [0055]: In at least one embodiment, first set of output nodes 212 learn target labels from IND training set 206 during training. in [0051]: In at least one embodiment, target labels are ground truth data that indicate what classifier functions are to infer. (BRI: A target label is an output label) - determining, for each of the plurality of features, a pseudo-label generated for the feature In [0059]: In at least one embodiment, third set of data is unlabeled. In at least one embodiment, third set of data includes inputs that lack corresponding ground truth data (e.g., labels, annotations, or metadata) that indicate what second portion is to infer. In at least one embodiment, technique 300 includes assigning pseudo-labels to OOD training samples at block 306. In [0106]: i n at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1100 by using weight parameters calculated through one or more training techniques described herein. In [0062]: in at least one embodiment, second portion is trained to cluster unlabeled OOD samples using rejector functions in a manner such that first portion maintains in-distribution classification performance - determining a second set of training data comprising a second plurality of pairs, each of the second plurality of pairs comprising one of the plurality of features and a pseudo-label determined for the one of the plurality of features; In [0060]: In at least one embodiment, first portion of neural network and second portion of neural network are trained with an automated technique. In at least one embodiment, first portion of neural network and second portion of neural network are trained using a machine learning (ML) approach to dynamically determine at least one of a ratio of IND samples to OOD samples, a learning rate, a dropout rate, and/or any other suitable parameters or hyperparameters in accordance with at least one performance metric. (BRI: the OOD sample has pseudo-labels. The mix of the OOD and IND is via ratio) In [0094]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, In [0094]: In at least one embodiment, untrained neural network 1006 is trained in a supervised manner processes inputs from training dataset 1002 and compares resulting outputs against a set of expected or desired outputs - and training an inference model to output an inferred label based on the first set of training data and the second set of training data In [0054]: model architecture 200 includes a neural network 202 and training data sets 204. In at least one embodiment, training data sets 204 include an in- distribution (IND) training data set 206 and an out-of- distribution (OOD) training data set 208, In [0072]: performing other actions at block 510 includes leveraging an ensemble of multiple DNN blocks, In [0094]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, In [0051]: classifier functions learn target labels from a labeled in-distribution training set during a supervised learning technique, In [ 0094]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, In [0061]: a ratio of IND samples to OOD samples is progressively altered during training OOD detectors from a predetermined initial ratio to a predetermined final ratio, in [0060: dynamically determine at least one of a ratio of IND samples to OOD samples, a learning rate, a dropout rate, and/or any other suitable parameter s or hyper parameter s in accordance with at least one performance metric. (BRI: a setting that can be adjusted to fine-tune the model's predictions based on learning . The learning rate or a dropout rate is a hyperparameter learned by altering the ratio) Mohseni does not explicitly disclose: A method comprising: training each of a plurality of models based on a first set of training data comprising a first plurality of pairs, each of the first plurality of pairs comprising a feature and a corresponding label; inputting a plurality of the first sets of values and a plurality of the second sets of values to the first inference model and to the second inference model; receiving first labels output from the first inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; receiving second labels output from the second inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; - determination of a first difference between the first labels output from the first inference model and the labels corresponding to the plurality of the first sets of values and the pseudo-labels determined for the plurality of the second sets of values; - determination of a second difference between the first labels output from the first inference model and the second labels output from the second inference model and a third difference between the second labels output from the second inference model and the labels corresponding to the plurality of the first sets of values and the pseudo-labels determined for the plurality of the second sets of values; modifying the first inference model based on the first difference and modifying the second inference model based on the first difference and the third difference However, Brown discloses: A method comprising: training each of a plurality of models based on a first set of training data comprising a first plurality of pairs, each of the first plurality of pairs comprising a feature and a corresponding label; In [0520]: In at least one embodiment, for each instance of imaging data 3808 (or other data type used by machine learning models), there may be corresponding ground truth data generated by training system 3804, In [0520]: In at least one embodiment, ground truth data may be synthetically produced (e.g., generated from computer models or renderings), real produced (e.g., designed and produced from real-world data),machine-automated (e.g., using feature analysis and learning to extract features from data and then generate labels), inputting a plurality of the first sets of values and a plurality of the second sets of values to the first inference model and to the second inference model; in [0082]: In at least one embodiment, a system performing at least a part of process 500 includes executable code to update 508 a model being trained using training data, labels, augmented training data, and pseudo-labels. In at least one embodiment, an updated model is used to infer data labels from data input . in [0078]: FIG. 4 illustrates a diagram 400 of generating pseudo-labels 406 to train neural networks, according to at least one embodiment. In at least one embodiment, a model 408 is trained over one or more iterations 410, where at each round of training, weights of said model 408 are adjusted to cause said model 408 to make inference s that match labels and pseudo-labels of data used to train said model 408. In at least one embodiment, GPU determines, for a model 408 that is trained using training data, a set of weights. (BRI: model 408 is an inference model) receiving first labels output from the first inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; in [0060]: in at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ense mble consistency regularization in [0113]: machine learning models or predict or infer information using one or more machine learning models (BRI: Within the context of ensemble models, the first label and second label output may correspond to use of respective predicted labeled output ) in [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training data set 1002 includes an input paired with a desired output for an input in [0082]: In at least one embodiment, a system performing at least a part of process 500 includes executable code to update 508 a model being trained using training data, labels, augmented training data, and pseudo-labels. In at least one embodiment, an updated model is used to infer data labels from data input . (BRI: the desired output is the label output) receiving second labels output from the second inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; in [0060]: in at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ense mble consistency regularization in [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training data set 1002 includes an input paired with a desired output for an input - determination of a first difference between the first labels output from the first inference model and the labels corresponding to the plurality of the first sets of values and the pseudo-labels determined for the plurality of the second sets of values; In [0011]: FIG. 9A illustrates inference and/or training logic, according to at least one embodiment In [0060]: In at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ensemb le consistency regularization in [0113]: machine learning models or predict or infer information using one or more machine learning models (BRI: Within the context of ensemble models, the first label and second label output may correspond to use of respective predicted labeled output ) in [0080]: In at least one embodiment, a training set comprises image data and a corresponding set of labels, which is also referred to as annotations. In at least one embodiment, a sub set of training data comprises an image and a label (where a label corresponds to text data classifying contents of image). In at least one embodiment, a system trains a model using an image and a corresponding label. In at least one embodiment, a label is text data, audio data, or metadata associated with an image In [0006]: FIG. 4 illustrates a diagram of generating pseu do-labels to train neural networks, according to at least one embodiment; In [0067]: In at least one embodiment, GPU processes an image (e.g., image of an airplane) from content input 202 using an average model 204 to make a prediction on image 202 and generate an output 206. In at least one embodiment, output 206 comprises a pseudo-label. (BRI: output 206 is the label output say “X”) in[ 0067]: FIG. 2 illustrates another diagram 200 of determining a model and training scheme for image classification In [0070]: in at least one embodiment, techniques described herein combines a noise-robust task loss with augmentation, pseudo-labeling, and consistency regularization. In at least one embodiment, techniques described herein leverages generalized cross entropy GCE (which is a theoretically grounded noise-robust loss function that can be seen as a generalization of mean absolute error (MAE) (BRI: MAE is the loss considered as first difference between label output and labels associated with regularization application of set of inputs where first difference is within the context of first inference model using one or more models [See 0113]) in [0069]: a consistency regularization term is applied where outputs from another model 208 are made to agree with output 206 in [0062]: a neural network classifies input data by inferring one or more labels for each datum of a set of input data, such as by inferring from an image an object appearing in said image. (BRI: with consistency regularization, particularly when combined with pseudo-labeling , the regularization technique forces the model's outputs to be consistent with its own predictions, effectively using its own outputs (pseudo-labels) as plurality of series of values for unlabeled data to guide the training process) In [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, or where training dataset 1002 includes input having a known output, - determination of a second difference between the first labels output from the first inference model and the second labels output from the second inference model in [0011]: FIG. 9A illustrates inference and/or training logic, according to at least one embodiment In [0060]: In at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ensemble consistency regularization in [0113]: machine learning models or predict or infer information using one or more machine learning models (BRI: Within the context of ensemble models, the first label and second label output may correspond to use of respective predicted labeled output) In [0006]: FIG. 4 illustrates a diagram of generating pseudo-labels to train neural networks, according to at least one embodiment; In [0067]: In at least one embodiment, GPU processes an image (e.g., image of an airplane) from content input 202 using an average model 204 to make a prediction on image 202 and generate an output 206. In at least one embodiment, output 206 comprises a pseudo-label. (BRI: output 206 is the label output) in[ 0067]: FIG. 2 illustrates another diagram 200 of determining a model and training scheme for image classification In [0070]: in at least one embodiment, techniques described herein combines a noise-robust task loss with augmentation, pseudo-labeling, and consistency regularization. In at least one embodiment, techniques described herein leverages generalized cross entropy GCE (which is a theoretically grounded noise-robust loss function that can be seen as a generalization of mean absolute error (MAE) (BRI: MAE is the loss as a result of second difference between label output from the first inference model and second label output from the second inference model where first and second inference model are within the context of one or more models [See 0113]) in [0069]: a consistency regularization term is applied where outputs from another model 208 are made to agree with output 206 in [0062]: a neural network classifies input data by inferring one or more labels for each datum of a set of input data, such as by inferring from an image an object appearing in said image. (BRI: with consistency regularization, particularly when combined with pseudo-labeling , the regularization technique forces the model's outputs to be consistent with its own predictions, effectively using its own outputs (pseudo-labels) as plurality of series of values for unlabeled data to guide the training process) In [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, or where training dataset 1002 includes input having a known output, - and a third difference between the second labels output from the second inference model and the labels corresponding to the plurality of the first sets of values and the pseudo-labels determined for the plurality of the second sets of values; In [0011]: FIG. 9A illustrates inference and/or training logic, according to at least one embodiment In [0060]: In at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ensemb le consistency regularization in [0113]: machine learning models or predict or infer information using one or more machine learning models In [0006]: FIG. 4 illustrates a diagram of generating pseu do-labels to train neural networks, according to at least one embodiment; In [0067]: In at least one embodiment, GPU processes an image (e.g., image of an airplane) from content input 202 using an average model 204 to make a prediction on image 202 and generate an output 206. In at least one embodiment, output 206 comprises a pseudo-label. (BRI: output 206 is the label output) in[ 0067]: FIG. 2 illustrates another diagram 200 of determining a model and training scheme for image classification In [0070]: in at least one embodiment, techniques described herein combines a noise-robust task loss with augmentation, pseudo-labeling, and consistency regularization. In at least one embodiment, techniques described herein leverages generalized cross entropy GCE (which is a theoretically grounded noise-robust loss function that can be seen as a generalization of mean absolute error (MAE) (BRI: MAE is the loss as a result of third difference between label output and labels associated with regularization application of set of inputs where the third difference associates to a loss and use a second inference model within the context of plurality of models [0113])) in [0069]: a consistency regularization term is applied where output s from another model 208 are made to agree with output 206 in [0062]: a neural network classifies input data by inferring one or more labels for each datum of a set of input data, such as by inferring from an image an object appearing in said image. (BRI: with consistency regularization, particularly when combined with pseudo-labeling , the regularization technique forces the model's outputs to be consistent with its own predictions, effectively using its own outputs (pseudo-labels) as plurality of series of values for unlabeled data to guide the training process) In [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, or where training dataset 1002 includes input having a known output, In [0070]: in at least one embodiment, techniques described herein combines a noise-robust task loss with augmentation, pseudo-labeling, and consistency regularization. In at least one embodiment, techniques described herein leverages generalized cross entropy GCE (which is a theoretically grounded noise-robust loss function that can be seen as a generalization of mean absolute error (MAE) (BRI: MAE is the loss as a result of third difference between label output and labels associated with regularization application of set of inputs where the third difference associates to a loss and use a second inference model within the context of plurality of models [0113])) In [0085]: in at least one embodiment, average model is used 604 to train a student model. In at least one embodiment, a student model is a task loss model that processes one or more augmented images. In [0102]: In at least one embodiment, training framework 1004 trains untrained neural network 1006 repeatedly while adjust weights to refine an output of untrained neural network 1006 using a loss function and adjustment algorithm, such as stochastic gradient descent. In at least one embodiment, training framework 1004 trains untrained neural network 1006 until untrained neural network 1006 achieves a desired accuracy. In at least one embodiment, trained neural network 1008 can then be deployed to implement any number of machine learning operations. In [0059]: pseudo-labels that are generated are assigned to modified training data, such as images that have been augmented in various ways. In at least one embodiment, training a neural network in this way simultaneously utilizes both task loss (updating weights of said neural network so that said neural network infers from datum a label that was provided with said datum) and consistency loss (updating weights of said neural network so that said neural network infers a corresponding pseudo-label consistently for multiple augmentations of said datum). In at least one embodiment, a number of augmentations used in training is increased as training progresses. (BRI: the above is weighted loss providing third difference from the first and second loss(differences) within the context of applying multiple augmentation) modifying the first inference model based on the first difference in [0081]: generate 506 modified version of first version of training data comprises augmented training data. In at least one embodiment, a system causes a data processing service to perform a modification on training data to modify training data. (BRI: modifying the training data can modify the model through processes like retraining or fine-tuning and may influence model's learning) In [0502]: in at least one embodiment, once a model is trained—or partially trained—at one location, a machine learning model may be added to model registry 3824. In at least one embodiment, a machine learning model may then be retrain ed, or updated, at any number of other facilities, and a retrain ed or updated model may be made available in model registry 3824, in [0559]: In at least one embodiment, to retrain, or update, initial model 4204, output or loss layer(s) of initial model 4204 may be reset, or deleted, and/or replaced with an updated or new output or loss layer(s). (BRI: updating the loss layer may modify and retrain a model. The loss function's explicit purpose is to guide the optimization process) and modifying the second inference model based on the second difference and the third difference in [0081]: generate 506 modified version of first version of training data comprises augmented training data. In at least one embodiment, a system causes a data processing service to perform a modification on training data to modify training data. (BRI: modifying the training data can modify the model through processes like retraining or fine-tuning and may influence model's learning) In [0502]: in at least one embodiment, once a model is trained—or partially trained—at one location, a machine learning model may be added to model registry 3824. In at least one embodiment, a machine learning model may then be retrain ed, or updated, at any number of other facilities, and a retrain ed or updated model may be made available in model registry 3824, in [0559]: In at least one embodiment, to retrain, or update, initial model 4204, output or loss layer(s) of initial model 4204 may be reset, or deleted, and/or replaced with an updated or new output or loss layer(s). (BRI: updating the loss layer may modify and retrain a model. The loss function's explicit purpose is to guide the optimization process) It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, and Brown. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. One of ordinary skill would have motivation to combine Mohseni, and Brown that can provide a measure of confidence interpreted as a probability of detections of objects in an image (classification)(Brown [0164]). Mohseni and Brown do not explicitly disclose: - modification of free parameters of the first inference model based on the first difference and not based on the second difference and the third difference; - and modification of free parameters of the second inference model based on the second difference and the third difference, and not based on the first difference However, Xu discloses: - modification of free parameters of the first inference model based on the first difference and not based on the second difference and the third difference; [Col 11, lines 36-40]: in one or more embodiments, the lead management system 102 uses weights of a fuzzy augmentation model to update a previously generated loss function (e.g., for optimization-based algorithms like logistic regression) [Col 11, lines 45-54]: The lead management system 102 can then use the loss function 216 to generate a trained lead scoring model 218 . Specifically, the lead management system 102 uses the function(s) and/or value(s) of the loss function 216 to modify one or more parameters or algorithms of the lead scoring model 200. Modifying parameters or algorithms of the lead scoring model 200 results in the trained lead scoring model 218 that takes into account the synthetic outcomes 214 of the imputed dataset 212 in addition to the actual outcomes 208 of the original dataset 202. [Col 10, lines 9-18]: The lead management system 102 uses a selected reject inference model 210 to generate synthetic data based on the rejected leads 206 of the original dataset 202. As briefly mentioned previously, the reject inference model 210 can be a simple augmentation model or a fuzzy augmentation model. Each of the reject inference models generates synthetic data using different operations, and therefore result in different synthetic data. Furthermore, the reject inference models can also output different amounts and/or types of synthetic data. [Col 2, lines 54-60]: The lead management system selects a reject inference model for generating the imputed dataset based on a plurality of simulations on historical prospect data . The lead management system then uses the selected reject inference model to generate an imputed dataset including synthetic outcome data representing simulated outcomes for the reject data. [Col 3, lines 26-29]: The synthetic outcome data can include a label indicating the outcome of a given rejected prospect, and in some cases a weighting/score, depending on the reject inference model that the lead management system uses. [Col 10, lines 19-20]: In one or more embodiments, the reject inference model 210 builds upon a known good- bad (“KGB”) model. I n particular, a KGB models (used by some conventional models for training a scoring model, typically in credit scoring applications) uses only the known “good” labels and known “bad” labels , thereby only using known labels (while excluding unknown labels in the reject data) to train scoring models. Thus, the KGB model utilizes only leads with known outcomes , which may include only accepted leads 204 in at least some embodiments. For embodiments in which the known labels include at least some rejected leads with outcomes , the KGB model may also be trained on the reject data with outcomes (i.e., a subset of rejected leads that have engagement and outcome information). only the accepted leads are engaged and only labels for A are available, the KGB model is developed based on (X.sub.A, y.sub.A) by dividing A into pseudo -accepted leads and pseudo-rejected leads. (BRI : in the context of selected reject inference for lead scoring, generating synthetic data from rejected leads, and using this synthetic dataset to modify or retrain the lead scoring model by adjustig its parameter does represents modifying free parameters of the inference model) [Col 14, lines 47-56]: apply a KGB model that uses only the known labels. If only the accepted leads are engaged and only labels for A are available, the KGB model is developed based on (X.sub.A, y.sub.A) by dividing A into pseudo-accepted leads and pseudo-rejected leads . Because the KGB model utilizes only known labels, however the resulting performance metric is likely to be biased. Additionally, the KGB model may not split the accepted leads into pseudo -accepted/ pseudo -rejected leads (BRI using a model’s predictions to assign provisional labels to unlabeled data and then including those as part of the training set represents performing pseudo-labeling . The term “pseudo accepted leads” ( or “pseudo accepted labels ”) is essentially a rephrasing of “ pseudo-labels ,” where the labels are not ground truth but are generated by the model itself ) [Col 11, lines 25-40]: the process of generating synthetic data for updating a lead scoring model includes using the imputed dataset 212 to optionally generate a loss function 216 . The loss function 216 indicates a difference between the lead scoring model 200 output (e.g., the scores as predictions of successful outcomes) and the actual outcomes (e.g., outcomes 208 ). To generate the loss function 216, the lead management system 102 compares the resulting outcomes to the predicted outcomes based on the scores from the lead scoring model 200. The loss function 216 can include one or more functions or values representing the difference(s) between the predicted and actual values. In one or more embodiments, the lead management system 102 uses weights of a fuzzy augmentation model to update a previously generated loss function (e.g., for optimization-based algorithms like logistic regression). [Col 12, lines 21-34]: A process for determining whether to generate an imputed dataset includes comparing characteristics of a lead scoring model 300 to a plurality of thresholds. The process begins by the lead management system 102 determining whether a scoring split of the lead scoring model 300 meets a scoring split threshold 302. For instance, the lead management system 102 can analyze the original dataset to determine how accurately the lead scoring model 300 labeled the accepted leads and the rejected leads . To illustrate, the lead management system 102 can determine whether the lead scoring model 300 is accurately indicating that the leads that are most likely to result in a successful outcome are labeled as accepted lead s, while those that are not likely to result in a successful outcome are labeled as rejected leads. [Col 14, lines 61-67]: the lead management system 102 determines an estimate for the area under the receiver operating characteristic curve (i.e., area under ROC curve or simply “AUC”). In particular, the lead management system 102 first estimates the overall true positive rate (“TPR”) and overall false positive rate (“FPR”) for the original dataset. The TPR and FPR may be reasonably estimated via [Col 15, lines 1-2]: reweighting if the lead management system 102 obtains partial rejects via random sampling. [Col 15, lines 3-7]: The overall TPR is a weighted average of the TPR for accepted leads and the TPR for rejected leads. Similarly, the overall FPR is a weighted average of the FPR for accepted leads and the FPR for rejected leads. In particular, the overall TPR is represented as: PNG media_image1.png 32 202 media_image1.png Greyscale [Col 15, lines 14-19]: In which PNG media_image2.png 40 197 media_image2.png Greyscale (BRI: the approach is a form of threshold based model validation and not limited to first or second difference. In this context, the first difference is the change in metric TPR as moved from the threshold the next) - and modification of free parameters of the second inference model based on the second difference and the third difference, and not based on the first difference. [Col 13, lines 57-67]: the lead management system 102 can process the dataset for comparing to threshold s in a single operation or in multiple operations. For example, the lead management system 102 can determine the split effectiveness, success rate, and size simultaneously during a single group of simulations for the original dataset . Alternatively, the lead management system 102 can perform the operations of FIG. 3 in any order and may exclude one or more of the thresholds or include one or more additional thresholds based on the characteristics associated with the original dataset. [Col 15, lines 43-47]: If the split of the original dataset is effective, it is likely that r.sub.A≥r.sub.R. The observed number of “1” labels in the rejected leads, however, is at least the total number of “1” labels in the reje cted leads. Because sub.R.sub.sub.sup.+≤n.sub.R.sup.+=(r.sub.R)(n.sub.R)≤(r.sub.A)(n.sub.R), . a conserva tive estimate of TPR.sub.overall is TPR.sub.overall:=min (A, B) where [Col 15, lines 50-55]: PNG media_image3.png 56 581 media_image3.png Greyscale PNG media_image4.png 63 520 media_image4.png Greyscale (BRI: the approach is a form of threshold based model validation and not limited to first or second difference. In this context, the second difference is the change in the first difference indicating an improvement either TPR slows down or speed as the threshold increases. Changing threshold impact the TPR. Excluding thresholds increase TPR are overlay strict and adding thresholds decrease TPR if the threshold is too lenient (more false positive)) [Col 26, lines 36-44]: The lead management system 102 includes a lead scoring model manager 708 to facilitate the generation of scores for leads associated with an entity. For example, the lead scoring model manager 708 can utilize a lead scoring model to generate scores for leads by analyzing features of the leads and then ranking the leads on a scale. The lead scoring model manager 708 can also assign the leads labels indicating an accepted or rejected lead based on a predetermined threshold . [Col 12, lines 55-58]: the reject rate of the original dataset is small with a small mislabel rate . For example, a small reject rate indicates that a small number of unengaged rejected leads are incorrectly labeled. [Col 15, lines 36-41]: Information on whether a lead has been engaged may be missing for certain bases. For instance, failed engaged leads and unengaged rejected leads may be labeled as “0” because of a negative outcome or negative engagement. Accordingly, unengaged rejected leads that are labeled as “1” are mislabeled. (BRI: A small rejection rate means most leads are being accepted. If those accepted leads are mostly unengaged (i.e., they are actually negative cases but were mislabeled as positive), then the proportion of such misclassifications among the accepted set will be high — and that will directly increase the FPR. It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, Brown and Xu. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. Xu teaches updating the free parameters in an inference model. One of ordinary skill would have motivation to combine Mohseni, Brown and Xu that improve the performance of the lead scoring model for future datasets)( Xu [Col 3, lines 36-41]) In regard to claim 9 : (Previously Presented) Mohseni does not explicitly disclose: - wherein each of the plurality of models conforms to hyperparameters which are different from hyperparameters to which each other of the plurality of models conforms - and is trained using training parameters different from hyperparameters used to which each of the plurality of models conforms However, Brown discloses: - wherein each of the plurality of models conforms to hyperparameters which are different from hyperparameters to which each other of the plurality of models conforms In [0090]: In at least one embodiment, inference and/or training logic 915 may include, without limitation, code and/or data storage 901 to store forward and/or output weight and/or input/output data, and/or other parameter s to configure neurons or layers of a neural network trained and/or used for inferencing in aspects of one or more embodiments (BRI: parameters that are used to configure a neural network that are set before the training process begins. They control the learning process and the architecture of the model) In [0090]: in at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on an architecture of a neural network to which such code corresponds. In at least one embodiment, code and/or data storage 901 stores weight parameter s and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input/output data and/or weight parameter s during training and/or inferencing using aspects of one or more embodiments. (BRI: the weight is a model parameter which is different from hyperparameters) - and is trained using training parameters different from hyperparameters used to which each of the plurality of models conforms In [0102]: n at least one embodiment, untrained neural network 1006 is trained in a supervised manner and processes inputs from training dataset 1002 and compares resulting outputs against a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through untrained neural network 1006. In at least one embodiment, training framework 1004 adjust s weights that control untrained neural network 1006. In at least one embodiment, training framework 1004 includes tools to monitor how well untrained neural network 1006 is converging towards a model, such as trained neural network 1008, (BRI: Refers to training data , which is the dataset used to adjust a model's parameters to recognize patterns or make decisions) It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, and Brown. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. One of ordinary skill would have motivation to combine Mohseni, and Brown that can provide a measure of confidence interpreted as a probability of detections of objects in an image (classification)(Brown [0164]). In regard to claim 10 : (Previously Presented) Mohseni does not explicitly disclose: - wherein the first inference model and the second inference model conform to hyperparameters which are different from hyperparameters to which each of the plurality of models conforms. However, Brown discloses: - wherein the first inference model and the second inference model conform to hyperparameters which are different from hyperparameters to which each of the plurality of models conforms. In [0102]: in at least one embodiment, untrained neural network 1006 is trained in a supervised manner and processes inputs from training dataset 1002 and compares resulting outputs against a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through untrained neural network 1006. In at least one embodiment, training framework 1004 adjust s weights that control untrained neural network 1006. In at least one embodiment, training framework 1004 includes tools to monitor how well untrained neural network 1006 is converging towards a model, such as trained neural network 1008, (BRI: Refers to training data , which is the dataset used to adjust a model's parameters to recognize patterns or make decisions) In regard to claim 11 : (Original) Mohseni does not explicitly disclose: - wherein determination of a pseudo-label for a feature comprises determining an average of the plurality of output labels generated for the feature. However, Brown discloses: - wherein determination of a pseudo-label for a feature comprises determining an average of the plurality of output labels generated for the feature. In [0065]: In at least one embodiment, GPU 108 receives training data 102 and uses average model 110 to process training data 102. In at least one embodiment, GPU 108 uses average model 110 to make a prediction on an image from a first version of training data 102. In at least one embodiment, an average model 110 is generated by averaging weights from one or more previous versions of one or more neural networks. In at least one embodiment, previous versions of one or more neural networks include weights from each training round (e.g., training iteration, training step) after said one or more neural networks processes training data 102. In [0086] : In at least one embodiment, a system determines that training will continue as an average model can still be improved. In at least one embodiment, as more modified training data is processed by student model, an average model is updated with more information from student model. In at least one embodiment, teacher model is continually made stronger and is trained to infer same information from multiple different modifications of same input. In at least one embodiment, as training continues, data processing service provides additional augmented images to student model for training. In [0066]: In at least one embodiment, each round of training updates weights so that a trained model matches an image to a label, said trained model then matches image to a pseudo-label that is generated from average model based on image input, and then further matches augmented images to pseudo-labels generated by average models based on one or more augmented images. (BRI: a model that uses an average of its weights (often called a "teacher" model) to generate pseudo-labels based on features of training data will produce a more stable and averaged output label than a single model trained from scratch ) It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, and Brown. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. One of ordinary skill would have motivation to combine Mohseni, and Brown that can provide a measure of confidence interpreted as a probability of detections of objects in an image (classification)(Brown [0164]). In regard to claim 13 : (Previously Presented) Mohseni discloses: wherein the plurality of the first sets of values and the plurality of the second sets of values comprise a fixed ratio of pairs from the first set of training data and pairs from the second set of training data. In [0052]: neural network 100 includes a first portion 112 that includes first set of output nodes 104. In at least one embodiment, neural network 100 includes a second portion 114 that includes second set of output nodes 106. In [0051]: in at least one embodiment, output layer 102 includes a first set of output nodes 104 and a second set of output nodes 106. In at least one embodiment, first set of output nodes 104 includes at least one output node 108. In at least one embodiment, output nodes 108 are classification output nodes . In [0052]: In at least one embodiment, neural network 100 is used to identify out-of-distribution (OOD) input data by producing an output at second set of output nodes 106. In [0052]: In at least one embodiment, neural network 100 is used during inferencing to identify OOD input data as unknown objects In [0057]: In at least one embodiment, technique 300 includes, at a block 302, training a first portion of a neural network on a first set of data. In at least one embodiment, first set of data is an in-distribution training set such as IND training set 206 of FIG. 2. In at least one embodiment, training first portion includes training first portion to classify in-distribution input data to greater than a first predefined classification metric. In [0057]: In at least one embodiment, said first predefined classification metric is based on at least one of a false posive rate (FPR) and a true positive rate (TPR), such as a FPR at a TPR of 0.95, or any other suitable FPR and TPR relationship. In [ 0060]: In at least one embodiment, pseudo-labels assigned at block 306 are different than labels associated with an IND training set. In at least one embodiment, IND labels include numbers from 0 to 99 for a one hundred class classifier, and pseudo-labels associated an OOD training set include numbers from 100 to a predetermined number greater than 100. In at least one embodiment, technique 300 keeps some in-distribution samples in each mini-batch while training OOD samples at block 306 so neural network model does not forget in-distribution features learned at block 302 while training OOD detectors at block 306. In at least one embodiment, a predetermined ratio of IND samples to OOD samples is used during training OOD detectors, such as a ratio of one IND sample to five OOD samples, or a ratio of one IND sample to four OOD samples. In at least one embodiment, a ratio of IND samples to OOD samples is progressively altered during training OOD detectors from a predetermined initial ratio to a predetermined final ratio. (BRI: a predetermined ratio of in-distribution and out-of-distribution samples during training can influence how the training samples are split) In regard to claim 15 : (Currently Amended) Mohseni d iscloses: - train each of a plurality of models to output labels based on a first set of training data comprising a first set of plurality of features and a corresponding label; In [0106]: train one or more machine learning models or predict or infer information using one or more machine learning models according to one or more embodiments described herein in [0106]: in at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information in [0108]: Inference and/or training logic 915 are used to perform inferencing and/or training operations associated with one or more embodiments In [0096]: In at least one embodiment, semi-supervised learning may be used, which is a technique in which in training dataset 1002 includes a mix of labeled and unlabeled data. In [0055]: In at least one embodiment, first set of output nodes 212 are classifier functions. In [0055]: In at least one embodiment, first set of output nodes 212 learn target label s from IND training set 206 during training. (BRI: target label is the “output label”) input second sets of values of each of plurality of features into each of the plurality of trained models to generate, for each of the second sets of values, a plurality of output labels; In [0106]: i n at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1100 by using weight parameters calculated through one or more training techniques described herein. In [0062]: in at least one embodiment, second portion is trained to cluster unlabeled OOD samples using rejector functions in a manner such that first portion maintains in-distribution classification performance in [ 0055]: In at least one embodiment, first set of output nodes 212 are classifier functions. In at least one embodiment, first set of output nodes 212 are output nodes for a different type of machine learning technique In [0055]: In at least one embodiment, first set of output nodes 212 learn target labels from IND training set 206 during training. in [0051]: In at least one embodiment, target labels are ground truth data that indicate what classifier functions are to infer. (BRI: A target label is an output label) - determine, for each of the second set of values, pseudo-label based on the plurality of output labels generated for the second set of values In [0059]: In at least one embodiment, third set of data is unlabeled. In at least one embodiment, third set of data includes inputs that lack corresponding ground truth data (e.g., labels, annotations, or metadata) that indicate what second portion is to infer. In at least one embodiment, technique 300 includes assigning pseudo-labels to OOD training samples at block 306. In [0106]: i n at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1100 by using weight parameters calculated through one or more training techniques described herein. In [0062]: in at least one embodiment, second portion is trained to cluster unlabeled OOD samples using rejector functions in a manner such that first portion maintains in-distribution classification performance - determine a second set of training data comprising a second plurality of pairs, each of the second plurality of pairs comprising one of the second sets of values and a pseudo-label determined for the one of the second sets of values In [0060]: In at least one embodiment, first portion of neural network and second portion of neural network are trained with an automated technique. In at least one embodiment, first portion of neural network and second portion of neural network are trained using a machine learning (ML) approach to dynamically determine at least one of a ratio of IND samples to OOD samples, a learning rate, a dropout rate, and/or any other suitable parameters or hyperparameters in accordance with at least one performance metric. (BRI: the OOD sample has pseudo-labels. The mix of the OOD and IND is via ratio) In [0094]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, In [0094]: In at least one embodiment, untrained neural network 1006 is trained in a supervised manner processes inputs from training dataset 1002 and compares resulting outputs against a set of expected or desired outputs - train a first inference model and a second inference model having fewer free parameters than the first inference model to each output an inferred label based on the first set of training data and the second set of training data by: In [0054]: model architecture 200 includes a neural network 202 and training data sets 204. In at least one embodiment, training data sets 204 include an in- distribution (IND) training data set 206 and an out-of- distribution (OOD) training data set 208, In [0072]: performing other actions at block 510 includes leveraging an ensemble of multiple DNN blocks, In [0094]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, In [0051]: classifier functions learn target labels from a labeled in-distribution training set during a supervised learning technique, In [ 0094]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input paired with a desired output for an input, In [0061]: a ratio of IND samples to OOD samples is progressively altered during training OOD detectors from a predetermined initial ratio to a predetermined final ratio, in [0060: dynamically determine at least one of a ratio of IND samples to OOD samples, a learning rate, a dropout rate, and/or any other suitable parameters or hyperparameters in accordance with at least one performance metric. (BRI: a setting that can be adjusted to fine-tune the model's predictions based on learning. The learning rate or a dropout rate is a hyperparameter learned by altering the ratio) Mohseni does not explicitly disclose: A non-transitory medium storing processor-executable program code executable by a processing unit of a computing system to cause the computing system to: input of a plurality of the first sets of values and a plurality of the second sets of values to the first inference model and to the second inference model; reception of first labels output from the first inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; reception of second labels output from the first inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; determination of a first difference between the labels between the first labels output from the inference model and the labels corresponding to the plurality of the first sets of values and the pseudo-labels determined for the plurality of the second sets of values; determination of a second difference between the labels between the first labels output from the inference model and the labels corresponding to the plurality of the first sets of values and the pseudo-labels determined for the plurality of the second sets of values; a third difference between the second labels output from the second inference model and the labels corresponding to the plurality of the first sets of values and the pseudo-labels determined for the plurality of the second sets of values; modification of the first inference model based on the first difference and modification of the second inference model based on the second difference and the third difference However, Brown discloses: - A non-transitory medium storing processor-executable program code executable by a processing unit of a computing system to cause the computing system to: In [0079] input of a plurality of the first sets of values and a plurality of the second sets of values to the first inference model and to the second inference model; in [0082]: In at least one embodiment, a system performing at least a part of process 500 includes executable code to update 508 a model being trained using training data, labels, augmented training data, and pseudo-labels. In at least one embodiment, an updated model is used to infer data labels from data input . in [0078]: FIG. 4 illustrates a diagram 400 of generating pseudo-labels 406 to train neural networks, according to at least one embodiment. In at least one embodiment, a model 408 is trained over one or more iterations 410, where at each round of training, weights of said model 408 are adjusted to cause said model 408 to make inference s that match labels and pseudo-labels of data used to train said model 408. In at least one embodiment, GPU determines, for a model 408 that is trained using training data, a set of weights. (BRI: model 408 is an inference model) reception of first labels output from the first inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; in [0060]: in at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ense mble consistency regularization in [0113]: machine learning models or predict or infer information using one or more machine learning models (BRI: Within the context of ensemble models, the first label and second label output may correspond to use of respective predicted labeled output ) in [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training data set 1002 includes an input paired with a desired output for an input in [0080]: In at least one embodiment, a training set comprises image data and a corresponding set of labels, which is also referred to as annotations. In at least one embodiment, a sub set of training data comprises an image and a label (where a label corresponds to text data classifying contents of image). In at least one embodiment, a system trains a model using an image and a corresponding label. In at least one embodiment, a label is text data, audio data, or metadata associated with an image in [0082]: In at least one embodiment, a system performing at least a part of process 500 includes executable code to update 508 a model being trained using training data, labels, augmented training data, and pseudo-labels. In at least one embodiment, an updated model is used to infer data labels from data input . (BRI: the desired output is the label output) reception of first labels output from the first inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; in [0060]: in at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ense mble consistency regularization in [0113]: machine learning models or predict or infer information using one or more machine learning models (BRI: Within the context of ensemble models, the first label and second label output may correspond to use of respective predicted labeled output ) in [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training data set 1002 includes an input paired with a desired output for an input in [0082]: In at least one embodiment, a system performing at least a part of process 500 includes executable code to update 508 a model being trained using training data, labels, augmented training data, and pseudo-labels. In at least one embodiment, an updated model is used to infer data labels from data input . (BRI: the desired output is the label output) - reception of second labels output from the second inference model in response to the input of the plurality of the first sets of values and the plurality of the second sets of values; In [0106]: i n at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1100 by using weight parameters calculated through one or more training techniques described herein. determination of a first difference between the labels between the first labels output from the inference model and the labels corresponding to the plurality of the first sets of values and the pseudo-labels determined for the plurality of the second sets of values; In [0011]: FIG. 9A illustrates inference and/or training logic, according to at least one embodiment In [0060]: In at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ensemb le consistency regularization in [0113]: machine learning models or predict or infer information using one or more machine learning models (BRI: Within the context of ensemble models, the first label and second label output may correspond to use of respective predicted labeled output ) In [0006]: FIG. 4 illustrates a diagram of generating pseu do-labels to train neural networks, according to at least one embodiment; In [0067]: In at least one embodiment, GPU processes an image (e.g., image of an airplane) from content input 202 using an average model 204 to make a prediction on image 202 and generate an output 206. In at least one embodiment, output 206 comprises a pseudo-label. (BRI: output 206 is the label output say “X”) in[ 0067]: FIG. 2 illustrates another diagram 200 of determining a model and training scheme for image classification In [0070]: in at least one embodiment, techniques described herein combines a noise-robust task loss with augmentation, pseudo-labeling, and consistency regularization. In at least one embodiment, techniques described herein leverages generalized cross entropy GCE (which is a theoretically grounded noise-robust loss function that can be seen as a generalization of mean absolute error (MAE) (BRI: MAE is the loss considered as first difference between label output and labels associated with regularization application of set of inputs where first difference is within the context of first inference model using one or more models [See 0113]) in [0069]: a consistency regularization term is applied where output s from another model 208 are made to agree with output 206 in [0062]: a neural network classifies input data by inferring one or more labels for each datum of a set of input data, such as by inferring from an image an object appearing in said image. (BRI: with consistency regularization, particularly when combined with pseudo-labeling , the regularization technique forces the model's outputs to be consistent with its own predictions, effectively using its own outputs (pseudo-labels) as plurality of series of values for unlabeled data to guide the training process) In [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, or where training dataset 1002 includes input having a known output, - determination of a second difference between the labels between the first labels output from the second inference model In [0011]: FIG. 9A illustrates inference and/or training logic, according to at least one embodiment In [0060]: In at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ensemb le consistency regularization in [0113]: machine learning models or predict or infer information using one or more machine learning models (BRI: Within the context of ensemble models, the first label and second label output may correspond to use of respective predicted labeled output) In [0006]: FIG. 4 illustrates a diagram of generating pseu do-labels to train neural networks, according to at least one embodiment; In [0067]: In at least one embodiment, GPU processes an image (e.g., image of an airplane) from content input 202 using an average model 204 to make a prediction on image 202 and generate an output 206. In at least one embodiment, output 206 comprises a pseudolabel. (BRI: output 206 is the label output) in[ 0067]: FIG. 2 illustrates another diagram 200 of determining a model and training scheme for image classification In [0070]: in at least one embodiment, techniques described herein combines a noise-robust task loss with augmentation, pseudo-labeling, and consistency regularization. In at least one embodiment, techniques described herein leverages generalized cross entropy GCE (which is a theoretically grounded noise-robust loss function that can be seen as a generalization of mean absolute error (MAE) (BRI: MAE is the loss as a result of second difference between label output from the first inference model and second label output from the second inference model where first and second inference model are within the context of one or more models [See 0113]) in [0069]: a consistency regularization term is applied where output s from another model 208 are made to agree with output 206 in [0062]: a neural network classifies input data by inferring one or more labels for each datum of a set of input data, such as by inferring from an image an object appearing in said image. (BRI: with consistency regularization, particularly when combined with pseudo-labeling , the regularization technique forces the model's outputs to be consistent with its own predictions, effectively using its own outputs (pseudo-labels) as plurality of series of values for unlabeled data to guide the training process) In [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, or where training dataset 1002 includes input having a known output, and a third difference between the second labels output from the second inference model and the labels corresponding to the plurality of the first sets of values and the pseudo-labels determined for the plurality of the second sets of values; In [0011]: FIG. 9A illustrates inference and/or training logic, according to at least one embodiment In [0060]: In at least one embodiment, improvised image classification accuracy is achieved by modulating (e.g., varying, adjusting) a number of ensemb le consistency regularization in [0113]: machine learning models or predict or infer information using one or more machine learning models In [0006]: FIG. 4 illustrates a diagram of generating pseu do-labels to train neural networks, according to at least one embodiment; In [0067]: In at least one embodiment, GPU processes an image (e.g., image of an airplane) from content input 202 using an average model 204 to make a prediction on image 202 and generate an output 206. In at least one embodiment, output 206 comprises a pseudo-label. (BRI: output 206 is the label output) in[ 0067]: FIG. 2 illustrates another diagram 200 of determining a model and training scheme for image classification In [0070]: in at least one embodiment, techniques described herein combines a noise-robust task loss with augmentation, pseudo-labeling, and consistency regularization. In at least one embodiment, techniques described herein leverages generalized cross entropy GCE (which is a theoretically grounded noise-robust loss function that can be seen as a generalization of mean absolute error (MAE) (BRI: MAE is the loss as a result of third difference between label output and labels associated with regularization application of set of inputs where the third difference associates to a loss and use a second inference model within the context of plurality of models [0113])) in [0069]: a consistency regularization term is applied where output s from another model 208 are made to agree with output 206 in [0062]: a neural network classifies input data by inferring one or more labels for each datum of a set of input data, such as by inferring from an image an object appearing in said image. (BRI: with consistency regularization, particularly when combined with pseudo-labeling , the regularization technique forces the model's outputs to be consistent with its own predictions, effectively using its own outputs (pseudo-labels) as plurality of series of values for unlabeled data to guide the training process) In [0102]: In at least one embodiment, untrained neural network 1006 is trained using supervised learning, wherein training dataset 1002 includes an input pair ed with a desired output for an input, or where training dataset 1002 includes input having a known output, In [0070]: in at least one embodiment, techniques described herein combines a noise-robust task loss with augmentation, pseudo-labeling, and consistency regularization. In at least one embodiment, techniques described herein leverages generalized cross entropy GCE (which is a theoretically grounded noise-robust loss function that can be seen as a generalization of mean absolute error (MAE) (BRI: MAE is the loss as a result of third difference between label output and labels associated with regularization application of set of inputs where the third difference associates to a loss and use a second inference model within the context of plurality of models [0113])) In [0085]: in at least one embodiment, average model is used 604 to train a student model. In at least one embodiment, a student model is a task loss model that processes one or more augmented images. In [0102]: In at least one embodiment, training framework 1004 trains untrained neural network 1006 repeatedly while adjust weights to refine an output of untrained neural network 1006 using a loss function and adjustment algorithm, such as stochastic gradient descent. In at least one embodiment, training framework 1004 trains untrained neural network 1006 until untrained neural network 1006 achieves a desired accuracy. In at least one embodiment, trained neural network 1008 can then be deployed to implement any number of machine learning operations. In [0059]: pseudo-labels that are generated are assigned to modified training data, such as images that have been augmented in various ways. In at least one embodiment, training a neural network in this way simultaneously utilizes both task loss (updating weights of said neural network so that said neural network infers from datum a label that was provided with said datum) and consistency loss (updating weights of said neural network so that said neural network infers a corresponding pseudo-label consistently for multiple augmentations of said datum). In at least one embodiment, a number of augmentations used in training is increased as training progresses. (BRI: the above is weighted loss providing third difference from the first and second loss(differences) within the context of applying multiple augmentation) modification of the first inference model based on the first difference in [0081]: generate 506 modified version of first version of training data comprises augmented training data. In at least one embodiment, a system causes a data processing service to perform a modification on training data to modify training data. (BRI: modifying the training data can modify the model through processes like retraining or fine-tuning and may influence model's learning) In [0502]: in at least one embodiment, once a model is trained—or partially trained—at one location, a machine learning model may be added to model registry 3824. In at least one embodiment, a machine learning model may then be retrain ed, or updated, at any number of other facilities, and a retrain ed or updated model may be made available in model registry 3824, in [0559]: In at least one embodiment, to retrain, or update, initial model 4204, output or loss layer(s) of initial model 4204 may be reset, or deleted, and/or replaced with an updated or new output or loss layer(s). (BRI: updating the loss layer may modify and retrain a model. The loss function's explicit purpose is to guide the optimization process) and modification of the second inference model based on the second difference and the third difference in [0081]: generate 506 modified version of first version of training data comprises augmented training data. In at least one embodiment, a system causes a data processing service to perform a modification on training data to modify training data. (BRI: modifying the training data can modify the model through processes like retraining or fine-tuning and may influence model's learning) In [0502]: in at least one embodiment, once a model is trained—or partially trained—at one location, a machine learning model may be added to model registry 3824. In at least one embodiment, a machine learning model may then be retrain ed, or updated, at any number of other facilities, and a retrain ed or updated model may be made available in model registry 3824, in [0559]: In at least one embodiment, to retrain, or update, initial model 4204, output or loss layer(s) of initial model 4204 may be reset, or deleted, and/or replaced with an updated or new output or loss layer(s). (BRI: updating the loss layer may modify and retrain a model. The loss function's explicit purpose is to guide the optimization process) It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, and Brown. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. One of ordinary skill would have motivation to combine Mohseni, and Brown that can provide a measure of confidence interpreted as a probability of detections of objects in an image (classification)(Brown [0164]). Mohseni and Brown do not explicitly disclose: - modification of free parameters of the first inference model based on the first difference and not based on the second difference and the third difference; - and modification of free parameters of the second inference model based on the second difference and the third difference, and not based on the first difference However, Xu discloses: - modification of free parameters of the first inference model based on the first difference and not based on the second difference and the third difference; [Col 11, lines 36-40]: in one or more embodiments, the lead management system 102 uses weights of a fuzzy augmentation model to update a previously generated loss function (e.g., for optimization-based algorithms like logistic regression) [Col 11, lines 45-54]: The lead management system 102 can then use the loss function 216 to generate a trained lead scoring model 218 . Specifically, the lead management system 102 uses the function(s) and/or value(s) of the loss function 216 to modify one or more parameters or algorithms of the lead scoring model 200. Modifying parameters or algorithms of the lead scoring model 200 results in the trained lead scoring model 218 that takes into account the synthetic outcomes 214 of the imputed dataset 212 in addition to the actual outcomes 208 of the original dataset 202. [Col 10, lines 9-18]: The lead management system 102 uses a selected reject inference model 210 to generate synthetic data based on the rejected leads 206 of the original dataset 202. As briefly mentioned previously, the reject inference model 210 can be a simple augmentation model or a fuzzy augmentation model. Each of the reject inference models generates synthetic data using different operations, and therefore result in different synthetic data. Furthermore, the reject inference models can also output different amounts and/or types of synthetic data. [Col 2, lines 54-60]: The lead management system selects a reject inference model for generating the imputed dataset based on a plurality of simulations on historical prospect data . The lead management system then uses the selected reject inference model to generate an imputed dataset including synthetic outcome data representing simulated outcomes for the reject data. [Col 3, lines 26-29]: The synthetic outcome data can include a label indicating the outcome of a given rejected prospect, and in some cases a weighting/score, depending on the reject inference model that the lead management system uses. [Col 10, lines 19-20]: In one or more embodiments, the reject inference model 210 builds upon a known good- bad (“KGB”) model. I n particular, a KGB models (used by some conventional models for training a scoring model, typically in credit scoring applications) uses only the known “good” labels and known “bad” labels , thereby only using known labels (while excluding unknown labels in the reject data) to train scoring models. Thus, the KGB model utilizes only leads with known outcomes , which may include only accepted leads 204 in at least some embodiments. For embodiments in which the known labels include at least some rejected leads with outcomes , the KGB model may also be trained on the reject data with outcomes (i.e., a subset of rejected leads that have engagement and outcome information). only the accepted leads are engaged and only labels for A are available, the KGB model is developed based on (X.sub.A, y.sub.A) by dividing A into pseudo -accepted leads and pseudo-rejected leads. (BRI : in the context of selected reject inference for lead scoring, generating synthetic data from rejected leads, and using this synthetic dataset to modify or retrain the lead scoring model by adjustig its parameter does represents modifying free parameters of the inference model) [Col 14, lines 47-56]: apply a KGB model that uses only the known labels. If only the accepted leads are engaged and only labels for A are available, the KGB model is developed based on (X.sub.A, y.sub.A) by dividing A into pseudo-accepted leads and pseudo-rejected leads . Because the KGB model utilizes only known labels, however the resulting performance metric is likely to be biased. Additionally, the KGB model may not split the accepted leads into pseudo -accepted/ pseudo -rejected leads (BRI using a model’s predictions to assign provisional labels to unlabeled data and then including those as part of the training set represents performing pseudo-labeling . The term “pseudo accepted leads” ( or “pseudo accepted labels ”) is essentially a rephrasing of “ pseudo-labels ,” where the labels are not ground truth but are generated by the model itself ) [Col 11, lines 25-40]: the process of generating synthetic data for updating a lead scoring model includes using the imputed dataset 212 to optionally generate a loss function 216 . The loss function 216 indicates a difference between the lead scoring model 200 output (e.g., the scores as predictions of successful outcomes) and the actual outcomes (e.g., outcomes 208 ). To generate the loss function 216, the lead management system 102 compares the resulting outcomes to the predicted outcomes based on the scores from the lead scoring model 200. The loss function 216 can include one or more functions or values representing the difference(s) between the predicted and actual values. In one or more embodiments, the lead management system 102 uses weights of a fuzzy augmentation model to update a previously generated loss function (e.g., for optimization-based algorithms like logistic regression). [Col 12, lines 21-34]: A process for determining whether to generate an imputed dataset includes comparing characteristics of a lead scoring model 300 to a plurality of thresholds. The process begins by the lead management system 102 determining whether a scoring split of the lead scoring model 300 meets a scoring split threshold 302. For instance, the lead management system 102 can analyze the original dataset to determine how accurately the lead scoring model 300 labeled the accepted leads and the rejected leads . To illustrate, the lead management system 102 can determine whether the lead scoring model 300 is accurately indicating that the leads that are most likely to result in a successful outcome are labeled as accepted lead s, while those that are not likely to result in a successful outcome are labeled as rejected leads. [Col 14, lines 61-67]: the lead management system 102 determines an estimate for the area under the receiver operating characteristic curve (i.e., area under ROC curve or simply “AUC”). In particular, the lead management system 102 first estimates the overall true positive rate (“TPR”) and overall false positive rate (“FPR”) for the original dataset. The TPR and FPR may be reasonably estimated via [Col 15, lines 1-2]: reweighting if the lead management system 102 obtains partial rejects via random sampling. [Col 15, lines 3-7]: The overall TPR is a weighted average of the TPR for accepted leads and the TPR for rejected leads. Similarly, the overall FPR is a weighted average of the FPR for accepted leads and the FPR for rejected leads. In particular, the overall TPR is represented as: PNG media_image1.png 32 202 media_image1.png Greyscale [Col 15, lines 14-19]: In which PNG media_image2.png 40 197 media_image2.png Greyscale (BRI: the approach is a form of threshold based model validation and not limited to first or second difference. In this context, the first difference is the change in metric TPR as moved from the threshold the next) - and modification of free parameters of the second inference model based on the second difference and the third difference, and not based on the first difference. [Col 13, lines 57-67]: the lead management system 102 can process the dataset for comparing to threshold s in a single operation or in multiple operations. For example, the lead management system 102 can determine the split effectiveness, success rate, and size simultaneously during a single group of simulations for the original dataset . Alternatively, the lead management system 102 can perform the operations of FIG. 3 in any order and may exclude one or more of the thresholds or include one or more additional thresholds based on the characteristics associated with the original dataset. [Col 15, lines 43-47]: If the split of the original dataset is effective, it is likely that r.sub.A≥r.sub.R. The observed number of “1” labels in the rejected leads, however, is at least the total number of “1” labels in the reje cted leads. Because sub.R.sub.sub.sup.+≤n.sub.R.sup.+=(r.sub.R)(n.sub.R)≤(r.sub.A)(n.sub.R), . a conserva tive estimate of TPR.sub.overall is TPR.sub.overall:=min (A, B) where [Col 15, lines 50-55]: PNG media_image3.png 56 581 media_image3.png Greyscale PNG media_image4.png 63 520 media_image4.png Greyscale (BRI: the approach is a form of threshold based model validation and not limited to first or second difference. In this context, the second difference is the change in the first difference indicating an improvement either TPR slows down or speed as the threshold increases. Changing threshold impact the TPR. Excluding thresholds increase TPR are overlay strict and adding thresholds decrease TPR if the threshold is too lenient (more false positive)) [Col 26, lines 36-44]: The lead management system 102 includes a lead scoring model manager 708 to facilitate the generation of scores for leads associated with an entity. For example, the lead scoring model manager 708 can utilize a lead scoring model to generate scores for leads by analyzing features of the leads and then ranking the leads on a scale. The lead scoring model manager 708 can also assign the leads labels indicating an accepted or rejected lead based on a predetermined threshold . [Col 12, lines 55-58]: the reject rate of the original dataset is small with a small mislabel rate . For example, a small reject rate indicates that a small number of unengaged rejected leads are incorrectly labeled. [Col 15, lines 36-41]: Information on whether a lead has been engaged may be missing for certain bases. For instance, failed engaged leads and unengaged rejected leads may be labeled as “0” because of a negative outcome or negative engagement. Accordingly, unengaged rejected leads that are labeled as “1” are mislabeled. (BRI: A small rejection rate means most leads are being accepted. If those accepted leads are mostly unengaged (i.e., they are actually negative cases but were mislabeled as positive), then the proportion of such misclassifications among the accepted set will be high — and that will directly increase the FPR. It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, Brown and Xu. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. Xu teaches updating the free parameters in an inference model. One of ordinary skill would have motivation to combine Mohseni, Brown and Xu that improve the performance of the lead scoring model for future datasets)( Xu [Col 3, lines 36-41]) In regard to claim 16 : (Previously Presented) Mohseni does not explicitly disclose: - wherein each of the plurality of models conforms to hyperparameters which are different from hyperparameters to which each other of the plurality of models conforms - and is trained using training parameters different from hyperparameters used to which each of the plurality of models conforms However, Brown discloses: - wherein each of the plurality of models conforms to hyperparameters which are different from hyperparameters to which each other of the plurality of models conforms In [0090]: In at least one embodiment, inference and/or training logic 915 may include, without limitation, code and/or data storage 901 to store forward and/or output weight and/or input/output data, and/or other parameter s to configure neurons or layers of a neural network trained and/or used for inferencing in aspects of one or more embodiments (BRI: parameters that are used to configure a neural network that are set before the training process begins. They control the learning process and the architecture of the model) In [0090]: in at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on an architecture of a neural network to which such code corresponds. In at least one embodiment, code and/or data storage 901 stores weight parameter s and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input/output data and/or weight parameter s during training and/or inferencing using aspects of one or more embodiments. (BRI: the weight is a model parameter which is different from hyperparameters) - and is trained using training parameters different from hyperparameters used to which each of the plurality of models conforms In [0102]: n at least one embodiment, untrained neural network 1006 is trained in a supervised manner and processes inputs from training dataset 1002 and compares resulting outputs against a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through untrained neural network 1006. In at least one embodiment, training framework 1004 adjust s weights that control untrained neural network 1006. In at least one embodiment, training framework 1004 includes tools to monitor how well untrained neural network 1006 is converging towards a model, such as trained neural network 1008, (BRI: Refers to training data , which is the dataset used to adjust a model's parameters to recognize patterns or make decisions) It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, and Brown. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. One of ordinary skill would have motivation to combine Mohseni, and Brown that can provide a measure of confidence interpreted as a probability of detections of objects in an image (classification)(Brown [0164]). In regard to claim 17 : (Original) Mohseni does not explicitly disclose: - wherein determination of a pseudo-label for a feature comprises determining an average of the plurality of output labels generated for the feature. However, Brown discloses: - wherein determination of a pseudo-label for a feature comprises determining an average of the plurality of output labels generated for the feature. In [0065]: In at least one embodiment, GPU 108 receives training data 102 and uses average model 110 to process training data 102. In at least one embodiment, GPU 108 uses average model 110 to make a prediction on an image from a first version of training data 102. In at least one embodiment, an average model 110 is generated by averaging weights from one or more previous versions of one or more neural networks. In at least one embodiment, previous versions of one or more neural networks include weights from each training round (e.g., training iteration, training step) after said one or more neural networks processes training data 102. In [0086] : In at least one embodiment, a system determines that training will continue as an average model can still be improved. In at least one embodiment, as more modified training data is processed by student model, an average model is updated with more information from student model. In at least one embodiment, teacher model is continually made stronger and is trained to infer same information from multiple different modifications of same input. In at least one embodiment, as training continues, data processing service provides additional augmented images to student model for training. In [0066]: In at least one embodiment, each round of training updates weights so that a trained model matches an image to a label, said trained model then matches image to a pseudo-label that is generated from average model based on image input, and then further matches augmented images to pseudo-labels generated by average models based on one or more augmented images. (BRI: a model that uses an average of its weights (often called a "teacher" model) to generate pseudo-labels based on features of training data will produce a more stable and averaged output label than a single model trained from scratch ) It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, and Brown. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. One of ordinary skill would have motivation to combine Mohseni, and Brown that can provide a measure of confidence interpreted as a probability of detections of objects in an image (classification)(Brown [0164]). In regard to claim 19 : (Previously Presented) Mohseni discloses: wherein the plurality of the first sets of values and the plurality of the second sets of values comprise a fixed ratio of pairs from the first set of training data and pairs from the second set of training data. In [0052]: neural network 100 includes a first portion 112 that includes first set of output nodes 104. In at least one embodiment, neural network 100 includes a second portion 114 that includes second set of output nodes 106. In [0051]: in at least one embodiment, output layer 102 includes a first set of output nodes 104 and a second set of output nodes 106. In at least one embodiment, first set of output nodes 104 includes at least one output node 108. In at least one embodiment, output nodes 108 are classification output nodes . In [0052]: In at least one embodiment, neural network 100 is used to identify out-of-distribution (OOD) input data by producing an output at second set of output nodes 106. In [0052]: In at least one embodiment, neural network 100 is used during inferencing to identify OOD input data as unknown objects In [0057]: In at least one embodiment, technique 300 includes, at a block 302, training a first portion of a neural network on a first set of data. In at least one embodiment, first set of data is an in-distribution training set such as IND training set 206 of FIG. 2. In at least one embodiment, training first portion includes training first portion to classify in-distribution input data to greater than a first predefined classification metric. In [0057]: In at least one embodiment, said first predefined classification metric is based on at least one of a false posive rate (FPR) and a true positive rate (TPR), such as a FPR at a TPR of 0.95, or any other suitable FPR and TPR relationship. In [ 0060]: In at least one embodiment, pseudo-labels assigned at block 306 are different than labels associated with an IND training set. In at least one embodiment, IND labels include numbers from 0 to 99 for a one hundred class classifier, and pseudo-labels associated an OOD training set include numbers from 100 to a predetermined number greater than 100. In at least one embodiment, technique 300 keeps some in-distribution samples in each mini-batch while training OOD samples at block 306 so neural network model does not forget in-distribution features learned at block 302 while training OOD detectors at block 306. In at least one embodiment, a predetermined ratio of IND samples to OOD samples is used during training OOD detectors, such as a ratio of one IND sample to five OOD samples, or a ratio of one IND sample to four OOD samples. In at least one embodiment, a ratio of IND samples to OOD samples is progressively altered during training OOD detectors from a predetermined initial ratio to a predetermined final ratio . 07-21-aia AIA Claim s 5, 12, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Sina Mohseni et.al. (hereinafter Mohseni ) US 2021/0142160 A1, in view of Abel Karl Brown et.al. (hereinafter Brown ) US 2022/0101112 A1, in view of Maoqi Xu et.al. (hereinafter Xu ) US 11514515 B2. further in view of Makoto Takamoto et.al. (hereinafter Takamoto) US 2024/0289633 A1. In regard to claim 5 : (Original) Mohseni, Brown and Xu do not explicitly disclose: - wherein determination of a pseudo-label for a feature comprises determining a most-often occurring value of the plurality of output labels generated for the feature. However, Takamoto discloses: - wherein determination of a pseudo-label for a feature comprises determining a most-often occurring value of the plurality of output labels generated for the feature. In [0089]: The pseudo-label evaluation unit 150 evaluates the pseudo-label by using the plurality of evaluation models 151. Specifically, the pseudo-label evaluation unit 150 first outputs an evaluation result from each of the plurality of evaluation models 151, and outputs one final evaluation result in accordance with the plurality of evaluation results. More specifically, an overall evaluation result may be outputted by majority vote of the respective evaluation results of the plurality of evaluation models 151, It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, Brown, Xu and Takamoto. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. Xu teaches updating the free parameters in an inference model. Takamoto teaches determination of most-often occurring values (majority vote). One of ordinary skill would have motivation to combine Mohseni, Brown , Xu and Takamoto that can improve the accuracy using adding a pseudo label to the unlabeled data (Takamoto [0050]) In regard to claim 12 : (Original) Mohseni, Brown and Xu do not explicitly disclose: - wherein determination of a pseudo-label for a feature comprises determining a most-often occurring value of the plurality of output labels generated for the feature. However, Takamoto discloses: - wherein determination of a pseudo-label for a feature comprises determining a most-often occurring value of the plurality of output labels generated for the feature. In [0089]: The pseudo-label evaluation unit 150 evaluates the pseudo-label by using the plurality of evaluation models 151. Specifically, the pseudo-label evaluation unit 150 first outputs an evaluation result from each of the plurality of evaluation models 151, and outputs one final evaluation result in accordance with the plurality of evaluation results. More specifically, an overall evaluation result may be outputted by majority vote of the respective evaluation results of the plurality of evaluation models 151, It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, Brown, Xu and Takamoto. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. Xu teaches updating the free parameters in an inference model. Takamoto teaches determination of most-often occurring values (majority vote). One of ordinary skill would have motivation to combine Mohseni, Brown , Xu and Takamoto that can improve the accuracy using adding a pseudo label to the unlabeled data (Takamoto [0050]) In regard to claim 18 : (Original) Mohseni, Brown and Xu do not explicitly disclose: - wherein determination of a pseudo-label for a feature comprises determining a most-often occurring value of the plurality of output labels generated for the feature. However, Takamoto discloses: - wherein determination of a pseudo-label for a feature comprises determining a most-often occurring value of the plurality of output labels generated for the feature. In [0089]: The pseudo-label evaluation unit 150 evaluates the pseudo-label by using the plurality of evaluation models 151. Specifically, the pseudo-label evaluation unit 150 first outputs an evaluation result from each of the plurality of evaluation models 151, and outputs one final evaluation result in accordance with the plurality of evaluation results. More specifically, an overall evaluation result may be outputted by majority vote of the respective evaluation results of the plurality of evaluation models 151, It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Mohseni, Brown, Xu and Takamoto. Mohseni teaches plurality of training data with labels and pseudo labels and features. Brown teaches a system to perform the training within the context of pseudo-labeling and determining the determination of difference between the labels output corresponding to the pseudo labels. Xu teaches updating the free parameters in an inference model. Takamoto teaches determination of most-often occurring values (majority vote). One of ordinary skill would have motivation to combine Mohseni, Brown , Xu and Takamoto that can improve the accuracy using adding a pseudo label to the unlabeled data (Takamoto [0050]) Conclusion 07-40 AIA Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL . See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIRUMALE KRISHNASWAMY RAMESH whose telephone number is (571)272-4605. The examiner can normally be reached by phone. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached on phone (571-272-3768). The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /TIRUMALE K RAMESH/Examiner, Art Unit 2121 /Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121 Application/Control Number: 17/469,075 Page 2 Art Unit: 2121 Application/Control Number: 17/469,075 Page 3 Art Unit: 2121 Application/Control Number: 17/469,075 Page 4 Art Unit: 2121 Application/Control Number: 17/469,075 Page 5 Art Unit: 2121 Application/Control Number: 17/469,075 Page 6 Art Unit: 2121 Application/Control Number: 17/469,075 Page 7 Art Unit: 2121 Application/Control Number: 17/469,075 Page 8 Art Unit: 2121 Application/Control Number: 17/469,075 Page 9 Art Unit: 2121 Application/Control Number: 17/469,075 Page 10 Art Unit: 2121 Application/Control Number: 17/469,075 Page 11 Art Unit: 2121 Application/Control Number: 17/469,075 Page 12 Art Unit: 2121 Application/Control Number: 17/469,075 Page 13 Art Unit: 2121 Application/Control Number: 17/469,075 Page 14 Art Unit: 2121 Application/Control Number: 17/469,075 Page 15 Art Unit: 2121 Application/Control Number: 17/469,075 Page 16 Art Unit: 2121 Application/Control Number: 17/469,075 Page 17 Art Unit: 2121 Application/Control Number: 17/469,075 Page 18 Art Unit: 2121 Application/Control Number: 17/469,075 Page 19 Art Unit: 2121 Application/Control Number: 17/469,075 Page 20 Art Unit: 2121 Application/Control Number: 17/469,075 Page 21 Art Unit: 2121 Application/Control Number: 17/469,075 Page 22 Art Unit: 2121 Application/Control Number: 17/469,075 Page 23 Art Unit: 2121 Application/Control Number: 17/469,075 Page 24 Art Unit: 2121 Application/Control Number: 17/469,075 Page 25 Art Unit: 2121 Application/Control Number: 17/469,075 Page 26 Art Unit: 2121 Application/Control Number: 17/469,075 Page 27 Art Unit: 2121 Application/Control Number: 17/469,075 Page 28 Art Unit: 2121 Application/Control Number: 17/469,075 Page 29 Art Unit: 2121 Application/Control Number: 17/469,075 Page 30 Art Unit: 2121 Application/Control Number: 17/469,075 Page 31 Art Unit: 2121 Application/Control Number: 17/469,075 Page 32 Art Unit: 2121 Application/Control Number: 17/469,075 Page 33 Art Unit: 2121 Application/Control Number: 17/469,075 Page 34 Art Unit: 2121 Application/Control Number: 17/469,075 Page 35 Art Unit: 2121 Application/Control Number: 17/469,075 Page 36 Art Unit: 2121 Application/Control Number: 17/469,075 Page 37 Art Unit: 2121 Application/Control Number: 17/469,075 Page 38 Art Unit: 2121 Application/Control Number: 17/469,075 Page 39 Art Unit: 2121 Application/Control Number: 17/469,075 Page 40 Art Unit: 2121 Application/Control Number: 17/469,075 Page 41 Art Unit: 2121 Application/Control Number: 17/469,075 Page 42 Art Unit: 2121 Application/Control Number: 17/469,075 Page 43 Art Unit: 2121 Application/Control Number: 17/469,075 Page 44 Art Unit: 2121 Application/Control Number: 17/469,075 Page 45 Art Unit: 2121 Application/Control Number: 17/469,075 Page 46 Art Unit: 2121 Application/Control Number: 17/469,075 Page 47 Art Unit: 2121 Application/Control Number: 17/469,075 Page 48 Art Unit: 2121 Application/Control Number: 17/469,075 Page 49 Art Unit: 2121 Application/Control Number: 17/469,075 Page 50 Art Unit: 2121 Application/Control Number: 17/469,075 Page 51 Art Unit: 2121 Application/Control Number: 17/469,075 Page 52 Art Unit: 2121 Application/Control Number: 17/469,075 Page 53 Art Unit: 2121 Application/Control Number: 17/469,075 Page 54 Art Unit: 2121 Application/Control Number: 17/469,075 Page 55 Art Unit: 2121 Application/Control Number: 17/469,075 Page 56 Art Unit: 2121 Application/Control Number: 17/469,075 Page 57 Art Unit: 2121 Application/Control Number: 17/469,075 Page 58 Art Unit: 2121 Application/Control Number: 17/469,075 Page 59 Art Unit: 2121 Application/Control Number: 17/469,075 Page 60 Art Unit: 2121 Application/Control Number: 17/469,075 Page 61 Art Unit: 2121 Application/Control Number: 17/469,075 Page 62 Art Unit: 2121 Application/Control Number: 17/469,075 Page 63 Art Unit: 2121 Application/Control Number: 17/469,075 Page 64 Art Unit: 2121 Application/Control Number: 17/469,075 Page 65 Art Unit: 2121 Application/Control Number: 17/469,075 Page 66 Art Unit: 2121 Application/Control Number: 17/469,075 Page 67 Art Unit: 2121 Application/Control Number: 17/469,075 Page 68 Art Unit: 2121 Application/Control Number: 17/469,075 Page 69 Art Unit: 2121 Application/Control Number: 17/469,075 Page 70 Art Unit: 2121 Application/Control Number: 17/469,075 Page 71 Art Unit: 2121 Application/Control Number: 17/469,075 Page 72 Art Unit: 2121 Application/Control Number: 17/469,075 Page 73 Art Unit: 2121 Application/Control Number: 17/469,075 Page 74 Art Unit: 2121 Application/Control Number: 17/469,075 Page 75 Art Unit: 2121 Application/Control Number: 17/469,075 Page 76 Art Unit: 2121
Prosecution Insights