DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant claims the benefit of prior-filed a U.S. Provisional Application No. 63/209,282, filed on June 10, 2021, which is acknowledged.
Drawings
The drawings were received on 06/08/2022. These drawings are acceptable.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on the following date(s): 12/07/2022 and 06/08/2022 have been considered by the examiner.
Response to Arguments
Applicant's arguments filed 5/14/2026 have been fully considered.
Regarding applicant’s remarks directed to the rejection of claims under 35 USC 103, the remarks are directed to the amended subject matter that has not been examined. See the rejection of the amended claim limitations below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Lu et al. (US 20220343175, hereinafter ‘Lu’) in further view of Lai et al. (US 20210182662, hereinafter “Lai’) in view of Ji et al. (US 20220366263, hereinafter ‘Ji’).
Regarding independent claim 1, Lu teaches a method of training a student model, the method comprising: (in [0004] Knowledge Distillation (KD) is a compression technique used to transfer the knowledge of a large trained neural network model (i.e. a neural network model with many learned parameters) to a smaller neural network model (i.e. a neural network model with fewer learned parameters than the large trained neural network model). KD utilizes the generalization ability of the larger trained neural network model (referred to as the “teacher model” or “teacher”) using the inference data output by the larger trained model as “soft targets”, which are used as a supervision signal for training a smaller neural network model (called the “student model” or “student”)…)
providing an input to a teacher model that is larger than the student model, wherein a layer of the teacher model outputs a first output vector; providing the input to the student model, wherein a layer of the student model outputs a second output vector; (in [0004] Knowledge Distillation (KD) is a compression technique used to transfer the knowledge of a large trained neural network model (i.e. a neural network model with many learned parameters) to a smaller neural network model (i.e. a neural network model with fewer learned parameters than the large trained neural network model). KD utilizes the generalization ability of the larger trained neural network model (referred to as the “teacher model” or “teacher”) [providing an input to a teacher model that is larger than the student model] using the inference data output by the larger trained model as “soft targets”[ wherein a layer of the teacher model outputs a first output vector], which are used as a supervision signal for training a smaller neural network model (called the “student model” or “student”) [providing the input to the student model]….; And in [0016] In some aspects, the present disclosure provides a device, comprising a processor and a memory. The memory has stored thereon instructions which, when executed by the processor, cause the device to perform a number of operations. A batch of training data comprising one or more labeled training data samples is obtained. Each labeled training data sample has a respective ground truth label. The batch of training data is processed, using a student model comprising a plurality of learnable parameters, to generate, for input data in each data sample in the batch of training data, a student prediction. For each labeled training data sample in the batch of training data, the student prediction and the ground truth label are processed to compute a respective ground truth loss. The batch of training data is processed, using a trained teacher model, to generate, for each labeled training data sample in the batch of training data, a teacher prediction. For each labeled data sample in the batch of training data, the student prediction [providing the input to the student model, wherein a layer of the student model outputs a second output vector] and the teacher prediction [providing an input to a teacher model that is larger than the student model, wherein a layer of the teacher model outputs a first output vector] are processed to compute a respective knowledge distillation loss…
determining an importance value associated with each dimension of the first output vector based on gradients of the teacher model; (in [0011] In some examples, the reweighting module may determine a ground truth weight to emphasize a ground truth label of a training data sample, and a knowledge distillation weight [determining an importance value…] to emphasize the soft label from the teacher [determining an importance value associated with each dimension of the first output vector based on gradients of the teacher model], based on user input (such as input from an expert with insight into the characteristics of a given batch of training data). In some examples, the reweighting module may determine the ground truth weight and the knowledge distillation weight using a meta-reweighting process… Meta-reweighting is a meta-learning technique that uses machine learning to assign weights to training data samples based on their gradient directions [associated with each dimension of the first output vector based on gradients of the teacher model]. To determine the weights in the Ren reference, a meta gradient descent step is performed on existing training weights to minimize the generalization loss on a validation set… Examiner notes that a gradient is of the teacher model training data examples for making teaching model predictions for each vector direction based on their gradient direction is within the scope of claim limitations. Also see [0005] In KD, the student receives both soft targets and hard targets as supervision signals. This allows the student to achieve better performance after being trained on the same dataset as the teacher, as the soft targets provide higher entropy and less variance (i.e. better generalization) than the hard targets…)
and updating at least one parameter of the student model to minimize a difference between the second output vector and the first output vector based on the importance value, wherein the gradients represent respectively different weights corresponding to different importance values. (in [0067] KL divergence is used to measure the difference between the student predictions (i.e. the student predicted logits 306) and the teacher predictions (i.e. the teacher predicted logits 310). Minimizing the KL divergence, and therefore the KD loss 314 [updating at least one parameter of the student model to minimize a difference between the second output vector and the first output vector based on the importance value.], by adjusting the values of the learnable parameters of the student [and updating at least one parameter of the student model to minimize a difference between the second output vector and the first output vector] 234 should result in the student 234 learning to output student inference data 34 that is close to the teacher inference data 24. And in [0069] At 412, the gradient descent module 218 performs a gradient descent operation on the student 234 using the weighted loss function L to identify an adjusted set of values of the learnable parameters of the student 234. A gradient descent operation may be performed using any appropriate technique known in the field of machine learning to adjust each learnable parameter of the student 234, for example using backpropagation to perform gradient descent on each of the learnable parameters of the student 234. The gradient descent operation performed by the gradient descent module 218 is intended to compute or estimate a partial derivative of the value of each of the learnable parameters of the student 234 [and updating at least one parameter of the student model to minimize a difference between the second output vector and the first output vector] with respect to the weighted loss function L, using the chain rule as necessary to propagate the weighted loss 330 backward from the output nodes (e.g. an output layer of neurons) through the other nodes of the student 234. The adjusted values of the learnable parameters may be identified as values of the learned parameters that would result in a lower or minimized reweighted loss 330 [wherein the gradients represent respectively different weights corresponding to different importance values] with respect the current labeled training data sample x, or with respect to the entire batch of training data X …. [0076] At 804, the meta-reweighting module 510 determines an estimated optimized value of the knowledge distillation weight λ.sup.KD [wherein the gradients represent respectively different weights corresponding to different importance values as the knowledge distillation weight of the respective re-weighting stage process] and an estimated optimized value of the ground truth weight λ.sup.CE with respect to a second batch of data (in this example, a batch of validation data obtained from the validation dataset 250). This operation 804 may be referred to as a reweighting stage of the meta-reweighting process, and is described in greater detail below with reference to FIGS. 7 and 10.
Lu teaches the Knowledge Distillation training process using vector data, as noted above. Lu does not expressly teach the output information in vector form.
Lai does expressly teach the output information in vector form, in [0025] For example, a loss function generation module of the model training system generates a loss function L3 that is based on a comparison of the above discussed probability vectors Ps3 and Pt3 generated by the student and teacher models [providing an input to a teacher model that is larger than the student model, wherein a layer of the teacher model outputs a first output vector; providing the input to the student model, wherein a layer of the student model outputs a second output vector], respectively. An example loss function is L3=H (σ(t3/T), σ(s3/T)), where σ is the softmax function, t3 is a logit of the probability vector Pt3 […wherein a layer of the teacher model outputs a first output vector; providing the input to the student model, …], s3 is the logit of the probability vector Ps3 [… wherein a layer of the student model outputs a second output vector], T is a temperature hyperparameter of the teacher and/or the student models, and the function H(.) is a cross-entropy function. The logit function and this example loss function equation is discussed herein later in further detail.
Lai and Lu are analogous art because both involve developing information retrieval and processing techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for training a reduced scale model based using a full-scale model for executing machine learning tasks as disclosed by Lai with the method of developing information retrieval and processing task using knowledge distillation techniques as disclosed by Lu.
One of ordinary skill in the arts would have been motivated to combine the methods disclosed by Lai and Lu noted above; Doing so allowing for developing parameter tuning techniques that tunes the parameters of a student/smaller model, such that the student model mimics the behavior of a teacher/larger model, (Lai, 0029).
Lai and Lu teach the information processing methods based on distill knowledge techniques using the gradients of the teacher and student model learn the modeling parameters. One of ordinary skill would know that in training a student model using a teacher model a gradient is applied on the teacher model in a distill knowledge data processing technique using neural networks.
Ji expressly teaches a teacher model a gradient is applied on the teacher model in a distill knowledge data processing technique using neural networks, in [0062] After generating an instance of the teacher machine learning model 220 as described above and prior to using it to train the student model 210, a training engine 250 of the training system 200 trains the teacher model 220 [each dimension of the first output vector based on gradients of the teacher model] on the first task by using the labeled training data 240… This can prepare the teacher model 220 for training the student model 210 using knowledge distillation with a minimally required amount of labeled data that is specific to the first task… [0077] The system trains the teacher machine learning model on the first training data to obtain a trained teacher machine learning model (406). The system can do this by using conventional supervised learning techniques that rely on using the ground truth outputs as training objectives to determine trained parameter values of the teacher machine learning model. For example, if the teacher machine learning model is a neural network, the system can use a gradient descent with backpropagation technique [each dimension of the first output vector based on gradients of the teacher model] to iteratively adjust the values of the parameters of the teacher machine learning model.
Ji, Lai and Lu are analogous art because both involve developing information retrieval and processing techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for training a distilled student machine learning model using a neural network teacher machine learning model for executing machine learning tasks as disclosed by Ji with the method of developing information retrieval and processing task using knowledge distillation techniques as collectively disclosed by Lai and Lu.
One of ordinary skill in the arts would have been motivated to combine the methods disclosed by Ji, Lai and Lu noted above; Doing so allowing for developing a distilled student machine learning model that is easier to deploy than a cumbersome teacher machine learning model, i.e., because it requires less computation, memory, or both, to generate outputs at run time than the cumbersome teacher machine learning model, (Ji, 0012).
Regarding claim 2, the rejection of claim 1 is incorporated and Lu in combination with Lai teaches the method of claim 1, wherein further comprising determining a weighted knowledge distillation (WKD) loss based on the first output vector, the second output vector and the importance value, and wherein the at least one parameter of the student model is updated based on the determined WKD loss. (in [0068] At 410, the reweighting module 220 determines a weighted loss [wherein further comprising determining a weighted knowledge distillation (WKD) loss based on the first output vector, the second output vector and the importance value], shown here as reweighted loss 330, using a weighted loss function based on the knowledge distillation loss 314 [based on the first output vector, the second output vector and the importance value] and ground truth loss 312 for each respective labeled training data sample x in the batch of training data 302. In some embodiments, the reweighting module 220 determines the reweighted loss 330 by determining a knowledge distillation weight λ.sub.x.sup.KD [and the importance value] for the respective labeled training data sample x, determining a ground truth weight λ.sub.x.sup.CE for the respective labeled training data sample x, and computing the reweighted loss 330 (denoted as L) as the sum of the knowledge distillation loss L.sub.KD(x) weighted by the knowledge distillation weight λ.sub.x.sup.KD, and the ground truth loss L.sub.CE(x) [the first output vector, the second output vector] weighted by the ground truth weight λ.sub.x.sup.CE The reweighted loss 330 may be computed as a mean across the entire batch of training data X (i.e. the batch of training data 302) using the following weighted loss function:… [0069] At 412, the gradient descent module 218 performs a gradient descent operation on the student 234 using the weighted loss function L to identify an adjusted set of values of the learnable parameters of the student 234 [wherein the at least one parameter of the student model is updated based on the determined WKD loss]... The adjusted values of the learnable parameters may be identified as values of the learned parameters that would result in a lower or minimized reweighted loss 330 with respect the current labeled training data sample x, or with respect to the entire batch of training data X… [0071] In some embodiments, the method 400 may be repeated one or more times with additional batches of training data obtained (e.g., stochastically) from the training dataset 240... )
Regarding claim 3, the rejection of claim 1 is incorporated and Lu in combination with Lai teaches the method of claim 1, wherein the importance value are determined based on a probability of a ground-truth class when the ground-truth class is known. (in [0042] L.sub.CE is therefore a function that is used to compute a Cross-Entropy (CE) loss between the ground-truth label of the labeled training data sample input into the teacher and the student and the output of the student, S.sub.θ(x), and L.sub.KD is a function that is used to compute a KD loss [wherein the importance value are determined based on a probability of a ground-truth class when the ground-truth class is known] based on the Kullback-Leibler (KL) divergence between the teacher prediction data 24 and the student prediction data 34. L.sub.KD may be defined such that the comparison between teacher prediction data 24 and student prediction data 34 is congruent. For example, the teacher prediction data 24 and student prediction data 34 used by L.sub.KD may be the respective models' logits (i.e. pre-normalized predictions), whereas the student prediction data 34 used by L.sub.CE may be the normalized predictions, such as a predicted probability distribution over a plurality of classes for a classification task [wherein the importance value are determined based on a probability of a ground-truth class when the ground-truth class is known], or the student's predicted label for the labeled training data sample x, such that the predicted probability distribution or student's predicted label can be compared to the ground truth label [when the ground-truth class is known] of the labeled training data sample x. Because the student's predicted probability distribution can be derived from the student's logits (by normalizing using a softmax function) [wherein the importance value are determined based on a probability of a ground-truth class when the ground-truth class is known], and the student's predicted label may be derived from the student's predicted probability distribution (by applying an argmax function), various types of comparison operations using different student outputs can be performed by properly defining L.sub.KD and L.sub.CE)
Regarding claim 4, the rejection of claim 3 is incorporated and Lu in combination with Lai teaches the method of claim 3, wherein the importance value are determined based on a last output vector of the teacher model that is output from a last layer of the teacher model. (in [0042] L.sub.CE is therefore a function that is used to compute a Cross-Entropy (CE) loss between the ground-truth label of the labeled training data sample input into the teacher and the student and the output of the student, S.sub.θ(x), and L.sub.KD is a function that is used to compute a KD loss based on the Kullback-Leibler (KL) divergence between the teacher prediction data 24 and the student prediction data 34 [wherein the importance value are determined based on a last output vector of the teacher model that is output from a last layer of the teacher model]. L.sub.KD may be defined such that the comparison between teacher prediction data 24 and student prediction data 34 is congruent. For example, the teacher prediction data 24 and student prediction data 34 used by L.sub.KD may be the respective models' logits (i.e. pre-normalized predictions)…)
Regarding claim 5, the rejection of claim 3 is incorporated and Lu in combination with Lai teaches the method of claim 3, wherein the first output vector of a first dimension and the second output vector of a second dimension are normalized to have a same dimension. (in [0042] L.sub.CE is therefore a function that is used to compute a Cross-Entropy (CE) loss between the ground-truth label of the labeled training data sample input into the teacher and the student and the output of the student, S.sub.θ(x), and L.sub.KD is a function that is used to compute a KD loss based on the Kullback-Leibler (KL) divergence between the teacher prediction data 24 and the student prediction data 34 [wherein the first output vector of a first dimension and the second output vector of a second dimension are normalized to have a same dimension]. L.sub.KD may be defined such that the comparison between teacher prediction data 24 and student prediction data 34 is congruent. For example, the teacher prediction data 24 and student prediction data 34 used by L.sub.KD may be the respective models' logits (i.e. pre-normalized predictions) [wherein the first output vector of a first dimension and the second output vector of a second dimension are normalized to have a same dimension] …)
Regarding claim 6, the rejection of claim 1 is incorporated and Lu in combination with Lai teaches the method of claim 1, wherein the layer of the teacher model corresponds to an intermediate layer of the teacher model. (in [0059] In the example of FIG. 3, the teacher 232, denoted as T(.Math.), is a relatively large neural network model for an inference task [wherein the layer of the teacher model corresponds to an intermediate layer of the teacher model as a large neural network model with that has a layer corresponding the one of the two or more intermediate layers between the input and layer for making inference] (i.e. a neural network including a large number of learnable parameters implementing a model for an inference task) which has been trained to optimize the values of the learnable parameters of the large neural network model, and which is to be compressed using KD. The student 234, denoted as S.sub.θ(.Math.), is a relatively smaller neural network model (i.e. a neural network including a smaller number of learnable parameters than the teacher and implementing a model for the inference task) which, once trained using KD, is to be deployed to a computing device having limited computing resources (e.g. memory and/or processing power) for inference (i.e. to output student inference data for new input data).)
Regarding claim 7, the rejection of claim 1 is incorporated and Lu in combination with Lai teaches the method of claim 1, wherein the layer of the teacher model corresponds to a last layer of the teacher model. (in [0059] In the example of FIG. 3, the teacher 232, denoted as T(.Math.), is a relatively large neural network model for an inference task [wherein the layer of the teacher model corresponds to a last layer of the teacher model as a large neural network model with that has a layer corresponding the last layer for making inference] (i.e. a neural network including a large number of learnable parameters implementing a model for an inference task) which has been trained to optimize the values of the learnable parameters of the large neural network model, and which is to be compressed using KD. The student 234, denoted as S.sub.θ(.Math.), is a relatively smaller neural network model (i.e. a neural network including a smaller number of learnable parameters than the teacher and implementing a model for the inference task) which, once trained using KD, is to be deployed to a computing device having limited computing resources (e.g. memory and/or processing power) for inference (i.e. to output student inference data for new input data).)
Regarding independent claim 8, Lu in combination with Lai teaches a system for training a student model, the system comprising: a memory storing instructions; and a processor configured to execute the instructions to: (in [0109] … Accordingly, the technical solution of the present disclosure may be embodied in the form of a software product. A suitable software product may be stored in a pre-recorded storage device or other similar non-volatile or non-transitory computer readable medium, including DVDs, CD-ROMs, USB flash disk, a removable hard disk, or other storage media, for example. The software product includes machine-executable instructions tangibly stored thereon that enable a processing device (e.g., a personal computer, a server, or a network device) to execute examples of the methods disclosed herein [the system comprising: a memory storing instructions; and a processor configured to execute the instructions to]; And in in [0004] Knowledge Distillation (KD) is a compression technique used to transfer the knowledge of a large trained neural network model (i.e. a neural network model with many learned parameters) to a smaller neural network model (i.e. a neural network model with fewer learned parameters than the large trained neural network model). KD utilizes the generalization ability of the larger trained neural network model (referred to as the “teacher model” or “teacher”) using the inference data output by the larger trained model as “soft targets”, which are used as a supervision signal for training a smaller neural network model (called the “student model” or “student”)…)
Regarding the remaining limitations of claim 8, the limitations are similar to those in claim 1, and are thus rejected under the same rationale.
Regarding claims 9-14, the limitations are similar with those in claims 2-7, and are thus rejected under the same rationale.
Regarding independent claim 15, Lu in combination with Lai teaches a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to: (in [0109] … Accordingly, the technical solution of the present disclosure may be embodied in the form of a software product. A suitable software product may be stored in a pre-recorded storage device or other similar non-volatile or non-transitory computer readable medium [non-transitory computer-readable storage medium storing instructions that, …], including DVDs, CD-ROMs, USB flash disk, a removable hard disk, or other storage media, for example. The software product includes machine-executable instructions tangibly stored thereon that enable a processing device (e.g., a personal computer, a server, or a network device) to execute examples of the methods disclosed herein [non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to]; And in in [0004] Knowledge Distillation (KD) is a compression technique used to transfer the knowledge of a large trained neural network model (i.e. a neural network model with many learned parameters) to a smaller neural network model (i.e. a neural network model with fewer learned parameters than the large trained neural network model). KD utilizes the generalization ability of the larger trained neural network model (referred to as the “teacher model” or “teacher”) using the inference data output by the larger trained model as “soft targets”, which are used as a supervision signal for training a smaller neural network model (called the “student model” or “student”)…)
Regarding the remaining limitations of claim 15, the limitations are similar to those in claim 1, and are thus rejected under the same rationale.
Regarding claims 16-20, the limitations are similar with those in claims 2-6, and are thus rejected under the same rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wagner et al. (US 11537890): teaches the use of the gradient in co-training of teacher and student models using distill knowledge techniques, in 4:26-40: After the values of the set of weights 220 of student network 215 are updated, teacher network 205 and student network 215 can continue to co-train. FIG. 6 illustrates co-training of teacher network 205 and student network 215. As shown, training data 600 (e.g., an input matrix) is input into teacher network 205 and processed based on the values of its set of weights 210 to produce an output (e.g., an output matrix). Teacher network 205 uses the output to generate loss 605 in the same way as that used to generate loss 230. In some embodiments, teacher network 205 generates loss 605 using a backpropagation technique based on the output of teacher network 205. FIG. 6 also shows the updating of the set of weights 210 of teacher network 205. Specifically, teacher network 205 generates loss 615 based on losses 605 and 610.
Gou et al. (US 20200110982): teaches in [0160] In some non-limiting embodiments, knowledge distillation may be used to compress cumbersome models (e.g, first predictive model 410) into light-weight models (e.g., second predictive model 420) for various purposes (e.g., simplifying the deployment of a model and/or the like). For example, a small second (e.g., student) predictive model 420 may be trained using knowledge distilled from a cumbersome first (e.g., teacher) predictive model 410 (which may be pre-trained, trained before training the second model, and/or the like). In some non-limiting embodiments, the second predictive model 420 may be trained so that the logits (e.g., second predictions 428) of the second predictive model 420 match the logits (e.g., first predictions 418) of the first predictive model 410, as described herein. In some non-limiting embodiments, the first (e.g., teacher) predictive model 410 may include a pre-trained deep neural network (DNN), which may take data (e.g., images) as input and/or may output predictions (e.g., probabilities and/or the like) associated with classification(s) of the data. Additionally or alternatively, the second (e.g., student) predictive model 420 may also include a neural network (e.g., DNN and/or the like), and/or the second predictive model 420 may contain fewer (e.g., much fewer) layers than the first predictive model 410. In some non-limiting embodiments, the second predictive model 420 may be trained using the same inputs (e.g., images) as the first predictive model 410 and/or a portion (e.g., subset) thereof. In some non-limiting embodiments, the training loss of second (e.g., student) predictive model 420 may contains two parts, e.g., minimizing the difference between the predicted labels (second predictions 428) and true labels 408 (e.g., hard labels) and minimizing the difference between the predicted labels (second predictions 428) and predictions (e.g., first predictions 418) from the first (e.g., teacher) predictive model 410 (e.g., soft labels). For example, the soft labels (e.g., first predictions 418) may be associated with relative probabilities of different classes, and/or such soft labels (e.g., first predictions 418) provide rich information about how the first (e.g., teacher) predictive model 410 interprets the input data (e.g., images)… [0162] In some non-limiting embodiments, knowledge distillation may be used to distill and/or transfer knowledge between models (e.g., first (e.g, teacher) predictive model 410 and second (e.g., student) predictive model 420) for the purpose of interpretation. Additionally or alternatively, the second (e.g., student) predictive model 420 may locally approximate (but not necessarily fully mimic) behaviors of the first (e.g., teacher) predictive model 410, and, therefore, the second (e.g., student) predictive model 420 may be much simpler and more explainable/interpretable.
Ping et al. (US 20190180732): teaches in [0042] Knowledge distillation is originally proposed for compressing large models to smaller ones. In deep learning, a smaller student network is distilled from the teacher network by minimizing the loss between their outputs (e.g., L2 or cross-entropy)…
Fukuda (US 20220188643): teaches in abstract method of training a student neural network is provided. The method includes feeding a data set including a plurality of input vectors into a teacher neural network to generate a plurality of output values, and converting two of the plurality of output values from the teacher neural network for two corresponding input vectors into two corresponding soft labels. The method further includes combining the two corresponding input vectors to form a synthesized data vector, and forming a masked soft label vector from the two corresponding soft labels. The method further includes feeding the synthesized data vector into the student neural network, using the masked soft label vector to determine an error for modifying weights of the student neural network, and modifying the weights of the student neural network. And in [0002] In artificial neural networks (ANN), “learning” occurs through a change in weights applied to the data inputs of each neuron in the neural network. An artificial neural network can have one or more layers of neurons depending on the neural network architecture. Training of the neural network can be conducted using training pairs, including input data and an expected output/result (i.e., hard labels). Training the neural network then involves feeding the training pairs into the neural network and generating a prediction about the output (i.e., soft labels). The resulting or predicted output can be compared to the expected output for each of the training pairs to determine the correctness or incorrectness of the predictions… [0022] In various embodiments, a complex deep neural network (i.e., the teacher network) can be trained using a complete dataset with hard targets/labels, where this can be conducted offline. The deep neural network can be a multilayer perceptron. A correspondence can be established between the intermediate outputs of the teacher network and the student network. The outputs from the teacher network can be used to backpropagate calculated error values through the student network, so that the student network can learn to replicate the behavior of the teacher network, rather than learn directly with the hard targets/labels. A teacher network can, thereby, effectively transfer its knowledge to student networks of a smaller size.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OLUWATOSIN ALABI whose telephone number is (571)272-0516. The examiner can normally be reached Monday-Friday, 8:00am-5:00pm EST..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OLUWATOSIN ALABI/ Primary Examiner, Art Unit 2129
Prosecution Insights