DETAILED ACTION
This Office Action is sent in response to the Applicant’s Communication received on 06/11/2026 for application number 18/313,189. The Office hereby acknowledges receipt of the following and placed of record in file: Specification, Drawings, Abstract, Oath/Declaration, IDS, and Claims.
Claims 1, 4, 10, 11, 14, and 20 are amended.
Claims 1-20 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
35 USC 103
On page 7 of the remarks section, the Applicant argues that the cited art, whether taken alone or in combination, fails to teach or suggest knowledge distillation where the generation of outputs from the student and teacher models occurs synchronously for each individual training sample. Specifically, the cited art does not teach or suggest that "generating the first output and generating the second output are performed synchronously at substantially the same time," and that "adjusting the one or more parameters of the student neural network model is performed prior to processing a subsequent training sample," particularly, in combination with "repeating, for each subsequent training sample." These limitations define a specific, sample-by-sample, real-time interactive training loop that is structurally and functionally distinct from the methods disclosed in the references.
Applicant’s arguments have been considered but are moot. Further search and consideration was given in light of the newly amended claim limitations. A new ground of rejection, relying on a new combination of references, was applied.
On page 8 of the remarks section, the Applicant further argues that Huh does not teach or suggest that the student and teacher models operate "synchronously at substantially the same time" to generate outputs for a given sample. Furthermore, Huh does not teach that the student model is adjusted based on a sample's loss "prior to processing a subsequent training sample."
Applicant’s arguments have been considered but are moot. Further search and consideration was given in light of the newly amended claim limitations. A new ground of rejection, relying on a new combination of references, was applied.
On page 8 of the remarks section, Applicant further argues that Yuan's architecture is explicitly asynchronous and decoupled. Yuan's simulator uploads data to a buffer and moves on, with the training happening asynchronously on sampled data from that buffer. Therefore, Yuan teaches away from the synchronous and sample-by-sample processing recited in claim 1.
Applicant’s arguments have been considered but are moot. Further search and consideration was given in light of the newly amended claim limitations. A new ground of rejection, relying on a new combination of references, was applied.
On page 8 of the remarks section, the Applicant further argues that if the general student-teacher method of Huh were combined with the P2P architecture of Yuan, a person of ordinary skill in the art would adopt Yuan's distributed architecture, resulting in an asynchronous, buffered system where the student and remote teacher would independently generate outputs for a batch of samples, which would then be collected and used for training. This combination would lead a skilled artisan away from "generating the first output and generating the second output are performed synchronously at substantially the same time" and "adjusting the one or more parameters of the student neural network model is performed prior to processing a subsequent training sample of the plurality of training samples," particularly in combination with "repeating, for each subsequent training sample of the plurality of training samples, generating the first output, generating the second output, determining the first loss, and adjusting the one or more parameters of the student neural network model," as now claimed.
Applicant’s arguments have been considered but are moot. Further search and consideration was given in light of the newly amended claim limitations. A new ground of rejection, relying on a new combination of references, was applied.
For at least the reasons above, a similar rationale is given to independent claim 11. Therefore, the 35 USC 103 rejection is maintained for claims 1, 11, and their respective dependent claims.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
A broad range or limitation together with a narrow range or limitation that falls within the broad range or limitation (in the same claim) may be considered indefinite if the resulting claim does not clearly set forth the metes and bounds of the patent protection desired. See MPEP § 2173.05(c). In the present instance, claims 1 and 11 recite the broad recitation “at substantially at the same time”, and the claim also recites “synchronously” which is the narrower statement of the range/limitation. Dependent claims 2-10 and 12-20 depend from and contain the same deficiencies as claims 1 and 11 and are therefore rejected for the same reason. The claim(s) are considered indefinite because there is a question or doubt as to whether the feature introduced by such narrower language is (a) merely exemplary of the remainder of the claim, and therefore not required, or (b) a required feature of the claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3, 6, 7, 11, 13, 16, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Zu (US 12633094 B2), hereinafter Zu, in view of Yuan et al. (CN114676823A, see attached translation), hereinafter Yuan, Herard et al. (US 20240169251 A1), hereinafter Herard, and Ll et al. (US 20210182661 A1), hereinafter Ll.
Regarding claim 1, Zu teaches
A computer-implemented method executed by data processing hardware that causes the data processing hardware to perform operations [Abstract, A neural network inference acceleration method; Col 3, lines 43-48, a neural network inference acceleration device, including a memory; and a processor coupled to the memory, the processor is configured to, based on instructions stored in the memory, perform the steps of the above neural network inference acceleration method] comprising:
obtaining a plurality of training samples [Col 1, lines 55-56, acquiring… an accelerated data set; Col 6, lines 64-67, the accelerated data set includes calibration data and test data, etc., wherein the calibration data is a set of samples for training, mainly used to train parameters of the neural network model during accelerating process; Abstract, the accelerating process includes… model distillation];
generating, using a student neural network model, a first output based on a first training sample of the plurality of training samples [Col 3, lines 1-2, distill loss information obtained by comparing an output of each of the student models];
generating, using a teacher neural network model, a second output based on the first training sample of the plurality of training samples [Col 3, lines 1-3, distill loss information obtained by comparing… with an output of multiple the teacher models],
wherein generating the first output and generating the second output are performed synchronously at substantially the same time [Col 11, lines 65-67, when training the multiple student models and the teacher models, synchronously calculating three kinds of loss information in each iteration process];
determining, based on the first output and the second output, a first loss [Col 12, lines 20-21, distill1 represents distill loss information between student model 1 and teacher model 1 and teacher model 2];
adjusting, based on the first loss, one or more parameters of the student neural network model [Col 12, lines 59-62, after the calculated three losses are added, they will be uniformly fed back to multiple student models through back propagation (BP) algorithm to achieve more optimized results].
Zu does not teach using a student neural network model executing on a first processing unit; a teacher neural network model executing on a second processing unit, the second processing unit remote from the first processing unit; wherein adjusting the one or more parameters of the student neural network model is performed prior to processing a subsequent training sample of the plurality of training samples; repeating, for each subsequent training sample of the plurality of training samples, generating the first output, generating the second output, determining the first loss, and adjusting the one or more parameters of the student neural network model.
Yuan teaches,
using a student neural network model (Para 0008, the simulator module) executing on a first processing unit (Para 0008, the simulator module exists in each computing node), and using a teacher neural network model executing on a second processing unit (Para 0008, an evaluator module), the second processing unit remote from the first processing unit (Para 0013, in the reinforcement learning framework based on P2P networks… the current best neural network model will also provide inference services to the outside world through an RPC remote procedure call protocol) [Para 0008, A reinforcement learning framework based on a P2P network includes a simulator module, a trainer module, and an evaluator module; wherein, the simulator module exists in each computing node, loads the current best neural network model, performs self-play training to generate a large amount of data, and uploads the generated data to a common data node ReplayBuffer; the trainer module samples data from the common data node ReplayBuffer, trains a new neural network model, and saves the trained neural network model in the data node Model Pool; the evaluator module loads a new neural network model from the data node Model Pool, performs multiple rounds of competition with the current best neural network model, calculates the win rate, and iterates the new neural network model with the high win rate into the current best neural network model; Para 0013, in the reinforcement learning framework based on P2P networks, after iterating a new neural network model with a high win rate into the current best neural network model, the current best neural network model will also provide inference services to the outside world through an RPC remote procedure call protocol or the Internet].
Yuan is analogous to the claimed invention as they both relate to learning from distributed decentralized learning. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Yuan and provide processing the two models on individual processing units, where the second processing unit is remote, in order to provide [Yuan, 0007] a learning framework based on P2P networks that has a low threshold, is accessible to everyone, has a high upper limit, can solve many large problems, is flexible, and is compatible with various large learning algorithms.
Zu-Yuan teach the above limitations of claim 1 including the one or more parameters of the student neural network model (Zu, Col 12, lines 59-62) and the plurality of training samples (Zu, Col 1, lines 55-56).
Zu-Yuan do not teach wherein adjusting student model is performed prior to processing a subsequent training sample; repeating, for each subsequent training sample of the plurality of training samples, generating the first output, generating the second output, determining the first loss, and adjusting student model.
Herard teaches,
wherein adjusting model is performed prior to processing a subsequent training sample [Para 0024, During the model training phase, model training module 106b executes a trained NLP teacher model 107 on unlabeled text from database 112b to predict compliance pseudo-labels for each sentence in the unlabeled text, prepares an input data set comprising the unlabeled sentences and pseudo-labels using labeling module 109, uses the sentences and pseudo-labels as input to train an NLP student model 108 (including the injection of noise from noise injection module 110), and then uses the newly-trained NLP student model 108 as the new NLP teacher model 107 for a subsequent corpus of unlabeled sentences].
Herard is analogous to the claimed invention as they both relate to knowledge distillation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Herard and provide adjusting model prior to processing a subsequent data in order to optimize model output by training one sample at a time.
Zu-Yuan-Herard teach the above limitations of claim 1 including the plurality of training samples and subsequent training sample (Herard, para 0024).
Zu-Yuan-Herard do not teach repeating, for each training samples, generating first output, generating second output, determining first loss, and adjusting model.
Ll teaches,
repeating, for each training samples, generating first output, generating second output, determining first loss, and adjusting model [Para 0076, Each sample of the audio data 430 in a training set is supplied to both the first and second versions of the neural network model 410 and 420, respectively. The first and second versions of the neural network model 410 and 420, respectively, provide a logit output from respective logit layers 418 and 428. The logit outputs from the first and second versions of the neural network model 410 and 420 are passed to a loss function 440 for the second version of the neural network model 420. The loss function may be based on the Kullback Leibler distance or relative entropy between the two sets of logits; this effectively aims to teach the second version of the neural network model 420, as a student, to model the probability distributions as per the first version of the neural network model 410, acting as a teacher. As shown in FIG. 4A, the loss function 440 is used to determine a parameter update for the second version of the neural network model 420, e.g. based on optimization methods (e.g., gradient descent and the like). The process is repeated for a plurality of audio samples 430 (e.g. batches of input data) in order to train the second version of the neural network model 420].
Ll is analogous to the claimed invention as they both relate to knowledge distillation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Ll and provide repeating the training process in order to iteratively improve model performance.
Regarding claim 3, Zu-Yuan-Herard-Ll teach the limitations of claim 1.
Yuan further teaches,
Transmitting (Para 0008, iterates the new neural network model… into the current best neural network model) a remote procedure call (RPC) to the teacher neural network model (Para 0008, an evaluator module) to generate each output (Para 0008, win rate) [Para 0008, A reinforcement learning framework based on a P2P network includes a simulator module, a trainer module, and an evaluator module; wherein, the simulator module exists in each computing node, loads the current best neural network model, performs self-play training to generate a large amount of data, and uploads the generated data to a common data node ReplayBuffer; the trainer module samples data from the common data node ReplayBuffer, trains a new neural network model, and saves the trained neural network model in the data node Model Pool; the evaluator module loads a new neural network model from the data node Model Pool, performs multiple rounds of competition with the current best neural network model, calculates the win rate, and iterates the new neural network model with the high win rate into the current best neural network model; Para 0013, in the reinforcement learning framework based on P2P networks, after iterating a new neural network model with a high win rate into the current best neural network model, the current best neural network model will also provide inference services to the outside world through an RPC remote procedure call protocol or the Internet].
Yuan is analogous to the claimed invention as they both relate to learning from distributed decentralized learning. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Yuan and provide an RPC to the teacher network for [Yuan, 0007] ease of remote accessibility.
Regarding claim 6, Zu-Yuan-Herard-Ll teach the limitations of claim 1.
Zu further teaches,
wherein the teacher neural network model comprises a trained model [Col 12, lines 43-44, when distilling the neural network model to be accelerated, multiple teacher models with large parameters are first trained to be optimal].
Regarding claim 7, Huh-Yuan teach the limitations of claim 1.
Yuan further teaches,
wherein first processing unit belongs to a first entity and second processing unit belongs to a second entity different from the first entity [Para 0008, A reinforcement learning framework based on a P2P network includes a simulator module, a trainer module, and an evaluator module; wherein, the simulator module exists in each computing node, loads the current best neural network model, performs self-play training to generate a large amount of data, and uploads the generated data to a common data node ReplayBuffer; the trainer module samples data from the common data node ReplayBuffer, trains a new neural network model, and saves the trained neural network model in the data node Model Pool; the evaluator module loads a new neural network model from the data node Model Pool, performs multiple rounds of competition with the current best neural network model, calculates the win rate, and iterates the new neural network model with the high win rate into the current best neural network model; Para 0013, in the reinforcement learning framework based on P2P networks, after iterating a new neural network model with a high win rate into the current best neural network model, the current best neural network model will also provide inference services to the outside world through an RPC remote procedure call protocol or the Internet].
Yuan is analogous to the claimed invention as they both relate to learning from distributed decentralized learning. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Yuan and provide separate processing entities in order to improve processing efficiency.
Regarding claim 11, Zu teaches,
A system [Col 19, lines 29-31, the system automatically quantifies the model to obtain the optimal execution speed and model compression ratio] comprising: data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations [Col 3, lines 43-48, a neural network inference acceleration device, including a memory; and a processor coupled to the memory, the processor is configured to, based on instructions stored in the memory, perform the steps of the above neural network inference acceleration method] comprising:
obtaining a plurality of training samples [Col 1, lines 55-56, acquiring… an accelerated data set; Col 6, lines 64-67, the accelerated data set includes calibration data and test data, etc., wherein the calibration data is a set of samples for training, mainly used to train parameters of the neural network model during accelerating process; Abstract, the accelerating process includes… model distillation];
generating, using a student neural network model, a first output based on a first training sample of the plurality of training samples [Col 3, lines 1-2, distill loss information obtained by comparing an output of each of the student models];
generating, using a teacher neural network model, a second output based on the first training sample of the plurality of training samples [Col 3, lines 1-3, distill loss information obtained by comparing… with an output of multiple the teacher models],
wherein generating the first output and generating the second output are performed synchronously at substantially the same time [Col 11, lines 65-67, when training the multiple student models and the teacher models, synchronously calculating three kinds of loss information in each iteration process];
determining, based on the first output and the second output, a first loss [Col 12, lines 20-21, distill1 represents distill loss information between student model 1 and teacher model 1 and teacher model 2];
adjusting, based on the first loss, one or more parameters of the student neural network model [Col 12, lines 59-62, after the calculated three losses are added, they will be uniformly fed back to multiple student models through back propagation (BP) algorithm to achieve more optimized results].
Zu does not teach using a student neural network model executing on a first processing unit; a teacher neural network model executing on a second processing unit, the second processing unit remote from the first processing unit; wherein adjusting the one or more parameters of the student neural network model is performed prior to processing a subsequent training sample of the plurality of training samples; repeating, for each subsequent training sample of the plurality of training samples, generating the first output, generating the second output, determining the first loss, and adjusting the one or more parameters of the student neural network model.
Yuan teaches,
using a student neural network model (Para 0008, the simulator module) executing on a first processing unit (Para 0008, the simulator module exists in each computing node), and using a teacher neural network model executing on a second processing unit (Para 0008, an evaluator module), the second processing unit remote from the first processing unit (Para 0013, in the reinforcement learning framework based on P2P networks… the current best neural network model will also provide inference services to the outside world through an RPC remote procedure call protocol) [Para 0008, A reinforcement learning framework based on a P2P network includes a simulator module, a trainer module, and an evaluator module; wherein, the simulator module exists in each computing node, loads the current best neural network model, performs self-play training to generate a large amount of data, and uploads the generated data to a common data node ReplayBuffer; the trainer module samples data from the common data node ReplayBuffer, trains a new neural network model, and saves the trained neural network model in the data node Model Pool; the evaluator module loads a new neural network model from the data node Model Pool, performs multiple rounds of competition with the current best neural network model, calculates the win rate, and iterates the new neural network model with the high win rate into the current best neural network model; Para 0013, in the reinforcement learning framework based on P2P networks, after iterating a new neural network model with a high win rate into the current best neural network model, the current best neural network model will also provide inference services to the outside world through an RPC remote procedure call protocol or the Internet].
Yuan is analogous to the claimed invention as they both relate to learning from distributed decentralized learning. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Yuan and provide processing the two models on individual processing units, where the second processing unit is remote, in order to provide [Yuan, 0007] a learning framework based on P2P networks that has a low threshold, is accessible to everyone, has a high upper limit, can solve many large problems, is flexible, and is compatible with various large learning algorithms.
Zu-Yuan teach the above limitations of claim 1 including the one or more parameters of the student neural network model (Zu, Col 12, lines 59-62) and the plurality of training samples (Zu, Col 1, lines 55-56).
Zu-Yuan do not teach wherein adjusting student model is performed prior to processing a subsequent training sample; repeating, for each subsequent training sample of the plurality of training samples, generating the first output, generating the second output, determining the first loss, and adjusting student model.
Herard teaches,
wherein adjusting model is performed prior to processing a subsequent training sample [Para 0024, During the model training phase, model training module 106b executes a trained NLP teacher model 107 on unlabeled text from database 112b to predict compliance pseudo-labels for each sentence in the unlabeled text, prepares an input data set comprising the unlabeled sentences and pseudo-labels using labeling module 109, uses the sentences and pseudo-labels as input to train an NLP student model 108 (including the injection of noise from noise injection module 110), and then uses the newly-trained NLP student model 108 as the new NLP teacher model 107 for a subsequent corpus of unlabeled sentences].
Herard is analogous to the claimed invention as they both relate to knowledge distillation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Herard and provide adjusting model prior to processing a subsequent data in order to optimize model output by training one sample at a time.
Zu-Yuan-Herard teach the above limitations of claim 1 including the plurality of training samples and subsequent training sample (Herard, para 0024).
Zu-Yuan-Herard do not teach repeating, for each training samples, generating first output, generating second output, determining first loss, and adjusting model.
Ll teaches,
repeating, for each training samples, generating first output, generating second output, determining first loss, and adjusting model [Para 0076, Each sample of the audio data 430 in a training set is supplied to both the first and second versions of the neural network model 410 and 420, respectively. The first and second versions of the neural network model 410 and 420, respectively, provide a logit output from respective logit layers 418 and 428. The logit outputs from the first and second versions of the neural network model 410 and 420 are passed to a loss function 440 for the second version of the neural network model 420. The loss function may be based on the Kullback Leibler distance or relative entropy between the two sets of logits; this effectively aims to teach the second version of the neural network model 420, as a student, to model the probability distributions as per the first version of the neural network model 410, acting as a teacher. As shown in FIG. 4A, the loss function 440 is used to determine a parameter update for the second version of the neural network model 420, e.g. based on optimization methods (e.g., gradient descent and the like). The process is repeated for a plurality of audio samples 430 (e.g. batches of input data) in order to train the second version of the neural network model 420].
Ll is analogous to the claimed invention as they both relate to knowledge distillation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Ll and provide repeating the training process in order to iteratively improve model performance.
Claims 13, 16, and 17 are system claims that recite identical limitations to method claims 3, 6, and 7, respectively. Therefore, claims 13, 16, and 17 are rejected using the same rationale as claims 3, 6, and 7, respectively.
Claim(s) 2, 4, 12, and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zu in view of Yuan, Herard, and Ll, and in further view of Jafari et al. (US 20230222326 A1), hereinafter Jafari.
Regarding claim 2, Zu-Yuan-Herard-Ll teach the limitations of claim 1.
Zu-Yuan-Herard-Ll do not teach first processing unit and second processing unit each comprise a respective tensor processing unit.
Jafari teaches,
first processing unit and second processing unit each comprise a respective tensor processing unit [Para 0055, The processing system 1200 may include one or more processing devices 1202, such as… a tensor processing unit (TPU)].
Jafari is analogous to the claimed invention as they both relate to knowledge distillation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Jafari and provide tensor processing units in order to optimize the model for cost-efficient scaling.
Regarding claim 4, Zu-Yuan-Herard-Ll teach the limitations of claim 1.
Zu-Yuan-Herard-Ll do not teach first output and second output each comprise a respective logit.
Jafari further teaches,
first output and second output each comprise a respective logit [Para 0035, the student neural network model 204 is trained using a first loss function L.sub.AKD that has the objective of minimizing a difference between the outputs (e.g., logits generated by a final layer of a neural network model, before a softmax layer of the neural network model) generated by the student neural network model].
Jafari is analogous to the claimed invention as they both relate to knowledge distillation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Jafari and provide outputs as logits in order to improve model robustness by utilizing an output with enhanced interpretability.
Claims 12 and 14 are system claims that recite identical limitations to method claims 2 and 4, respectively. Therefore, claims 12 and 14 are rejected using the same rationale as claims 2 and 4, respectively.
Claim(s) 5 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zu in view of Yuan, Herard, and Ll, and in further view of Passban et al. (US 20220076136 A1), hereinafter Passban
Regarding claim 5, Zu-Yuan-Herard-Ll teach the limitations of claim 1 including adjusting the one or more parameters of the student neural network model (Zu, Col 12, lines 59-62) and the first loss (Zu, Col 12, lines 20-21).
Zu-Yuan-Herard-Ll does not teach determining, based on loss, a gradient.
Passban teaches,
determining, based on loss, a gradient [Para 0113, Referring back to FIG. 4, at step 408, the gradients with respect to the loss values are computed].
Passban is analogous to the claimed invention as they both relate to knowledge distillation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Passban and provide gradients in order to improve prediction accuracy by minimizing errors.
Claim 15 is a system claim that recites identical limitations to method claim 5. Therefore, claim 15 is rejected using the same rationale as claim 5.
Claim(s) 8, 9, 18, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zu in view of Yuan, Herard, and Ll, and in further view of Jafari et al. (US 20210383238 A1), hereinafter Jafari(2).
Regarding claim 8, Zu-Yuan-Herard-Ll teach the limitations of claim 1 including the first training sample.
Zu-Yuan-Herard-Ll do not teach an unlabeled training sample.
Jafari(2) teaches,
an unlabeled training sample [Para 0047, In the example of FIG. 4, a trained teacher NN model 410, untrained student NN model 412 and initial unlabeled input values {x.sub.1, . . . , x.sub.i, . . . , x.sub.N} corresponding to an input training dataset X].
Jafari is analogous to the claimed invention as they both relate to knowledge distillation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Jafari and provide an unlabeled training sample in order to improve the model’s cost by being removed from human intervention.
Regarding claim 9, Zu-Yuan-Herard-Ll teach the limitations of claim 1 including the second output (Zu, Col 3, lines 1-3).
Zu-Yuan-Herard-Ll do not teach generating, based on output from teacher neural network model, a label for first training sample.
Jafari(2) further teaches,
generating, based on output from teacher neural network model, a label for first training sample [Para 0048, teacher NN model 410 is first used to compute a set of output values {y.sub.1, . . . , y.sub.i, . . . , y.sub.N} that correspond to the input values {x.sub.1, . . . , x.sub.i, . . . , x.sub.N}, providing a labelled training dataset X.].
Jafari is analogous to the claimed invention as they both relate to knowledge distillation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Jafari and provide a label for training samples in order to improve prediction precision and reliability by providing verified validation data.
Claims 18 and 19 are system claims that recite identical limitations to method claims 8 and 9, respectively. Therefore, claims 18 and 19 are rejected using the same rationale as claims 8 and 9, respectively.
Claim(s) 10 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zu in view of Yuan, Herard, Ll, and Jafari(2), and in further view of Chopde et al. (US 20230419170 A1), hereinafter Chopde.
Regarding claim 10, Zu-Yuan-Herard-Ll-Jafari(2) teach the limitations of claim 9.
Zu-Yuan-Herard-Ll-Jafari(2) do not teach generating, using a second student neural network model executing on a third processing unit, a third output based on the labeled first training sample; determining, based on the label and the third output, a second loss; and adjusting, based on the second loss, the second student neural network model.
Chopde teaches,
generating, using a second student neural network model executing on a third processing unit, a third output (Para 0021, Para 0022, a second data output by the trained student model) based on the labeled first training sample (Para 0021, finetuned teacher model using the labeled data… to output a trained student model) [Para 0011, a system for efficient machine learning includes a data analytics system… The data analytics system is executed by a computer processor configured to apply object detection and classification and deep learning algorithms to detect object information captured by the image; Para 0021, performing a semi-supervised learning guided by the finetuned teacher model using the labeled data, the unlabeled data, and the test data to output a trained student model; Para 0022, acquiring a first data output by the finetuned teacher model and a second data output by the trained student model];
determining, based on the label and the third output, a second loss [Para 0023, obtaining a distillation loss value according to a distillation loss function based on a difference between the first data output and the second data output]; and
adjusting, based on the second loss, the second student neural network model [Para 0024, updating parameters of the trained student model based on the distillation loss value].
Chopde is analogous to the claimed invention as they both relate to knowledge distillation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zu’s teachings to incorporate the teachings of Chopde and provide a second student model in order to increase diversity by capture different aspects of the teacher’s knowledge.
Claim 20 is a system claim that recites identical limitations to method claim 10. Therefore, claim 20 is rejected using the same rationale as claim 10.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SYED RAYHAN AHMED whose telephone number is (571)270-0286. The examiner can normally be reached Mon-Fri ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SYED RAYHAN AHMED/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
