DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is made final.
This office action is made in response to the amendments filed on April 17, 2026.
Claims 1, and 6 have been amended.
Response to Amendment
The amendment filed on April 17, 2026 has been entered. Claims 1-16 remain pending in the application.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. CN202010997722.4, filed on September 21, 2020.
Response to Arguments
Regarding the 101 arguments
Applicant’s arguments, see pages 7-9, filed April 17, 2026, with respect to Step 2A Prong 1, have been fully considered and are persuasive. The rejection of January 20, 2026 has been withdrawn.
Regarding the 103 arguments
Applicant's arguments filed April 17, 2026 have been fully considered but they are not persuasive. Specifically:
Applicant argues
Claims 1, 4-9, and 12-16 are rejected under 35 USC 103 as being unpatentable over Lu et al. (US 11461415 B2, referred to as Lu), in view of Kim et al. (US 20200364574 Al, referred to as Kim).
Claims 2, 3, 10, and 11 are rejected under 35 USC 103 as being unpatentable over Lu et al. (US 11461415 B2, referred to as Lu), in view of Kim et al. (US 20200364574 Al, referred to as Kim), in view of Lin (US 20200320392 Al, referred to as Lin).
Claim 1, as amended, recites "obtaining a first neural network model, a second neural network model, and a third neural network model, wherein the first neural network model comprises a transformer layer, the second neural network model comprises the first neural network model or a neural network model obtained by performing parameter update on the first neural network model, and the third neural network model is obtained by compressing the second neural network model".
With respect to the above-recited features, the Office Action points to FIG. 5 and columns 12-14 of Lu. (Office Action, p. 15). Kim is cited for other matters.
Lu discloses "Starting with FIG. 5, a first training system 502 produces a pre-trained
model 504. For example, in the case of the BERT model described by the above-cited Devlin, et al. paper, the first training system 502 can train a BERT model to perform two tasks. In a first task, the training system 502 trains the BERT model to predict the identity of a word that has been omitted in an input training example. In a second task, the first training system 502 trains the BERT model to predict, given two sentences, whether the second sentence properly follows the first sentence. The resultant pre-trained model 504 may be considered general-purpose in nature because it can be further trained or fine-tuned to perform different tasks. More specifically, a subsequent training operation fine tunes the pre-trained model 504 by further modifying its parameter values, such that the resultant fine-tuned model performs a desired task." (Lu, col. 12 lines 44-59). "In the example of FIG. 5, a second training system 506 trains the pre-trained model 504 based on a corpus of training examples in a data store 508. The training examples include a set of positive examples, corresponding to pairs of query expressions and target expressions that are considered to respectively match, and a set of negative examples, corresponding to pairs of query expressions and target expressions that are not considered to match. The second training system 506 fine-tunes the pre-trained model 504 such that the resultant model can successfully determine the semantic relation between any given query expression and target expression. The second training system 506 can perform its training using any technique, such as stochastic gradient descent. As a result of its training, the second training system 506 produces a fine-tuned model 510. Note that, however, at this stage, the fine-tuned model 510 is a single-chain BERT-derived model that simulates the output results of the dual-encoder neural network 202; the fine-tuned model 510 does not provide weighting values for use in the actual model that underlies the dual-encoder neural network 202 of FIG. 2." (Lu, col. 12 lines 60-67 - col. 13 lines 1-13).
Accordingly, from the above-quoted portions, Lu merely discloses a single pre-trained model 504 (i.e., BERT model) that went through a training process that includes a training system 502 that trains the BERT model (pre-trained model 504) to predict the identity of a word that has been omitted in an input training example, then trains the BERT model to predict, given two sentences, whether the second sentence properly follows the first sentence. The Lu training process further includes a second training system 506 that trains the pre-trained model 504 based on a corpus of training examples in a data store 508, and fine-tunes the pre-trained model 504 to produce fine-tuned model 510. There is simply no disclosure in Lu of obtaining three neural network models that include a first neural network model comprising a transformer layer, a second neural network model comprising the first neural network model or a neural network model obtained by performing parameter update on the first neural network model, and a third neural network model obtained by compressing the second neural network model.
Lu therefore cannot disclose "obtaining a first neural network model, a second neural network model, and a third neural network model, wherein the first neural network model comprises a transformer layer, the second neural network model comprises the first neural network model or a neural network model obtained by performing parameter update on the first neural network model, and the third neural network model is obtained by compressing the second neural network model" as specified by claim 1.
Therefore, for at least the reasons set forth above, it is respectfully submitted that independent claim 1 is patentable over Lu and Kim.
Independent claim 9 is also patentable over Lu and Kim, for at least the same reasons given above in connection with claim 1.
Given that the rest of the claims depend from one of the above independent claims, at
least for the reasons similar to those discussed above, it is respectfully submitted that the rest of the claims are patentable over the cited references.
Examiner Response
Applicant argues that Lu merely discloses a single pretrained model 504 that is trained/fine-tuned, and therefore does not disclose obtaining three neural network models, Including a third neural network model obtained by compressing the second neural network model. However, the rejection is not based on Lu alone. Lu is relied upon for teaching a pretrained /fine-tuned transformer model framework, including pretrained model 504, fine-tuned model 510, and use of fine-tuned model 510 as a teacher model to train student model 604. Kim is relied upon for teaching compression of an original/trained neural network model to obtain a compressed neural network model, including original model 301, first compression model 302, and final compression model 303. Thus, The combined teachings Lu and Kim teach or suggest obtaining multiple neural network models, including obtaining a neural network model by compressing another neural network model, as claimed.
Applicant presents substantially the same argument for independent claim 9 and does not present a separate patentability arguments for the dependent claims. Therefore, claims 2-8 Does not overcome their rejection. and 10-16 are not Separately persuasive for at least the same reasons discussed above with respect to independent claim 1 and claim 9. Accordingly, applicants argument does not overcome the rejection.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4-9, and 12-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu et al. (US 11461415 B2, referred to as Lu), in view of Kim et al. (US 20200364574 A1, referred to as Kim).
Regarding claim 1. Lu teaches, a method of model compression, comprising:
obtaining a first neural network model, a second neural network model, and a third neural network model, wherein the first neural network model comprises a transformer layer, the second neural network model comprises the first neural network model or a neural network model obtained by performing parameter update on the first neural network model (FIG. 5, Col. 12, lines 44-67 cont. Col. 13, lines 1-13: Describes obtaining multiple neural network models, including a pre-trained model, and performing a training operation that fine tunes the pre-trained model by modifying its parameter values to produce a fine-tuned model. The fine-tuned model is used as a teacher model to train a student model, these correspond to a first, second and third neural network models in a training framework. ;Col 13, lines 64067 cont. Col 14, lines 1-13: Describes that the encoders transform inputs using at least one transformation unit, and that each transformation unit includes a self-attention mechanism configures to determine relevance among parts of an input.)
Although Lu teaches obtaining a first neural network model, a second neural network model, and a third neural network model, wherein the first neural network model comprises a transformer layer, the second neural network model comprises the first neural network model or a neural network model obtained by performing parameter update on the first neural network model. It does not teach the third neural network model is obtained by compressing the second neural network model.
Kim teaches the third neural network model is obtained by compressing the second neural network model (FIG. 3, and [0083-0086]: Describes a model compression module 310 compressing original New World Network model 301 to acquire first compression model 302 using a compression algorithm, including weight pruning, channel pruning, matrix factorization, and quantization.);
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to have combined the model network of Lu with the model compression of Kim. Doing so would have reduced memory usage while maintaining model performance during deployment.
Lu further teaches, processing first to-be-processed data using the first neural network model, to obtain a first output (FIG. 5, Col. 12, lines 44-67 cont. Col. 13, lines 1-53: Describes using the fine-tuned model 510 as a teacher model and feeding training examples to the teacher model which generates a soft label for a training example. This then creates output produced by processing the data with the teacher model corresponding to a first output.);
processing the first to-be-processed data using the third neural network model, to obtain a second output (FIG. 6, Col. 12, lines 44-67 cont. Col. 13, lines 1-53: Describes that the third training system 602 trains a student model 604 and that the same training examples are fed to both the teacher model and the student model. The student model produces probabilities for the training example, which correspond to an output produced by processing the data with the student model to obtain a second output.);
determining a first target loss based on the first output and the second output, and updating the second neural network model based on the first target loss, to obtain an updated second neural network model (FIG. 5, FIG. 6, Col. 12, lines 44-67 cont. Col. 13, lines 1-53: Describes that the teacher model (fine-tuned model 510) generates a soft label for a training example, and that the student model 604 generates corresponding probability outputs. The training of the student model 604 based on a cross-entropy loss function that compares the teacher provided soft label with the students models output probabilities, and updating the student model through training (parameter updates) to obtain an updated trained student model.);
Kim further teaches, compressing the updated second neural network model to obtain a target neural network model ([0083]: Describes using a model compression module to compress an original neural network model to obtain a first compression model, corresponding to obtaining a model by compression another model.; [0224]: Describes compressing a trained neural network model (updated model) to reduce the size of the model, which obtains a compressed model. These correspond to outputting a final compressed model form an updated model.).
Regarding claim 4, Lu in view of Kim teaches the method according to claim 1 wherein
compressing the second neural network model comprises quantizing the second neural network model, and
compressing the updated second neural network model to obtain the target neural network model comprises:
quantizing the updated second neural network model to obtain the target neural network model (Kim, [0009] and [0086]: Describes compressing a neural network model using compression techniques including quantization. ; [0201-00202]: Describes compressing a trained neural network model to reduce model size, using quantization to obtain a target neural network model.).
Regarding claim 5, Lu in view of Kim teaches the method according to claim 1, wherein
the second neural network model and the third neural network model each comprises an embedding layer, a transformer layer, and an output layer (Lu, Col. 8 lines 50-67 cont. Col. 9 lines 1-8: Describes a linguistic embedding mechanism, that transforms tokens of an input expression into input embeddings, for use in the models input layer (embedding layer).; Col. 13, lines 14-67 cont. Col 14. Lines 1-19: Describes a transformation unit including a self-attention mechanism (transformer layer). This generates output [probabilities/labels form the model (output layer).);
the first output is an output of a target layer in the second neural network model; the second output is an output of a target layer in the third neural network model (Lu, Col. 13 lines 14-53: Describes generating a soft label as an output of the teacher model, and generates probability outputs form the student model. These outputs correspond to outputs of respective target layers in the second and third neural network models.);
the target layer in the second neural network model comprises at least one of the embedding layer of the second neural network model, the transformer layer of the second neural network model, or the output layer of the second neural network model; and
the target layer in the third neural network model comprises at least one of the embedding layer of the third neural network model, the transformer layer of the third neural network model, or the output layer of the third neural network model (Lu, Col. 8 lines 50-67 cont. Col. 9 lines 1-8 and Col. 13, lines 14-67 cont. Col 14. Lines 1-19: These describe generating outputs at different stages of the neural network, including input embeddings produced by a linguistic embedding mechanism, intermediate representations produced by transformation units including self-attention mechanisms, and a final output probabilities/label. The second and third neural network models may comprise one of these.).
Regarding claim 6, Lu in view of Kim teaches the method according to claim 1, wherein the method further comprises:
processing second to-be-processed data using the first neural network model, to obtain a third output(Lu, Col. 13, lines 1-53: Describes processing training examples using the teacher model to generate soft labels. The teacher model is used as the second model and processes second data using the first neural network model to then be used for additional models.);
processing the second to-be-processed data using the target neural network model, to obtain a fourth output(Lu, Col. 13, lines 1-53: Describes processing the same training examples using the student model to generate probability outputs, which processes the second data using the target neural network model to obtain a fourth output.);
determining a second target loss based on the third output and the fourth output, and updating the updated second neural network model based on the second target loss, to obtain a fourth neural network model(Lu, Col. 13, lines 1-53: Describes determining a loss function based on the teacher model output and the student model output and updating the student model through training to modify its parameter values. This is determining a second target loss based on the third and fourth outputs to then update the second model to create an updated neural network.); and
Although Lu teaches processing second to-be-processed data using the first neural network model, to obtain a third output processing the second to-be-processed data using the target neural network model, to obtain a fourth output determining a second target loss based on the third output and the fourth output, and updating the updated second neural network model based on the second target loss, to obtain a fourth neural network model. It does not teach compressing the fourth neural network model to obtain an updated target neural network model.
Kim teaches, compressing the fourth neural network model to obtain an updated target neural network model ([0083-0086]: Describes a model compression module that applies a compression algorithm to portions of the model and selects compression range/algorithm to achieve a target compression rate.).
Regarding claim 7, Lu in view of Kim teaches the method according to claim 1, wherein the first to-be-processed data comprises one of audio data, text data, or image data (Lu, Col. 9 lines 3-29: Describes processing query expressions represented as tokens (text data) and transforms the tokens into input embeddings for neural network processing.).
Regarding claim 8, Lu in view of Kim teaches the method according to claim 1, wherein the obtaining theft first neural network model comprises:
performing parameter fine-tuning or knowledge distillation on a pre-trained language model, to obtain the first neural network model, wherein processing precision of the first neural network model during a target task processing is higher than a preset value (Lu, FIG. 5, Col. 12, lines 44-67 cont. Col. 13, lines 1-13: Describes producing a pre-trained language model and performing fine-tuning by further modifying its parameter values to obtain a fine-tuned model 510, this obtains the first neural networks model by parameter fine tuning of a pre-trained language model. The fine-tuned model is a teacher model to train student models for knowledge distillation. “The second training system 506 fine-tunes the pre-trained model 504 such that the resultant model can successfully determine the semantic relation between any given query expression and target expression.” Shows that the processing precision during target task processing exceeds a preset value.).
Regarding claims 9, and 12-16, which recites substantially the same limitations as claims 1, and 4-8 and further recites an apparatus comprising one or more processors (Lu, Col. 14 lines 33-41: Describes a computing device to execute the methods using computer hardware executing software using computer components.)to perform the method steps of claims 1, and 4-8, respectively and are rejected for the same reasons as described above.
Claim(s) 2, and 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu et al. (US 11461415 B2, referred to as Lu), in view of Kim et al. (US 20200364574 A1, referred to as Kim), in view of Lin (US 20200320392 A1. referred to as Lin).
Regarding claim 2, Lu in view of Kim teaches the method according to claim 1, but they do not teach wherein a difference between processing results obtained by processing same data using the second neural network model and the first neural network model falls within a preset range.
Lin teaches, wherein a difference between processing results obtained by processing same data using the second neural network model and the first neural network model falls within a preset range ([0011]: Describes separately inputting the same data into a non-optimized neural network model and an optimized neural network model to obtain respective processing results, and comparing absolute values of differences between the prediction results, which corresponds to a difference between processing results obtained by the first and second neural network models falling within a preset range.).
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to have combined the model network of Lu in view of Kim with the output difference evaluation of Lin. Doing so would have enabled the system to maintain the second models results remain close to the first results to maintain performance with smaller models, improving reliability in the models.
Regarding claim 3, Lu in view of Kim in view of Lin teaches, wherein a difference between processing results obtained by processing same data using the updated second neural network model and the first neural network model falls within the preset range([0011]: Describes processing the same data using a baseline model and an optimized model and comparing differences between their prediction results.).
Regarding claims 10, and 11, which recites substantially the same limitations as claims 2, and 3 and further recites an apparatus comprising one or more processors (Lu, Col. 14 lines 33-41: Describes a computing device to execute the methods using computer hardware executing software using computer components.)to perform the method steps of claims 2, and 3, respectively and are rejected for the same reasons as described above.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DONALD T RODEN whose telephone number is (571)272-6441. The examiner can normally be reached Mon-Thur 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at (571) 272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.T.R./Examiner, Art Unit 2128
/OMAR F FERNANDEZ RIVAS/Supervisory Patent Examiner, Art Unit 2128
Prosecution Insights