DETAILED ACTION
This Office Action is sent in response to Applicant’s Communication received 2/15/2024 for application number 18/443,070.
Claims 1-20 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-3, 5-8, 12-16, and 19-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Bao et al., Using Distillation to Improve Network Performance after Pruning and Quantization (see NPL [U] in Notice of References Cited).
In reference to claim 1, Bao discloses a processor (4.1 Experimental environment, page 5) comprising: one or more circuits to cause precision of one or more layers of one or more first versions of a neural network (pruned student model is first version, and precision is changed by quantizing weights, 3. Methodology and figs. 1-3, pages 4-5) to be modified based, at least in part, on a change in accuracy of one or more corresponding layers of one or more second versions of the neural network caused by deactivation of one or more weights of the one or more corresponding layers (quantization is based on distillation loss between teacher model, which is the second version of the NN that is dense, and the current student model, which has been pruned, i.e. weights deactivated, 3. Methodology and figs. 1-3, pages 4-5; also see Polino et al. Model Compression via Distillation and Quantization, NPL [V] at pages 3-6, which is cited in Bao as the quantization method and gives additional detail on how quantization is performed).
In reference to claim 2, Bao discloses the processor of claim 1, wherein the one or more circuits are to quantify the change in accuracy of the one or more corresponding layers of the one or more second versions of the neural network based, at least in part, on one or more comparisons of activations of one or more sparse versions and one or more dense versions of the neural network (cross-entropy soft target of distillation loss is based on comparison of activations of teacher and student models, 3. Methodology, page 5).
In reference to claim 3, Bao discloses the processor of claim 1, wherein the one or more circuits are to generate one or more other weights to be applied to one or more loss values based, at least in part, on the change in accuracy of the one or more corresponding layers of the one or more second versions of the neural network (quantization is based on distillation loss between teacher model, 3. Methodology, page 5).
In reference to claim 5, Bao discloses the processor of claim 1, wherein the one or more circuits are to cause the precision of the one or more layers of the one or more first versions of a neural network to be modified using a scale factor based, at least in part, on one or more of hard label predictions, soft logits, or feature maps, of the one or more first and second versions of the neural network (quantization scaling function based on hard and soft distillation losses between teacher and student network, 3. Methodology page 5).
In reference to claim 6, Bao discloses the processor of claim 1, wherein the one or more circuits are to cause the precision of the one or more layers of the one or more first versions of the neural network to be modified based, at least in part, on a comparison of an accuracy metric of the one or more first versions of the neural network and an accuracy metric of the one or more second versions of the neural network (quantization based on distillation losses, which is an accuracy metric, between teacher and student network, 3. Methodology page 5).
In reference to claim 7, Bao discloses the processor of claim 1, wherein the one or more circuits are to cause the precision of the one or more layers of the one or more first versions of a neural network to be modified using one or more of hard label distillation losses, soft logit distillation losses, or feature-based distillation losses based, at least in part, on the one or more first versions of the neural network and the one or more second versions of a neural network (quantization scaling function based on hard and soft distillation losses between teacher and student network, 3. Methodology, page 5).
In reference to claim 8, this claim is directed to a system associated with the apparatus claimed in claim 1 and is therefore rejected under a similar rationale.
In reference to claim 12, Bao discloses the system of claim 8, wherein the one or more processors are to cause the precision of the one or more layers of the one or more first versions of a neural network to be modified using a scale factor based, at least in part, on an overall calibration loss of one or more sparse floating-point versions and the one or more first versions of the neural network (quantizing scaling based on loss between dense teacher and pruned, but not yet quantized student models, 3. Methodology, page 5).
In reference to claim 13, Bao discloses the system of claim 8, wherein the one or more processors are to cause the precision of the one or more layers of the one or more first versions of the neural network to be modified based, at least in part, on an overall pruning loss of one or more dense versions and the one or more second versions of the neural network (quantization scaling function based on losses from pruning, 3. Methodology page 5).
In reference to claim 14, Bao discloses the method of claim 15, wherein the one or more processors are to quantify the change in accuracy of the one or more corresponding layers of the one or more second versions of the neural network based, at least in part, on one or more comparisons of activations of one or more dense full-precision versions and one or more sparse full-precision versions of the neural network (pruned and quantized network can be used for inference, 4. Experiments and Results, pages 5-6).
In reference to claim 15, this claim is directed to a method associated with the apparatus claimed in claim 1 and is therefore rejected under a similar rationale.
In reference to claim 16, Bao discloses the method of claim 15, wherein the one or more processors are to quantify the change in accuracy of the one or more corresponding layers of the one or more second versions of the neural network based, at least in part, on one or more comparisons of activations of one or more dense full-precision versions and one or more sparse full-precision versions of the neural network (loss based on comparison of activations of dense teacher and pruned, but not yet quantized student models, 3. Methodology, page 5).
In reference to claim 19, Bao discloses the method of claim 15, wherein the one or more processors are to cause the precision of the one or more layers of the one or more first versions of a neural network to be modified based, at least in part, on one or more knowledge distillation neural network modification techniques (quantization based on knowledge distillation techniques, 3. Methodology, pages 4-5).
In reference to claim 20, Bao discloses the method of claim 15, wherein the one or more processors are to cause the precision of the one or more layers of the one or more first versions of the neural network to be modified based, at least in part, on an overall pruning loss of one or more dense floating-point versions of the neural network and the one or more second versions of the neural network that are one or more sparse full-precision versions of the neural network (quantization scaling function based on losses from pruning, 3. Methodology page 5).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 4, 9-11, and 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bao et al., Using Distillation to Improve Network Performance after Pruning and Quantization (see NPL [U] in Notice of References Cited) as applied to claims 1, 8, and 15 above, and in further view of Lin et al., Knowledge Distillation via the Target-aware Transformer (see NPL [W] in Notice of References Cited).
In reference to claim 4, Bao does not explicitly teach the processor of claim 1, wherein the one or more circuits are to quantify the change in accuracy of the one or more corresponding layers based, at least in part, on feature calibration loss values of the one or more corresponding layers.
Lin teaches the processor of claim 1, wherein the one or more circuits are to quantify the change in accuracy of the one or more corresponding layers based, at least in part, on feature calibration loss values of the one or more corresponding layers (during knowledge distillation, the overall distillation loss is based on a feature map loss between teacher and student networks, 3. Method, pages 10917-18).
It would have been obvious to one of ordinary skill in art, having the teachings of Bao and Lin before the earliest effective filing date, to modify the losses of Bao to include the feature calibration loss of Lin.
One of ordinary skill in the art would have been motivated to modify the losses of Bao to include the feature calibration loss of Lin because it helps boost the performance of vision models after knowledge distillation (Lin, page 10915).
In reference to claim 9, Bao teaches the system of claim 8, wherein the one or more processors are to quantify the change in accuracy of the one or more corresponding layers of the one or more second versions of the neural network based, at least in part, … [the] one or more sparse versions and one or more dense versions of the neural network (quantization based on distillation losses between teacher and student network, 3. Methodology page 5).
However, Bao does not explicitly teach one or more comparisons of activated features.
Lin teaches one or more comparisons of activated features (during knowledge distillation, the overall distillation loss is based on a feature map loss between teacher and student networks, 3. Method, pages 10917-18).
It would have been obvious to one of ordinary skill in art, having the teachings of Bao and Lin before the earliest effective filing date, to modify the losses of Bao to include the feature calibration loss of Lin.
One of ordinary skill in the art would have been motivated to modify the losses of Bao to include the feature calibration loss of Lin because it helps boost the performance of vision models after knowledge distillation (Lin, page 10915).
In reference to claim 10, Bao teaches the system of claim 8, wherein the one or more processors are to generate one or more other weights to be applied to one or more loss values based, at least in part, … the one or more corresponding layers of the one or more first versions and the one or more second versions of the neural network (quantization based on distillation losses between teacher and student network, 3. Methodology page 5).
However, Bao does not explicitly teach feature distillation loss values.
Lin teaches feature distillation loss values (during knowledge distillation, the overall distillation loss is based on a feature map loss between teacher and student networks, 3. Method, pages 10917-18).
It would have been obvious to one of ordinary skill in art, having the teachings of Bao and Lin before the earliest effective filing date, to modify the losses of Bao to include the feature calibration loss of Lin.
One of ordinary skill in the art would have been motivated to modify the losses of Bao to include the feature calibration loss of Lin because it helps boost the performance of vision models after knowledge distillation (Lin, page 10915).
In reference to claim 11, Bao teaches the system of claim 8, wherein the one or more processors are to cause the precision of the one or more layers of the one or more first versions of a neural network to be modified based, at least in part, on a comparison of … corresponding layers of one or more sparse floating-point versions and the one or more first versions of the neural network (quantization based on distillation losses between teacher and student network, 3. Methodology page 5).
However, Bao does not explicitly teach feature calibration loss values.
Lin teaches feature calibration loss values (during knowledge distillation, the overall distillation loss is based on a feature map loss between teacher and student networks, 3. Method, pages 10917-18).
It would have been obvious to one of ordinary skill in art, having the teachings of Bao and Lin before the earliest effective filing date, to modify the losses of Bao to include the feature calibration loss of Lin.
One of ordinary skill in the art would have been motivated to modify the losses of Bao to include the feature calibration loss of Lin because it helps boost the performance of vision models after knowledge distillation (Lin, page 10915).
In reference to claim 17, Bao teaches the method of claim 15, wherein the one or more processors are to modify one or more other weights applied to [losses] based, at least in part, on an overall calibration loss of the one or more first versions and the one or more second versions of the neural network (quantization based on distillation losses between teacher and student network, 3. Methodology page 5).
However, Bao does not explicitly teach one or more feature calibration loss values.
Lin teaches one or more feature calibration loss values (during knowledge distillation, the overall distillation loss is based on a feature map loss between teacher and student networks, 3. Method, pages 10917-18).
It would have been obvious to one of ordinary skill in art, having the teachings of Bao and Lin before the earliest effective filing date, to modify the losses of Bao to include the feature calibration loss of Lin.
One of ordinary skill in the art would have been motivated to modify the losses of Bao to include the feature calibration loss of Lin because it helps boost the performance of vision models after knowledge distillation (Lin, page 10915).
In reference to claim 18, Bao teaches the method of claim 15, wherein the one or more processors are to cause the precision of the one or more layers of the one or more first versions of a neural network to be modified based, at least in part, on a comparison of … corresponding layers of one or more sparse floating-point versions and the one or more first versions of the neural network that are one or more sparse quantized versions of the neural network (quantization based on distillation losses between teacher and student network, 3. Methodology page 5; it would be obvious that the teacher network could be the sparse model and the student the quantized model).
However, Bao does not explicitly teach feature calibration loss values.
Lin teaches feature distillation loss values (during knowledge distillation, the overall distillation loss is based on a feature map loss between teacher and student networks, 3. Method, pages 10917-18).
It would have been obvious to one of ordinary skill in art, having the teachings of Bao and Lin before the earliest effective filing date, to modify the losses of Bao to include the feature calibration loss of Lin.
One of ordinary skill in the art would have been motivated to modify the losses of Bao to include the feature calibration loss of Lin because it helps boost the performance of vision models after knowledge distillation (Lin, page 10915).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Andrew T. Chiusano whose telephone number is (571)272-5231. The examiner can normally be reached M-F, 10am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 571-272-4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW T CHIUSANO/Primary Examiner, Art Unit 2144
Prosecution Insights