Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to under 37 CFR 1.83(a). The drawings must show every feature of the invention specified in the claims. Therefore, the empty boxes in figures 1, 2 and 10-17 must be labeled or the features that correspond to the empty boxes must be canceled from the claims. No new matter should be entered.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
A series of singular dependent claims is permissible in which a dependent claim refers to a preceding claim which, in turn, refers to another preceding claim.
A claim which depends from a dependent claim should not be separated by any claim which does not also depend from said dependent claim. It should be kept in mind that a dependent claim may refer to any preceding independent claim. In general, applicant's sequence will not be changed. See MPEP § 608.01(n).
Claims 16-18 are out of order.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 7-11, 13-15, 17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Double Similarity Distillation for Semantic Image Segmentation by Feng et al, End-to-End Object Detection with Transformers by Carion et al and US20210182662A1 to Lai et al.
Claims 5 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Double Similarity Distillation for Semantic Image Segmentation by Feng et al, End-to-End Object Detection with Transformers by Carion et al, US20210182662A1 to Lai et al and MINILM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers by Wang et al.
Claims 16, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Double Similarity Distillation for Semantic Image Segmentation by Feng et al, End-to-End Object Detection with Transformers by Carion et al, US20210182662A1 to Lai et al and Zero-Shot Knowledge Distillation in Deep Networks by Nayak et al.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Double Similarity Distillation for Semantic Image Segmentation by Feng et al, End-to-End Object Detection with Transformers by Carion et al, US20210182662A1 to Lai et al and HalluciNet-ing Spatiotemporal Representations Using a 2D-CNN by Parmar et al.
Feng teaches claims 1, 14 and 15. A method, comprising: (Feng Fig. 3)
receiving, via an input, input data, wherein the input data includes at least one of: image data and/or video data; and (The input image in Feng fig. 3, below.)
PNG
media_image1.png
256
184
media_image1.png
Greyscale
generating location data indicative of a location of any detected at least one feature of interest in the received input data, wherein:
the location data is generated using a first machine learning, ML, model configured to detect whether or not there is at least one feature of interest in the received input data using a (Feng fig. 3 the location data is generated by the first student model’s backbone, below)
PNG
media_image2.png
104
132
media_image2.png
Greyscale
the first ML model is trained based on a learning process implemented by a second ML model configured to detect whether or not there is at least one feature of interest in the received input data, wherein the second ML model uses a (Feng fig. 3 second model is the teacher network. Feng fig. 3 p 5367 “Fig. 3. The pipeline of DSD framework. During the training process, teacher network is fixed and the student network is updated by the ground-truth labels and the knowledge transferred from the teacher network. The proposed pixel-wise similarity distillation module captures detailed spatial dependencies across the multiple layers of the network. The category-wise similarity distillation module strengthens the global category correlation.” Both the student and the teacher have the same transformer based architecture.)
the first ML model and the second ML model are each configured to use an attention mechanism to generate: (a) at least one attention map from at least one layer of the first ML model; and (b) a plurality of attention maps from a plurality of layers of the second ML model, (Feng fig. 3 residual attention maps.)
the first ML model comprises fewer layers than the second ML model; (Feng fig. 3 shows fewer layers, below.)
PNG
media_image3.png
356
164
media_image3.png
Greyscale
at least one attention map generated by the second ML model is used to train the first ML model; (Feng fig. 3 similarity loss, below)
PNG
media_image4.png
112
184
media_image4.png
Greyscale
comparing attention maps generated by the first and second ML models to determine whether or not the first ML model (Feng fig. 4 and p. 5367 “the residual attention map can be calculated through subtraction between attention maps, which are obtained by attention mapping operation on feature maps…. We adopt L2 loss to calculate the pixel-wise similarity distillation loss between the teacher and the student networks…” minimal L2 loss is the similarity metric.)
PNG
media_image5.png
194
348
media_image5.png
Greyscale
(Feng algorithm 1. Minimal L2 loss is the similarity metric, cross-entropy loss is loss based on ground truth. Total loss is the modified loss function that updates the parameters of the student network, model 1.)
Feng isn’t a transformer architecture.
However, Carion teaches a transformer architecture. (Carion abs. “We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed compo nents like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bi partite matching, and a transformer encoder-decoder architecture.”)
Carion, Feng and the claims are all processing images. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use an encoder-decoder transformer architecture in Feng because Carion’s “DETR can be easily generalized to produce panoptic segmentation in a unified manner.” Carion abs.
Feng doesn’t teach a “meets” determination.
However, Lai teaches first ML model meets a similarity metric… in response to determining that the first ML model does not meet the similarity metric… (Lai para 30 stops training “After the student model is fully trained, ideally, the loss functions tend to zero, close to zero, or less than a threshold value (e.g., depending on how well trained the student model is).”)
Feng, Lai and the claims all have a teacher-student model. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to update or not update the student after the loss between the student and teacher met a minimum value, and Lai just teaches a few varieties of minimum values, “zero, close to zero, or less than a threshold value…” Lai para 30.)
Feng teaches claim 2. The computer-implemented method of claim 1, wherein the first and second ML models are based on a detection transformer, DETR, architecture, wherein the at least one layer of the first and second ML models comprises a transformer layer. (Carion abs. “We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed compo nents like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bi partite matching, and a transformer encoder-decoder architecture.”)
Feng teaches claim 3. The method of claim 2, wherein the detection transformer architecture comprises a backbone neural network configured to down-sample the input data to produce a tensor of activations for processing by the at least one transformer layer of the first and second ML models, (Feng fig. 3 shows fewer layers, below.)
PNG
media_image3.png
356
164
media_image3.png
Greyscale
Feng doesn’t teach a transformer.
However, Carion teaches wherein the at least one transformer layer of the first and second ML models are based on an encoder-decoder transformer architecture for predicting the location of the at least one feature of interest and/or outputting data representative of the predicted location of the at least one feature of interest. (Carion abs. “We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed compo nents like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bi partite matching, and a transformer encoder-decoder architecture.”)
Feng teaches claim 5. The method of claim 14, wherein the similarity metric is based on a (Feng fig. 4 and p. 5367 “the residual attention map can be calculated through subtraction between attention maps, which are obtained by attention mapping operation on feature maps…. We adopt L2 loss to calculate the pixel-wise similarity distillation loss between the teacher and the student networks…” minimal L2 loss is the similarity metric.)
Feng doesn’t teach KL divergence.
However, x teaches a Kullback-Leibler, KL, divergence score. (Wang fig. 1 below)
PNG
media_image6.png
236
1114
media_image6.png
Greyscale
Wang, Feng and the claims all compare attention values. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use KL divergence in the comparison because “mean squared error (MSE) and KL-divergence are often used as loss functions.” Wang sec. 2.3.
Feng teaches claim 6. The method of claim 5, wherein the KL divergence score comprises a first component and a second component, wherein the first component is configured to apply knowledge distillation to the at least one attention map generated by the at least one layer of the first and second ML models by attempting to match the attention maps generated by the first and second ML models, (Wang fig. 1 attention transfer in fig. 1) and wherein the second component is configured to apply knowledge distillation to class label predictions. (Wang Fig. 1 “Figure 1. Overview of Deep Self-Attention Distillation. The student is trained by deeply mimicking the self-attention behavior of the last Transformer layer of the teacher. In addition to the self-attention distributions, we introduce the self-attention value-relation transfer to help the student achieve a deeper mimicry.” Value relation transfer is the second component.)
PNG
media_image7.png
458
1084
media_image7.png
Greyscale
Feng teaches claims 7, 17 and 19. The method of claim 1, further comprising using a hyper-parameter to control mixing between loss based on the similarity metric and loss based on the ground-truth target labels when training the first and second ML models. (Feng sec. III p. 5369 “where α and β are loss weights to make these loss values ranges comparable.” Alpha and beta are the hyperparameter for mixing losses. The cross entropy loss weight is one and everything is mixed with alpha and beta.)
Feng teaches claim 8. The method of claim 1, wherein the at least one attention map generated by the second ML model used to train the first ML model is distilled from the plurality of attention maps generated by the second ML model. (Feng p. 5365 “PSD module utilizes residual attention maps through subtraction between any two attention maps.” Feng fig. 4 below.)
PNG
media_image8.png
216
348
media_image8.png
Greyscale
Feng teaches claim 9. The method of claim 1, comprising generating, using the first ML model, an attention map representative of the generated location data. (Feng fig. 4 below and Fig. 3 showing input images.)
PNG
media_image8.png
216
348
media_image8.png
Greyscale
Feng teaches claim 10. The method of claim 9, wherein the attention map is generated:
by at least one encoder of the at least one layer;
(Feng fig. 3 encoder and generated attention map below.)
PNG
media_image9.png
232
286
media_image9.png
Greyscale
Feng doesn’t teach the decoder and the combination.
However, Carion teaches by at least one decoder of the at least one layer; or
based on a combination of the at least one encoder and decoder of the at least one layer. (Carion fig. 2 below, and p. 7 “Transformer decoder. The decoder follows the standard architecture of the transformer, transforming N embeddings of size d using multi-headed self- and encoder-decoder attention mechanisms.”)
PNG
media_image10.png
158
496
media_image10.png
Greyscale
Feng teaches claim 11. The method of claim 9, comprising causing a display to show the generated attention map. (Feng “Fig. 5. Visualization of the attention maps…” below.)
PNG
media_image11.png
160
348
media_image11.png
Greyscale
Feng teaches claim 12. The method of any of claim 1, wherein the received input data comprises (Feng fig. 3 shows data input to both models, below.)
PNG
media_image3.png
356
164
media_image3.png
Greyscale
Feng doesn’t teach a 3d teacher and a 2d convolutional student.
However, Parmar teaches received input data comprises three-dimensional data and/or temporal data used by the second ML model, the method further comprising implementing a convolution procedure to reduce the received input data to a dimensional format for use by the first ML model. (Parmar abs. “convolutional neural networks (CNN)…We propose to hallucinate spatiotemporal representations from a 3D-CNN teacher with a 2D CNN student.”)
Feng, Parmar and the claims all use a teacher student to process images. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to us Parmar in Feng because “experimental evaluation shows that the hallucination task indeed helps improve performance on action recognition, action quality assessment, and dynamic scene recognition tasks.” Parmar abs.
Feng teaches claim 13. The method of claim 1, comprising:
receiving an indication to use the second ML model instead of the first ML model to generate the location data from the received input data; and
in response to receiving the indication, generating the location data using the second ML model. (Feng fig. 3 below, the indication is the input image.)
PNG
media_image12.png
306
382
media_image12.png
Greyscale
Feng teaches claims 16, 18 and 20. The method of claim 7, wherein the hyper-parameter is selected such that the contribution of the loss based on the similarity metric to the loss function is (Feng sec. III p. 5369 “where α and β are loss weights to make these loss values ranges comparable.” Alpha and beta are the hyperparameter for mixing losses. The cross entropy loss weight is one and everything is mixed with alpha and beta.)
Feng doesn’t teach 60-90%.
However, Nayak teaches the hyper-parameter is selected such that the contribution of the loss based on the similarity metric to the loss function is between 60% and 90%. (Nayak p. 15 ” We take λ = 0.3 which is the weight given to cross entropy loss and the distillation loss is given the weight as 1.0.” This yields a mix 1/(1+.3) which is between 60 and 90%, ~76%.)
Feng, Nayak and the claims all teach student-teacher loss. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use a mix of about 76% because Nayak teaches that mix and Nayak says their “framework results in competitive generalization performance as achieved by distillation…” Nayak abs.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Austin Hicks whose telephone number is (571)270-3377. The examiner can normally be reached Monday - Thursday 8-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUSTIN HICKS/Primary Examiner, Art Unit 2142
Prosecution Insights