DETAILED ACTION
This action is in response to the filing on 01/23/2026. Claims 1-2, 4-9, 11-13, and 15-20, are pending and have been considered below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2, 4-9, 11-13, and 15-20 are rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more.
Independent Claims 1, 11, and 12
Step 1:
Claims 1, 11, and 12 recite a method, manufacture, and system, respectively; therefore, they are directed to one of the four categories of statutory subject matter (process/method, machine/product/apparatus, manufacture, or composition of matter).
Step 2A Prong 1:
Claims 1 and 12 recite a method and system comprising:
generating a plurality of attention maps by inputting training data into a previously trained teacher model — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
generating a single unified attention map based on the plurality of attention maps — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mental process, or a concept that can be performed in the human mind with the use of a physical aid (e.g. pen and paper), including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III)). Or a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
extracting some elements constituting the single unified attention map — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mental process, or a concept that can be performed in the human mind with the use of a physical aid (e.g. pen and paper), including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III)). Or a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
in response to a determination that a dimension of an attention vector composed of the extracted elements and a dimension of attention weights of a student model are not identical, generating attention weights of the teacher model by applying linear interpolation to the attention vector composed of the extracted elements — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mental process, or a concept that can be performed in the human mind with the use of a physical aid (e.g. pen and paper), including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III)). Or a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
generating the attention weights of the student model by inputting the training data into the student model — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
calculating a value of a first loss function based on the attention weights of the teacher model and the attention weights of the student model — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mental process, or a concept that can be performed in the human mind with the use of a physical aid (e.g. pen and paper), including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III)). Or a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
calculating a value of a second loss function according to an inference of the student model with respect to the training data — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mental process, or a concept that can be performed in the human mind with the use of a physical aid (e.g. pen and paper), including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III)). Or a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
training the student model based on the value of the first loss function and the value of the second loss function — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
Claim 11 recites a manufacture comprising:
a computer program to execute the method of claim 1 — Under its broadest reasonable interpretation, this limitation encompasses the same abstract ideas recited by claim 1 for the same reasons as claim 1 above.
Step 2A Prong 2:
This judicial exception is not integrated into a practical application.
Claim 1 recites the additional elements of:
A method for training a model, comprising — This element amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). This element merely limits the use of the abstract idea to training a generic model.
Claim 11 recites the additional elements of:
A non-transitory computer-readable recording medium (analyzed as “A non-transitory computer-readable recording medium” to fall within a statutory category) comprising a computer program — This element amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). This element merely limits the use of the abstract idea to a generic computer component.
Claim 12 recites the additional elements of:
A server for training a model, comprising a memory to store instructions and a processor, wherein the processor is connected to the memory and configured to — This element amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). This element merely limits the use of the abstract idea to a generic computer comprising generic computer components.
Step 2B:
The claims do not contain significantly more than the judicial exception.
Claim 1 recites the additional elements of:
A method for training a model, comprising — This element amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). This element merely limits the use of the abstract idea to training a generic model.
Claim 11 recites the additional elements of:
A computer-readable recording medium (analyzed as “A non-transitory computer-readable recording medium” to fall within a statutory category) comprising a computer program — This element amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). This element merely limits the use of the abstract idea to a generic computer component.
Claim 12 recites the additional elements of:
A server for training a model, comprising a memory to store instructions and a processor, wherein the processor is connected to the memory and configured to — This element amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). This element merely limits the use of the abstract idea to a generic computer comprising generic computer components.
As such claims 1, 11, and 12 are not patent eligible.
Dependent Claims 2-9 and 13-20
Step 1:
Claims 2, 4-9 and 13, and 15-20 recite a method and system respectively; therefore, they are directed to one of the four categories of statutory subject matter (process/method, machine/product/apparatus, manufacture, or composition of matter).
Step 2A Prong 1:
Claims 2, 4-9 and 13, and 15-20 merely narrow the previously cited abstract idea limitations. For the reasons described above with respect to independent claims 1 and 12 this judicial exception is not meaningfully integrated into a practical application, or significantly more than the abstract idea. The claim(s) disclose similar limitations described for the independent claim(s) above and do not provide anything more than the abstract idea.
Claim 2 recites a method comprising:
wherein the generating the plurality of attention maps comprises generating an attention map from each of the plurality of consecutive transformer layers as the training data is input to the teacher model — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
Claims 4 and 15 recites a method and system comprising:
generating the attention vector composed of the extracted elements as the attention weights of the teacher model when the dimension of the attention vector and the dimension of the attention weights are identical — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mental process, or a concept that can be performed in the human mind with the use of a physical aid (e.g. pen and paper), including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III)). Or a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
wherein the generating the attention weights of the teacher model is performed for the dimension of the attention vector composed of the extracted elements and the dimension of the attention weights of the student model to be identical — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mental process, or a concept that can be performed in the human mind with the use of a physical aid (e.g. pen and paper), including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III)). Or a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
Claims 5 and 16 recites a method and system comprising:
generating a feature map for each time step by inputting the training data into the student model — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mental process, or a concept that can be performed in the human mind with the use of a physical aid (e.g. pen and paper), including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III)). Or a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
generating the attention weights of the student model based on the feature map for each time step and a query vector — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mental process, or a concept that can be performed in the human mind with the use of a physical aid (e.g. pen and paper), including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III)). Or a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
Claims 6 and 17 recites a method and system comprising:
performing a vector operation between each feature map and the query vector — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mental process, or a concept that can be performed in the human mind with the use of a physical aid (e.g. pen and paper), including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III)). Or a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
applying an activated function to a vector having each result of the vector operation as a component — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mental process, or a concept that can be performed in the human mind with the use of a physical aid (e.g. pen and paper), including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III)). Or a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
Claims 7 and 18 recites a method and system comprising:
calculating a value of a final loss function by making a weighted sum of the value of the first loss function and the value of the second loss function — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mental process, or a concept that can be performed in the human mind with the use of a physical aid (e.g. pen and paper), including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III)). Or a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
updating at least some of a plurality of parameters of the student model and a query vector in a direction in which the value of the final loss function decreases — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mental process, or a concept that can be performed in the human mind with the use of a physical aid (e.g. pen and paper), including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III)). Or a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
Claim 13 recites a method comprising:
wherein the processor, in generating the plurality of attention maps, is configured to generate an attention map from each of the plurality of consecutive transformer layers as the training data is input to the teacher model — Under its broadest reasonable interpretation, this limitation encompasses the abstract idea of a mathematical concept (see MPEP § 2106.04(a)(2)(I)), specifically organizing information and manipulating information through mathematical correlations.
Step 2A Prong 2:
This judicial exception is not integrated into a practical application.
Claims 2 and 13 recite the additional element of:
wherein the teacher model comprises a plurality of consecutive transformer layers — This element amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). This element merely limits the use of the abstract idea to a model with consecutive transformer layers.
Claim 8 recites the additional element of:
further comprising deploying an original or a copy of the trained student model as an on-device model to a user terminal — This element amounts to no more than insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)), and is well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II), receiving or transmitting data over a network).
Claims 9 and 20 recite the additional elements of:
receiving a version update request of the on-device model from the user terminal — This element amounts to no more than insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)), and is well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II), receiving or transmitting data over a network).
providing the user terminal with information about one or more parameters of a recently updated student model in response to the version update request — This element amounts to no more than insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)), and is well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II), receiving or transmitting data over a network).
Claim 19 recites the additional element of:
wherein the processor is configured to deploy an original or a copy of the trained student model as an on-device model to a user terminal — This element amounts to no more than insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)), and is well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II), receiving or transmitting data over a network).
Step 2B:
The claims do not contain significantly more than the judicial exception.
Claims 2 and 13 recite the additional element of:
wherein the teacher model comprises a plurality of consecutive transformer layers — This element amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). This element merely limits the use of the abstract idea to a model with consecutive transformer layers.
Claim 8 recites the additional element of:
further comprising deploying an original or a copy of the trained student model as an on-device model to a user terminal — This element amounts to no more than insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)), and is well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II), receiving or transmitting data over a network).
Claims 9 and 20 recite the additional elements of:
receiving a version update request of the on-device model from the user terminal — This element amounts to no more than insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)), and is well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II), receiving or transmitting data over a network).
providing the user terminal with information about one or more parameters of a recently updated student model in response to the version update request — This element amounts to no more than insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)), and is well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II), receiving or transmitting data over a network).
Claim 19 recites the additional element of:
wherein the processor is configured to deploy an original or a copy of the trained student model as an on-device model to a user terminal — This element amounts to no more than insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)), and is well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II), receiving or transmitting data over a network).
As such claims 2, 4-9 and 13, and 15-20 are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-8, 11-13, and 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Rubin et al. (US 2024/0371500 A1), hereinafter Rubin, in view of Yin et al. (US 2022/0180202 A1), hereinafter Yin, and further in view of Lee et al. (Learning to Distill Convolutional Features into Compact Local Descriptors), hereinafter Lee.
Regarding claim 1, Rubin teaches a method for training a model, comprising: (The method 100 may therefore leverage knowledge distillation from the second ML model in order to improve training and/or use of the first ML model. [see Rubin, para. 60]):
generating a plurality of attention maps by inputting training data into a teacher model (Rubin discloses generating a plurality of attention maps [see Rubin, para. 75] by inputting training data into the teacher model [see Rubin, para. 74]);
generating a single unified attention map based on the plurality of attention maps (Rubin discloses distilling the plurality of attention maps of the teacher model into at least one attention map [see Rubin, para. 134]. Thus, the plurality of attention maps can be distilled into a single attention map which constitutes a set of attention weights);
extracting some elements constituting the single unified attention map (Rubin discloses distilling the plurality of attention maps of the teacher model into at least one attention map [see Rubin, para. 134]. Thus, the plurality of attention maps can be distilled into a single attention map which constitutes a set of attention weights);
in response to an attention vector composed of the extracted elements, generating attention weights of the teacher model based on the attention vector composed of the extracted elements (Rubin discloses distilling the plurality of attention maps of the teacher model into at least one attention map [see Rubin, para. 134]. Thus, the plurality of attention maps can be distilled into a single attention map which constitutes a set of attention weights);
generating the attention weights of the student model by inputting the training data into the student model (Rubin discloses generating a plurality of attention maps [see Rubin, para. 75] by inputting training data into the student model [see Rubin, para. 74]. Thus, the attention maps constitute a set of attention weights of the student model);
calculating a value of a first loss function based on the attention weights of the teacher model and the attention weights of the student model (Rubin discloses the loss function according to the attention matrices of the student and teacher networks using Kullback Leibler divergence [see Rubin, para. 113 and Equation 3]);
calculating a value of a second loss function according to an inference of the student model with respect to the training data (Rubin discloses the loss equation according to inference of the student model according to the ground truth from the training data [see Rubin, para. 108 and Equation 1]);
training the student model based on the value of the first loss function and the value of the second loss function (Rubin discloses training the student model based on the values of the attention loss function and inference loss function [see Rubin, para. 12, para. 114-115, and Equation 3]).
However, Rubin fails to teach a previously trained teacher model and in response to a determination that a dimension of an attention vector composed of the extracted elements and a dimension of attention weights of a student model are not identical, generating attention weights of the teacher model by applying linear interpolation to the attention vector composed of the extracted elements.
In the same field of endeavor, Yin teaches:
a previously trained teacher model (Yin discloses using a pre-trained teacher model to perform knowledge transfer to a student model [see Yin, para. 180]).
in response to a determination that a dimension of an attention vector of a teacher model and a dimension of an attention vector of a student model are not identical, generating attention weights by applying linear interpolation to the attention vector (Yin discloses identifying the student models representation dimensions being less than the teacher models representation dimensions and applying a linear transformation to correct it [see Yin, para. 224]).
It would have been obvious to one of ordinary skill, in the art at the time before the effective filing date of the invention to incorporate a previously trained teacher model and in response to a determination that a dimension of an attention vector of a teacher model and a dimension of an attention vector of a student model are not identical, generating attention weights by applying linear interpolation to the attention vector as suggested in Rubin into Yin because both methods perform knowledge distillation of a teacher model into a student model (see Rubin, Abstract; see Yin, Abstract). Incorporating the teaching of Rubin into Yin would provide a target student model by performing effective knowledge transfer on a student model based on sample data output by a pre-trained teacher model (see Yin, para. 180).
It would have been further obvious to one of ordinary skill, in the art at the time before the effective filing date of the invention that incorporating in response to a determination that a dimension of an attention vector of a teacher model and a dimension of an attention vector of a student model are not identical, generating attention weights by applying linear interpolation to the attention vector as suggested in Yin into Rubin would render obvious in response to a determination that a dimension of an attention vector composed of the extracted elements and a dimension of attention weights of a student model are not identical, generating attention weights of the teacher model by applying linear transformation to the attention vector composed of the extracted elements because Rubin discloses an attention vector composed of extracted elements from the plurality of attention maps [see Rubin, para. 41 and 134], and using linear transformations to match dimensions of attention vectors when one is smaller was used in Yin to match a student and teacher attention vector [see Yin, para. 224], thus, the same method of Yin could be applied to match the dimensions of the extracted attention vector to match the student attention vector, resulting in applying a linear transformation to the extracted attention vector to get attention weights for the teacher model matching the dimensions of the student model.
However, the combination of Rubin and Yin fails to teach generating attention weights of the teacher model by applying linear interpolation to the attention vector composed of the extracted elements.
In the same field of endeavor, Lee teaches:
applying linear interpolation to the matrix (Lee discloses applying linear interpolation to upsample feature maps to a defined dimensionality [see Lee, Figure 2 and Section 3.1, Subsection Feature fusion, para. 1]).
It would have been obvious to one of ordinary skill, in the art at the time before the effective filing date of the invention to incorporate applying linear interpolation to the matrix as suggested in Lee into the combination of Rubin and Yin to incorporate generating attention weights of the teacher model by applying linear interpolation to the attention vector composed of the extracted elements because through the combination of Rubin and Lee, it would have been obvious to one of ordinary skill in the art before the effective filing date that when distilling the plurality of attention maps of the teacher model into the at least one attention map [see Rubin, para. 134], if the dimensions of the student attention map are larger than the dimensions of the distilled feature map, then the linear interpolation method taught by Lee for feature map distillation [see Lee, Figure 2] could be used for attention map distillation as Rubin identified feature maps and attention maps are analogous types of 'location information' [see Rubin, para. 41]. Further, using the distillation method taught by Lee to distill the plurality of attention maps into one attention map would result in the extracted feature/attention map being of a smaller dimensionality than the student attention map, such that it would then require the upsampling by linear interpolation as taught by Lee [see Lee, Figure 2 and Section 3.1, Subsection Feature fusion, para. 1] and would achieve significantly improved performance over existing models, with up to a 100 times smaller size of descriptors (see Lee, Abstract). It would have been obvious to one of ordinary skill, in the art at the time before the effective filing date of the invention to incorporate the teaching of Lee into the combination of Rubin and Yin because both methods perform knowledge distillation in neural networks (see Rubin, para. 60; see Lee, Abstract).
Regarding claim 2, the combination of Rubin, Yin, and Lee as applied in claim 1 above teaches all the limitations of claim 1 and further teaches:
wherein the teacher model comprises a plurality of consecutive transformer layers (Rubin discloses that the teacher model uses the detection transformer, DETR, architecture which comprises a plurality of transformer layers [see para. 7]. Thus, the at least one intermediate layers can be a plurality of consecutive transformer layers), and
wherein the generating the plurality of attention maps comprises generating an attention map from each of the plurality of consecutive transformer layers as the training data is input to the teacher model (Rubin discloses generating a plurality of attention maps [see Rubin, para. 75] by inputting training data into the teacher model [see Rubin, para. 74], and that the layers that produce the attention maps are transformer encoder-decoder layers [see Rubin, para. 97]).
Regarding claim 4, the combination of Rubin, Yin, and Lee as applied in claim 1 above teaches all the limitations of claim 1 and further teaches:
generating the attention vector composed of the extracted elements as the attention weights of the teacher model when the dimension of the attention vector and the dimension of the attention weights are identical (Rubin discloses that the representations provided by the models include attention maps and other kinds of location maps like feature maps [see Rubin, para. 41], and further discloses an attention map composed of distilled elements of the plurality of attention maps [see Rubin, para. 134], and attention matrice(s) of the student and teacher models which would have at least one dimension [see Rubin, para 113]. Yin discloses identifying the student models representation dimensions being less than the teacher models preorientation dimensions and applying a linear transformation to correct it [see Yin, para. 224]. Thus, the representation in the form of an attention map would be identified by the methods of Yin as being identical after the linear transformation, and the distilled attention map taught by Rubin can be used as the attention of the teacher model), wherein
the generating the attention weights of the teacher model is performed for the dimension of the attention vector composed of the extracted elements and the dimension of the attention weights of the student model to be identical transformation (Rubin discloses that the representations provided by the models include attention maps and other kinds of location maps like feature maps [see Rubin, para. 41], and further discloses an attention map composed of distilled elements of the plurality of attention maps [see Rubin, para. 134], and attention matrice(s) of the student and teacher models which would have at least one dimension [see Rubin, para 113]. Yin discloses identifying the student models representation dimensions being less than the teacher models preorientation dimensions [see Yin, para. 224]. Thus, the representation in the form of an attention map would be identified by the methods of Yin as being identical and carry out the generation method of Rubin).
Regarding claim 5, the combination of Rubin, Yin, and Lee as applied in claim 1 above teaches all the limitations of claim 1 and further teaches:
wherein the generating the attention weights of the student model comprises: generating a feature map for each time step by inputting the training data into the student model (Rubin discloses generating a feature map by inputting data into the student model [see Rubin, para. 73]. Rubin further discloses that images or video, i.e. a collection of sequential images, are input into the networks [see Rubin, para. 74]. Further, Rubin discloses that the input data may include temporal data [see Rubin, para. 17]. Thus, when generating feature maps, there are a plurality of images to be input and processing each image one at a time would require multiple time steps, and may include temporal information); and
generating the attention weights of the student model based on the feature map for each time step and a query vector (Rubin discloses generating the attention map, including attention weights, based on the learned query matrices, network weight matrices, and feature representation sequence [see Rubin, para. 111-113 and Equation 2], wherein the feature representation sequence is, or includes, a feature map [see Rubin, para. 73], wherein the feature maps are generated for each image in a plurality of images and may include temporal information [see Rubin, para. 17 and 74]).
Regarding claim 6, the combination of Rubin, Yin, and Lee as applied in claim 1 above teaches all the limitations of claim 5 and further teaches:
wherein the generating the attention weights of the student model based on the feature map for each time step and the query vector comprises: performing a vector operation between each feature map and the query vector; and applying an activated function to a vector having each result of the vector operation as a component (Rubin discloses generating the attention map, including attention weights, based on the learned query matrices, network weight matrices, and feature representation sequence by performing a plurality of matrix operations and finally applying a softmax activation function [see Rubin, para. 111-113 and Equation 2], wherein the feature representation sequence is, or includes, a feature map [see Rubin, para. 73], wherein the feature maps are generated for each image in a plurality of images and may include temporal information [see Rubin, para. 17 and 74]).
Regarding claim 7, the combination of Rubin, Yin, and Lee as applied in claim 1 above teaches all the limitations of claim 1 and further teaches:
wherein the training the student model comprises: calculating a value of a final loss function by making a weighted sum of the value of the first loss function and the value of the second loss function (Rubin discloses training the student model based on the values of the attention loss function and inference loss function [see Rubin, para. 12, para. 114-115, and Equation 3]); and
updating at least some of a plurality of parameters of the student model and a query vector in a direction in which the value of the final loss function decreases (Rubin discloses that multi-headed scaled dot-product attention is applied to learned query and key matrices [see Rubin, para. 112 and Equation 2] which is further used in the final loss function in the attention matrices of the student and teacher model to update the student model [see Rubin, para. 12, 114-115, and Equation 3]).
Regarding claim 8, the combination of Rubin, Yin, and Lee as applied in claim 1 above teaches all the limitations of claim 1 and further teaches:
further comprising deploying an original or a copy of the trained student model as an on-device model to a user terminal (Rubin discloses that the student model may be deployed based on user requirements and/or conditions [see Rubin, para. 90]. Yin discloses that the user equipment includes an intelligent terminal which has requirements for a deployed model [see Yin, para. 5 and 106, and FIG. 1]. Thus, the combination of Rubin and Yin would deploy the student model to the user terminal based on the user requirements/conditions).
Regarding claim 11, claim 11 contains substantially similar limitations to those found in claim 1. Therefore it is rejected for the same reason as claim 1 above. Additionally, the combination of Rubin and Yin further teaches:
a non-transitory computer-readable recording medium comprising a computer program to execute the method (Embodiments in the present disclosure can be provided as methods, systems or as a combination of machine-readable instructions and processing circuitry. Such machine-readable instructions may be included on a non-transitory machine (for example, computer) readable storage medium (including but not limited to disc storage, CD-ROM, optical storage, etc.) having computer readable program codes therein or thereon. [see Rubin, para. 170]).
Regarding claim 12, claim 12 contains substantially similar limitations to those found in claim 1. Therefore it is rejected for the same reason as claim 1 above. Additionally, the combination of Rubin and Yin further teaches:
a server for training a model, comprising a memory to store instructions and a processor, wherein the processor is connected to the memory and configured to (Rubin discloses that the present disclosure can be operated by a system as a combination of machine-readable instructions and processing circuitry [see para. 170]. It would have been obvious to one of ordinary skill in the art to incorporate the combination machine-readable instructions and processing circuitry on a general purpose computer, with a processor and memory, as a server).
Regarding claim 13, claim 13 contains substantially similar limitations to those found in claim 2 above. Consequently, claim 13 is rejected for the same reasons.
Regarding claim 15, claim 15 contains substantially similar limitations to those found in claim 4 above. Consequently, claim 15 is rejected for the same reasons.
Regarding claim 16, claim 16 contains substantially similar limitations to those found in claim 5 above. Consequently, claim 16 is rejected for the same reasons.
Regarding claim 17, claim 17 contains substantially similar limitations to those found in claim 6 above. Consequently, claim 17 is rejected for the same reasons.
Regarding claim 18, claim 18 contains substantially similar limitations to those found in claim 7 above. Consequently, claim 18 is rejected for the same reasons.
Regarding claim 19, claim 19 contains substantially similar limitations to those found in claim 8 above. Consequently, claim 19 is rejected for the same reasons.
Claims 9 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Rubin et al. (US 2024/0371500 A1), hereinafter Rubin, in view of Yin et al. (US 2022/0180202 A1), hereinafter Yin, and further in view of Lee et al. (Learning to Distill Convolutional Features into Compact Local Descriptors), hereinafter Lee, as applied in claim 1 above, and further in view of Gill et al. (US 11,227,122 B1), hereinafter Gill, and further in view of REZAZADEGAN TAVAKOLI et al. (US 2022/0012637 A1), hereinafter RT.
Regarding claim 9, the combination of Rubin, Yin, and Lee as applied in claim 1 above teaches all the limitations of claim 8 and further teaches:
the on-device model from the user terminal (Rubin discloses that the student model may be deployed based on user requirements and/or conditions [see Rubin, para. 90]. Yin discloses that the user equipment includes an intelligent terminal which has requirements for a deployed model [see Yin, para. 5 and 106, and FIG. 1]. Thus, the combination of Rubin and Yin would deploy the student model to the user terminal based on the user requirements/conditions).
However, the combination of Rubin, Yin, and Lee fails to teach receiving a version update request of the on-device model; and providing the user terminal with information about one or more parameters of a recently updated student model in response to the version update request.
In the same field of endeavor, Gill teaches:
receiving a version update request of the on-device model (Gill discloses a student model requesting to be retrained or updating to begin the training process [see Gill, Col. 18, lines 57-62]);
training in response to the version update request (Gill discloses a student model requesting to be retrained or updating to begin the training process [see Gill, Col. 18, lines 57-62]).
It would have been obvious to one of ordinary skill, in the art at the time before the effective filing date of the invention to incorporate receiving a version update request of the on-device model and training in response to the version update request as suggested in Gill into the combination of Rubin, Yin, and Lee because both methods perform knowledge distillation of a teacher model into a student model (see Rubin, Abstract; see Gill, Col. 2, lines 53-57). It would have been obvious to one of ordinary skill in the art at the time of the effective filing date to combine the prior art elements of the student retraining/update request of Gill with the knowledge distillation of Rubin to incorporate receiving a version update request of the on-device model and training in response to the version update request to achieve the predictable result of training student models based on a teacher model.
However, the combination of Rubin, Yin, Lee, and Gill fails to teach providing the user terminal with information about one or more parameters of a recently updated student model.
In the same field of endeavor, RT teaches:
providing the node with information about one or more parameters of a recently updated student model (RT discloses updating an edge node's on-device student model of a federated learning system with the parameters of another student network [see RT, para. 25], this is achieved by exchanging model parameters during training of the local student models [see RT, para. 90]).
It would have been obvious to one of ordinary skill, in the art at the time before the effective filing date of the invention to incorporate providing the node with information about one or more parameters of a recently updated student model as suggested in RT into the combination of Rubin, Yin, Lee, and Gill because both methods perform knowledge distillation of a teacher model into a student model (see Rubin, Abstract; see RT, Abstract). It would have been obvious to one of ordinary skill in the art at the time of the effective filing date to combine the prior art elements of the federated student machine learning structure of RT with the knowledge distillation of Rubin and user terminal of Yin to incorporate providing the user terminal with information about one or more parameters of a recently updated student model to achieve the predictable result of training student models based on a teacher model.
Regarding claim 20, claim 20 contains substantially similar limitations to those found in claim 9 above. Consequently, claim 9 is rejected for the same reasons.
Response to Amendment
Applicant’s amendments, filed 01/23/2026, to the specification are accepted, and the objections to the specification and drawings are respectfully withdrawn.
Applicant’s amendments, filed 01/23/2026, to the claims are accepted, and the objections to the claims are respectfully withdrawn.
Response to Arguments
Applicant's arguments, filed 01/23/2026, traversing the rejection of claims 1-9 and 11-20 under 35 U.S.C. 101 have been fully considered and are not persuasive. Applicant argues that independent claims 1 and 12 are directed to eligible subject matter consistent with the precedential opinion in Ex Parte Desjardins. However, Applicant has not offered any support for this argument beyond the statement itself. Examiner respectfully disagrees.
Thus, the rejection of claims 1-2, 4-9, 11-13, and 15-20 under 35 U.S.C. 101 are respectfully maintained.
Applicant's arguments, filed 01/23/2026, traversing the rejection of claims 1-9 and 11-20 under 35 U.S.C. 103, on pg. 15-17, have been fully considered and are not persuasive. Applicant argues that neither, Rubin, Yin, or Lee do not disclose or suggest “generating attention weights of the teacher model by applying linear interpolation to the attention vector composed of the extracted elements” as recited in claim 1, nor no proper combination of Rubin, Yin, and Lee would disclose or suggest "extracting some elements constituting [a] single unified attention map; [and] ... generating attention weights of [a] teacher model by applying linear interpolation to [an] attention vector composed of the extracted elements,". Examiner respectfully disagrees.
With regard to Rubin, Yin, and Lee not disclosing or suggesting the linear interpolation, as stated above in the 103 section it is through the combination of Rubin, Yin, and Lee that the claim language of "generating attention weights of the teacher model by applying linear interpolation to the attention vector composed of the extracted elements” is rendered obvious and not the references individually suggesting or disclosing the claim language, Applicant is reminded that one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). Similar to what Applicant refers to from the previous interview on 01/05/2026, Rubin discloses the attention vector composed of extracted elements which is used to generate the attention weights of the teacher model, and Lee discloses a method of linear interpolation for feature maps which do not have matching dimensions. However, it is the combination of Rubin, Yin, and Lee which renders obvious the claim language, as Rubin in view of Yin would identify if the extracted attention vector of Rubin matches the dimensionality of the student attention vector, and if not, could apply linear transformations to match the dimensions such that the method of Rubin can proceed; and further incorporating the teachings of Lee would incorporate that the linear transformation could be linear interpolation as used in Lee when the dimensions of feature maps (which are also known as attention maps in the disclosure of Rubin) do not match. Thus, it is through the combination of prior art that would render obvious claim language, and not any of the art individually. The Examiner’s statement during the interview was meant not as a direct mapping of Rubin or Lee disclosing the claim language in its entirety, but that the concept of linear interpolation is found in the disclosure of Lee, and the concept of an attention vector extracted from a plurality of attention maps from Rubin.
With regard to Applicant’s argument that no proper combination of Rubin, Yin, and Lee would disclose or suggest "extracting some elements constituting [a] single unified attention map; [and] ... generating attention weights of [a] teacher model by applying linear interpolation to [an] attention vector composed of the extracted elements”, Applicant has not offered evidence of why the combination of prior art is improper, and the Examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, it is recognized that Rubin, Yin, and Lee are all directed to the same field of endeavor of knowledge distillation in machine learning, and that the disclosures of Yin and Lee provide [s]ome teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention (see MPEP 2143(I)(G)). Specifically, incorporating the teaching of Yin would provide a target student model by performing effective knowledge transfer on a student model based on sample data output by a pre-trained teacher model (see Yin, para. 180), and incorporating the teaching of Lee would achieve significantly improved performance over existing models, with up to a 100 times smaller size of descriptors (see Lee, Abstract).
Thus, for at least the aforementioned reasons, the rejection of claims 1-2, 4-9, 11-13, and 15-20 under 35 U.S.C. 103 is respectfully maintained.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
R. Takashima, S. Li and H. Kawai ("An Investigation of a Knowledge Distillation Method for CTC Acoustic Models," 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada, 2018, pp. 5809-5813, doi: 10.1109/ICASSP.2018.8461995.) discloses a method of knowledge distillation that uses linear interpolation between the student and teacher networks.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAKE BREEN whose telephone number is (571)272-0456. The examiner can normally be reached Monday - Friday, 7:00 AM - 3:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.T.B./Examiner, Art Unit 2143
/JENNIFER N WELCH/Supervisory Patent Examiner, Art Unit 2143
Prosecution Insights