DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
The present application is being examined under the claims filed 06/09/2026.
Claims 1, 3-12, 14-19, and 21-22 are pending.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/09/2026 has been entered.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 07/09/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
This Office Action is in response to Applicant’s communication filed 06/09/2026 in response to office action mailed 03/09/2026. The Applicant’s remarks and any amendments to the claims or specification have been considered with the results that follow.
Response to Arguments
In Remarks page 8, Argument 1
(Examiner summarizes Applicant’s argument) Applicant argues that claim 1 has been amended, obviating the objections.
Examiner’s response to Argument 1
In response, the claim objections have been withdrawn.
In Remarks pages 8-13, Argument 2
(Examiner summarizes Applicant’s arguments) Applicant makes various arguments regarding 35 U.S.C. 101 rejections
Examiner’s response to Argument 2
All rejections under 35 U.S.C. 101 have been withdrawn, rendering Applicant’s arguments moot.
In Remarks page 13-14, Argument 3
(Examiner summarizes Applicant's arguments) Applicant argues that claims 1 and 12 have been amended, arguing on the amended clustering limitations. Applicant argues that dependent claim 5 is patentable for similar reasons.
Examiner’s response to Argument 3
Refreshed search revealed new art to teach on the claims as amended in combination with Vu which addresses the amended clustering (see rejections under 35 U.S.C. 103 below). Accordingly, the 35 U.S.C. 102 rejections have been withdrawn. Examiner notes that it appears that new matter has been added into the claims. See rejection under 35 U.S.C. 112(a) below.
In Remarks page 14-15, Argument 4
(Examiner summarizes Applicant’s argument) Applicant argues that dependent claims 3, 4, 6, 7, 14, and 15 as well as 8-11 and 16 are allowable for the same reasons as their respective independent claims.
Examiner’s response to Argument 4
New rejections have been issued for the independent claims which address Applicant’s arguments.
In Remarks page 15-16, Argument 5
(Examiner summarizes Applicant's argument) Applicant argues that independent claim 17 is allowable for the same reasons as the other independent claims. Applicant further argues that the “sequentially adding a transfer learning model […] until a preset accuracy is satisfied” is not sufficiently taught by Mustafa.
Examiner’s response to Argument 5
The rejection of independent claim 17 has been updated with new art to address applicant’s arguments and amendments.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
A review of the specification shows that the following appears to be the corresponding structure described in the specification for the 35 U.S.C. 112(f) limitations:
a pre-trained model storage unit configured to store a plurality of pre-trained models (claim 1)
[0028] Moreover, each of terms such as "...unit", "...apparatus" and "module" described in specification denotes an element for performing at least one function or operation, and may be implemented in hardware, software or the combination of hardware and software.
[0041] In an embodiment, the pre-trained model storage unit 110 may pre- learn a pre-trained model generated by training multiple deep neural networks with one dataset. For example, the pre-trained model storage unit 110 may learn different types of models, such as ResNet or DenseNet, for the same data set. Alternatively, the pre-trained model storage unit 110may learn a plurality of ResNets each having 10, 20, or 30 layers for the same data.
[0042] As another embodiment, a plurality of pre-trained models may be generated by applying several different datasets of the same or heterogeneous domains to one deep neural network.
[0043] The pre-trained model storage unit 110 may store characteristics of the pre-trained model together. For example, the pre-trained model storage unit 110 may store a size of the pre-trained model (a file size or the number of parameters) or the amount of calculation (FLOPs, Multi-Adds) of the pre-trained model. In addition, the characteristics of the pre-training data may be stored together. In the case of an image-based classification problem, whether it is to classify the type of car or the type of flower may be specified. Alternatively, whether it is to classify a general image such as ImageNet (ILSVRC2012) data in the pre-trained model storage unit 110 may be specified.
a transfer learning data input unit configured to receive transfer learning data (claim 1)
[0028] Moreover, each of terms such as "...unit", "...apparatus" and "module" described in specification denotes an element for performing at least one function or operation, and may be implemented in hardware, software or the combination of hardware and software.
[0045] The transfer learning data input unit 120 receives transfer learning
data and transmits the transfer learning data to the pre-trained model selecting unit 140 and the transfer learning unit 150. The transfer learning data input unit 120 preferably includes a pre-processing function for the transfer learning data. For example, when the sizes of images of the pre-training data and the transfer learning data are different from each other, the transfer learning data input unit 120 may change the size of the image of the transfer learning data to the same image size as that of the pre-training data.
a pre-trained model selecting unit configured to select a pre-trained model (claim 1) wherein the pre-trained model selecting unit derives feature of the transfer learning data (claim 1, claim 4) the pre-trained model selecting unit performs performance evaluation (claim 5) the pre-trained model selecting unit calculates the performance evaluation score by purity (claim 6) the pre-trained model selecting unit calculates the performance evaluation score based on normalized mutual information (NMI) (claim 7)
[0028] Moreover, each of terms such as "...unit", "...apparatus" and "module" described in specification denotes an element for performing at least one function or operation, and may be implemented in hardware, software or the combination of hardware and software.
[0054] As previously described with reference to FIG. 2, the CNN 10 includes the first part 11 including convolutions and pooling operations and the second part 12 including a fully connected layer or a dense layer. Referring to FIG. 3, when transfer learning data is input through the transfer learning data input unit 120 (S210), the pre-trained model selecting unit 140 may regard an output of the first part or the second part of the plurality of stored pre-trained models as a feature of the transfer learning data or derive the feature of the transfer learning data using the output of the first part 11 or the second part 12. At this time, when the pre-trained model selecting unit 140 uses the user requirement, the output of the first or second part of the pre-trained model(s) corresponding to the user requirement is used as a feature of the transfer learning data (S220).
[0055] As described above, the first or second part of the pre-trained model using the deep neural network refines by stages to extract the main information of the input data, so the output of the first or the second part of the pre-trained model may be used as a feature of the transfer learning data.
[0056] In another embodiment, the pre-trained model selecting unit 140 may use the output of an intermediate layer between the input layer and a last layer of the first part as a feature of the transfer learning data, but preferably use an output of the last layer of the first part as a feature of the transfer learning data.
[0057] In another embodiment, the pre-trained model selecting unit 140 may use the output of the intermediate layer between the input layer and the last layer of the second part as a feature of the transfer learning data, but preferably use an output of the last layer of the second part as a feature of the transfer learning data.
[0058] After generating the output of the first part or the second part of the pre-trained model as a feature of the transfer learning data (S220), the pre-trained model selecting unit 140 performs clustering on the transfer learning data using the feature of the transfer learning data (S230). As examples of algorithms that may be used as a clustering module in the pre-trained model selecting unit 140, algorithms such as K-means, Fuzzy K-Means, K-Medoids, and hierarchical clustering, density- based clustering (DBScan), hierarchical density-based clustering (HDBScan), etc. may be used, and the clustering algorithm that may be used by the pre-trained model selecting unit 140 of the transfer learning system 100 for a deep neural network according to an embodiment of the present invention is not limited to the aforementioned clustering algorithm.
a transfer learning unit configured to generate one or more transfer learning models (claim 1)
[0028] Moreover, each of terms such as "...unit", "...apparatus" and "module" described in specification denotes an element for performing at least one function or operation, and may be implemented in hardware, software or the combination of hardware and software.
[0089] The transfer learning unit 150 generates a plurality of transfer learning models by performing transfer learning using the pre-trained model selected by the pre-trained model selecting unit 140 and the transfer learning data input from the transfer learning data input unit 120. The transfer learning unit 150 generates P transfer learning models by performing transfer [earning using the P selected pre- trained models and the transfer learning data. The transfer learning of the transfer learning unit 150 uses a known transfer learning method and is not limited to a specific learning method.
a user requirement input unit for inputting a user requirement (claim 3)
[0046] The user requirement input unit 130 receives a user requirement. In an embodiment, the user requirement input unit 130may input a user requirement received through an input device such as a keyboard, mouse, or touch screen provided in a computer or mobile terminal used by the user.
[0047] Here, the user requirement may be a requirement for the pre-trained model, such as the size of the pre-trained model (the file size or the number of parameters) or the amount of calculation (FLOPs, Multi-Adds) of the pre-trained model.For example, when the user prefers a small deep neural networkmodel, the user requirement input unit 130 may input an upper limit of the number of parameters of the pre-trained model. Alternatively, the user requirement input unit 130may input an upper limit of the amount of calculation (FLOPs, Multi-Adds) of the pre-trained model.
transfer learning model output unit configured to calculate classification performance accuracy (claim 8) the transfer learning model output unit generates and outputs a final transfer learning model (claim 9) wherein the transfer learning model output unit first selects a first transfer learning model having the highest classification performance accuracy, and selects a second transfer learning model having the greatest accuracy improvement (claim 10) wherein the transfer learning model output unit configures the ensemble (claim 11)
[0028] Moreover, each of terms such as "...unit", "...apparatus" and "module" described in specification denotes an element for performing at least one function or operation, and may be implemented in hardware, software or the combination of hardware and software.
[0090] In an embodiment, it is preferable to use some of the transfer learning data as validation data. The transfer learning unit 150 evaluates the classification perfornance of the P transfer learning models by using the verification data. Classification performance evaluation may be calculated by Top-1 accuracy or area under the receiver operating characteristic curve (ROC-AUC) score.
[0091] The transfer learning model output unit 160may calculate classification performance accuracy for the plurality of transfer learning models generated by the transfer learning unit 150, and select and output one or more transfer learning models in the order, starting from a transfer learning mode having the highest classification performance accuracy, among the plurality of transfer learning models generated by the transfer learning unit 150. In an embodiment, the transfer learning model output unit 160may select T (TSP) transfer learning models from among the P transfer learning models generated by the transfer learning unit 150. In this case, the transfer learning models may be selected based on classification perfornance calculated by the transfer learning unit 150 as a selection criterion.
[0092] When two or more transfer learning models are selected, the transfer learning model output unit 160may configure a final transfer learning model in an ensemble form. In an embodiment, the transfer learning model output unit 160 may generate and output a final transfer learning model y(x) by configuring an ensemble by Equation 13 below with respect to one or more selected transfer learning models
a pre-trained model storing step of storing a plurality of pre-trained models (claim 12)
[0028] Moreover, each of terms such as "...unit", "...apparatus" and "module" described in specification denotes an element for performing at least one function or operation, and may be implemented in hardware, software or the combination of hardware and software.
[0041] In an embodiment, the pre-trained model storage unit 110 may pre- learn a pre-trained model generated by training multiple deep neural networks with one dataset. For example, the pre-trained model storage unit 110 may learn different types of models, such as ResNet or DenseNet, for the same data set. Alternatively, the pre-trained model storage unit 110may learn a plurality of ResNets each having 10, 20, or 30 layers for the same data.
[0042] As another embodiment, a plurality of pre-trained models may be generated by applying several different datasets of the same or heterogeneous domains to one deep neural network.
[0043] The pre-trained model storage unit 110 may store characteristics of the pre-trained model together. For example, the pre-trained model storage unit 110 may store a size of the pre-trained model (a file size or the number of parameters) or the amount of calculation (FLOPs, Multi-Adds) of the pre-trained model. In addition, the characteristics of the pre-training data may be stored together. In the case of an image-based classification problem, whether it is to classify the type of car or the type of flower may be specified. Alternatively, whether it is to classify a general image such as ImageNet (ILSVRC2012) data in the pre-trained model storage unit 110 may be specified.
transfer learning data input step of inputting transfer learning data (claim 12)
[0045] The transfer learning data input unit 120 receives transfer learning data and transmits the transfer learning data to the pre-trained model selecting unit 140 and the transfer learning unit 150. The transfer learning data input unit 120 preferably includes a pre-processing function for the transfer learning data. For example, when the sizes of images of the pre-training data and the transfer learning data are different from each other, the transfer learning data input unit 120 may change the size of the image of the transfer learning data to the same image size as that of the pre-training data.
a pre-trained model selecting step of selecting a pre-trained model (claim 12, 17) wherein the pre-trained model selecting step includes: generating an output […] and selecting a pre-trained model (claim 12, 15) wherein the pre-trained model selecting step includes selecting a pre-trained model (claim 14)
[0054] As previously described with reference to FIG. 2, the CNN 10 includes the first part 11 including convolutions and pooling operations and the second part 12 including a fully connected layer or a dense layer. Referring to FIG. 3, when transfer learning data is input through the transfer learning data input unit 120 (S210), the pre-trained model selecting unit 140 may regard an output of the first part or the second part of the plurality of stored pre-trained models as a feature of the transfer learning data or derive the feature of the transfer learning data using the output of the first part 11 or the second part 12. At this time, when the pre-trained model selecting unit 140 uses the user requirement, the output of the first or second part of the pre-trained model(s) corresponding to the user requirement is used as a feature of the transfer learning data (S220).
[0055] As described above, the first or second part of the pre-trained model using the deep neural network refines by stages to extract the main information of the input data, so the output of the first or the second part of the pre-trained model may be used as a feature of the transfer learning data.
[0056] In another embodiment, the pre-trained model selecting unit 140 may use the output of an intermediate layer between the input layer and a last layer of the first part as a feature of the transfer learning data, but preferably use an output of the last layer of the first part as a feature of the transfer learning data.
[0057] In another embodiment, the pre-trained model selecting unit 140 may use the output of the intermediate layer between the input layer and the last layer of the second part as a feature of the transfer learning data, but preferably use an output of the last layer of the second part as a feature of the transfer learning data.
[0058] After generating the output of the first part or the second part of the pre-trained model as a feature of the transfer learning data (S220), the pre-trained model selecting unit 140 performs clustering on the transfer learning data using the feature of the transfer learning data (S230). As examples of algorithms that may be used as a clustering module in the pre-trained model selecting unit 140, algorithms such as K-means, Fuzzy K-Means, K-Medoids, and hierarchical clustering, density- based clustering (DBScan), hierarchical density-based clustering (HDBScan), etc. may be used, and the clustering algorithm that may be used by the pre-trained model selecting unit 140 of the transfer learning system 100 for a deep neural network according to an embodiment of the present invention is not limited to the aforementioned clustering algorithm.
a transfer learning step of generating a plurality of transfer learning models (claim 12, 17)
[0089] The transfer learning unit 150 generates a plurality of transfer learning models by performing transfer learning using the pre-trained model selected by the pre-trained model selecting unit 140 and the transfer learning data input from the transfer learning data input unit 120. The transfer learning unit 150 generates P transfer learning models by performing transfer [earning using the P selected pre- trained models and the transfer learning data. The transfer learning of the transfer learning unit 150 uses a known transfer learning method and is not limited to a specific learning method.
a user requirement input step of inputting a user requirement (claim 14)
[0046] The user requirement input unit 130 receives a user requirement. In an embodiment, the user requirement input unit 130may input a user requirement received through an input device such as a keyboard, mouse, or touch screen provided in a computer or mobile terminal used by the user.
[0047] Here, the user requirement may be a requirement for the pre-trained model, such as the size of the pre-trained model (the file size or the number of parameters) or the amount of calculation (FLOPs, Multi-Adds) of the pre-trained model.For example, when the user prefers a small deep neural networkmodel, the user requirement input unit 130 may input an upper limit of the number of parameters of the pre-trained model. Alternatively, the user requirement input unit 130may input an upper limit of the amount of calculation (FLOPs, Multi-Adds) of the pre-trained model.
a transfer learning model output step of calculating classification performance accuracy […] and selecting and outputting one or more transfer learning models (claim 16)
[0090] In an embodiment, it is preferable to use some of the transfer learning data as validation data. The transfer learning unit 150 evaluates the classification perfornance of the P transfer learning models by using the verification data. Classification performance evaluation may be calculated by Top-1 accuracy or area under the receiver operating characteristic curve (ROC-AUC) score.
[0091] The transfer learning model output unit 160may calculate classification performance accuracy for the plurality of transfer learning models generated by the transfer learning unit 150, and select and output one or more transfer learning models in the order, starting from a transfer learning mode having the highest classification performance accuracy, among the plurality of transfer learning models generated by the transfer learning unit 150. In an embodiment, the transfer learning model output unit 160may select T (TSP) transfer learning models from among the P transfer learning models generated by the transfer learning unit 150. In this case, the transfer learning models may be selected based on classification perfornance calculated by the transfer learning unit 150 as a selection criterion.
[0092] When two or more transfer learning models are selected, the transfer learning model output unit 160may configure a final transfer learning model in an ensemble form. In an embodiment, the transfer learning model output unit 160 may generate and output a final transfer learning model y(x) by configuring an ensemble by Equation 13 below with respect to one or more selected transfer learning models
a step of configuring an ensemble by selecting at least some of the plurality of transfer learning models (claim 17) the step of configuring the ensemble includes: first selecting a first transfer learning model having a highest classification performance accuracy, and selecting a second transfer learning model having the greatest accuracy improvement (claim 18) the step of configuring the ensemble includes configuring the ensemble by adding a transfer learning mode (claim 19) wherein the step of configuring the ensemble includes configuring an ensemble by adding a transfer learning model having the highest accuracy […] and a transfer learning model that contributes the most to improvement of accuracy (claim 12)
[0092] When two or more transfer learning models are selected, the transfer learning model output unit 160may configure a final transfer learning model in an ensemble form. In an embodiment, the transfer learning model output unit 160 may generate and output a final transfer learning model y(x) by configuring an ensemble by Equation 13 below with respect to one or more selected transfer learning models.
[0096] In another embodiment, the transfer learning model output unit 160
may sequentially configure an ensemble using the following method. In an embodiment, the transfer learning model output unit 160 may first select a first transfer learning model having the highest classification performance accuracy and select a second transfer learning model having the greatest accuracy improvement when configuring an ensemble with the first transfer learning model, to configure an ensemble.
Claim Rejections - 35 USC § 112(a)
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 1, 3-12, 14-19, and 21-22 are rejected under 35 U.S.C. 112(a) for failing to provide a written description of the claimed invention.
Regarding Claims 1, 12, and 17
Claim 1 recites the limitation:
performing clustering of the plurality of data items by assigning each of the plurality of data items to one of a plurality of clusters using the derived features
Claims 12 and 17 recite similar limitations and are rejected for the same reasons. The specification does not support this limitation. Applicant cites paragraphs 54-58, 61, and 63 in relation to this limitation. The specification provides:
(paragraph [0054]) “when transfer learning data is input through the transfer learning data input unit the pre-trained model selecting unit 140 may regard an output of the first part or the second part of the plurality of stored pre-trained models as a feature of the transfer learning data”
(paragraph [0058]) “After generating the output of the first part or the second part of the pre-trained model as a feature of the transfer learning data (S220), the pre-trained model selecting unit 140 performs clustering on the transfer learning data using the feature of the transfer learning data (S230).”
(paragraph [0063]) “Therefore, it is assumed that clustering is performed using the feature of transfer learning data by setting the number of clusters to 3 as well.”
However, the disclosure does not support “assigning each of the plurality of data items to one of a plurality of clusters using the derived features”.
Regarding Dependent Claims
Claims 3-11 and 21 are dependent upon claim 1, 14-16 and 22 are dependent upon claim 12, and 18-19 are dependent upon claim 17, and are therefore similarly rejected for including the deficiencies of claims 1, 12, and 17 respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 12, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over NPL reference Vu et al. “Exploring and Predicting Transferability across NLP Tasks” herein referred to as Vu in view of NPL reference Meiseles et al. “Source Model Selection for Deep Learning in the Time Series Domain” herein referred to as Meiseles.
Regarding Claim 1
Vu teaches:
A transfer learning system for a deep neural network, the transfer learning system comprising: a pre-trained model storage unit configured to store a plurality of pre-trained models that are deep neural network models learned using one or more pre-training datasets;
(page 2 column 1 paragraph 3) “We publicly release our task library, which consists of pretrained models[*Examiner notes: pretrained model storage unit] and task embeddings for the 33 NLP tasks we study[*Examiner notes: one or more pre-training datasets], along with a codebase that computes task embeddings for new tasks and identifies source tasks that will likely yield positive transferability”; (page 3 column 1 paragraph 2) “Table 1 lists the 33 datasets[*Examiner notes: pre-training datasets] in our study.”
a transfer learning data input unit configured to receive transfer learning data;
(page 3 column 1 paragraph 2) “Table 1 lists the 33 datasets in our study.7 We select these datasets by mostly following prior work: nine of the eleven CR tasks come from the GLUE benchmark (Wang et al., 2019b); all eleven QA tasks are from the MultiQA repository (Talmor and Berant, 2019b); and all eleven SL tasks were used by Liu et al. (2019a). We consider all possible pairs of source and target datasets[*Examiner notes: target dataset is the received transfer learning data];”
a pre-trained model selecting unit configured to select a pre-trained model corresponding to the transfer learning data from among the plurality of stored pre-trained models
(page 1 column 2 figure 1 caption) “Given a target task, we first compute its task embedding and then identify the most similar source task embedding (in this example, WikiHop) from a precomputed library via cosine similarity[*Examiner notes: select a pre-trained model].”; (page 7 column 2 last paragraph) “TASKEMB can substantially boost the quality of the rankings, frequently outperforming the other methods across different classes of problems, data regimes, and transfer scenarios. These results demonstrate that the task similarity between the computed embeddings is a robust predictor of effective transfer.”
and a transfer learning unit configured to generate one or more transfer learning models by performing transfer learning using the selected pre-trained model and the transfer learning data
(page 2 column 1 last paragraph) “In each experiment, we follow the STILTs pipeline of Phang et al. (2018) by taking a pretrained BERT model,5 fine-tuning it on an intermediate source task, and then fine-tuning the resulting model on a target task[*Examiner notes: performing transfer learning using selected model and the transfer learning data].”; Figure 1
PNG
media_image1.png
501
943
media_image1.png
Greyscale
wherein the pre-trained model selecting unit derives, for each of a plurality of data items of the transfer learning data, a feature of the data item from an output of a first part or a second part of each of the plurality of stored pre-trained models
(page 6 column 1 last paragraph) “To begin, we fine-tune BERT on the training dataset of a given task[*Examiner notes: plurality of stored pre-trained models]; the model without the final task-specific layer forms our feature extractor[*Examiner notes: second part]. Next, we feed the entire training dataset into the model[*Examiner notes: each data item of transfer learning data] and compute the task embedding based on the Fisher of the feature extractor’s parameters (weights) θ, i.e., the expected covariance of the gradients of the log-likelihood with respect to θ:”
Vu does not explicitly teach
Performs clustering of the plurality of data items by assigning each of the plurality of data items to one of a plurality of clusters using the derived features and selects a pre-trained model based on a result of the clustering
However, Meiseles teaches:
Performs clustering of the plurality of data items by assigning each of the plurality of data items to one of a plurality of clusters using the derived features
(page 6194 column 1 section B) “a) Encode the target training data using each of the source models truncated after the global average pooling layer. This is visually represented by the dashed rectangle in figure 1. b) Calculate the Mean Silhouette Coefficient (MSC) for each set of encodings using the cosine distance metric.”; (page 6194 column 1 last paragraph) “We suggest that if this is the case the encodings representing sequences belonging to the same target class will be close to each other and far from encodings representing sequences from the other classes, before fine-tuning the source model. In this sense we expect the encodings[*Examiner notes: using derived features] from the same class[*Examiner notes: one of a plurality of clusters] to be clustered together[*Examiner notes: clustering of data items].”
and selects a pre-trained model based on a result of the clustering
(page 6194 column 1 last paragraph) “We, therefore, calculate the MSC of all possible source models[*Examiner notes: based on a result of the clustering] to encode the target data and sort the source models by their MSC in decreasing order. We then use this list to select which source domain to transfer from.”
Vu, Meiseles, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transfer learning of Vu by selecting the model using the clustering technique as taught by Meiseles because (Meiseles page 6199 column 1 last paragraph) “We proved that when the MSC is 1, perfect classification can be achieved using the final layer of the target network alone. Our method achieves an MRR of 0.115 and top-1 accuracy of 7.1%, which is statistically significant compared to random model selection. The top-1 accuracy and MRR of our method are comparable to the state-of-the-art [6], despite not having access to the source training data.”
Regarding Claim 5
Vu in view of Meiseles teaches:
The transfer learning system of claim 1
(see rejection of claim 1)
Vu further teaches:
wherein the pre-trained model selecting unit performs performance evaluation on a result of performing clustering and selects the pre-trained model in the order of the highest performance evaluation score.
[*Examiner notes: Figure 2 below shows the clustering of transfer learning feature data]; (page 6 column 2 paragraph 3) “Our evaluation centers around the meta-task of selecting the best source task for a given target task[*Examiner notes: selects a pre-trained model]. Specifically, given a target task, we rank all the other source tasks in our library in descending order by the cosine similarity13[*Examiner notes: order of highest performance] between their task embeddings and the target task’s embedding[*Examiner notes: based on clustering]”; Figure 2
PNG
media_image2.png
414
384
media_image2.png
Greyscale
Regarding Claim 12
Vu teaches:
A transfer learning method for a deep neural network, the transfer learning method comprising: a pre-trained model storing step of storing a plurality of pre-trained models that are deep neural network models learned using a plurality of pre-training datasets
(page 2 column 1 paragraph 3) “We publicly release our task library, which consists of pretrained models[*Examiner notes: pretrained model storage unit] and task embeddings for the 33 NLP tasks we study[*Examiner notes: one or more pre-training datasets], along with a codebase that computes task embeddings for new tasks and identifies source tasks that will likely yield positive transferability”; (page 3 column 1 paragraph 2) “Table 1 lists the 33 datasets[*Examiner notes: pre-training datasets] in our study.”
a transfer learning data input step of inputting transfer learning data;
(page 3 column 1 paragraph 2) “Table 1 lists the 33 datasets in our study.7 We select these datasets by mostly following prior work: nine of the eleven CR tasks come from the GLUE benchmark (Wang et al., 2019b); all eleven QA tasks are from the MultiQA repository (Talmor and Berant, 2019b); and all eleven SL tasks were used by Liu et al. (2019a). We consider all possible pairs of source and target datasets[*Examiner notes: target dataset is the received transfer learning data];”
a pre-trained model selecting step of selecting a pre-trained model corresponding to the input transfer learning data from among the plurality of stored pre-trained models;
(page 1 column 2 figure 1 caption) “Given a target task, we first compute its task embedding and then identify the most similar source task embedding (in this example, WikiHop) from a precomputed library via cosine similarity[*Examiner notes: select a pre-trained model].”; (page 7 column 2 last paragraph) “TASKEMB can substantially boost the quality of the rankings, frequently outperforming the other methods across different classes of problems, data regimes, and transfer scenarios. These results demonstrate that the task similarity between the computed embeddings is a robust predictor of effective transfer.”
and a transfer learning step of generating a plurality of transfer learning models by performing transfer learning using the selected pre-trained model and the transfer learning data
(page 2 column 1 last paragraph) “In each experiment, we follow the STILTs pipeline of Phang et al. (2018) by taking a pretrained BERT model,5 fine-tuning it on an intermediate source task, and then fine-tuning the resulting model on a target task[*Examiner notes: performing transfer learning using selected model and the transfer learning data].”; Figure 1
PNG
media_image1.png
501
943
media_image1.png
Greyscale
wherein the pre-trained model selecting step includes: for each of a plurality of data items of the transfer learning data, generating an output of a first part or a second part of each of the plurality of stored pre-trained models as a feature of the data item
(page 6 column 1 last paragraph) “To begin, we fine-tune BERT on the training dataset of a given task[*Examiner notes: plurality of stored pre-trained models]; the model without the final task-specific layer forms our feature extractor[*Examiner notes: second part]. Next, we feed the entire training dataset into the model Examiner notes: each data item of transfer learning data] and compute the task embedding[*Examiner notes: output as a feature of the transfer learning data] based on the Fisher of the feature extractor’s parameters (weights) θ, i.e., the expected covariance of the gradients of the log-likelihood with respect to θ:”
Vu does not explicitly teach:
and selecting a pre-trained model by performing clustering of the plurality of data items by assigning each of the plurality of data items to one of a plurality of clusters using the features generated for the plurality of data items.
However, Meiseles teaches:
and selecting a pre-trained model by performing clustering of the plurality of data items by assigning each of the plurality of data items to one of a plurality of clusters using the features generated for the plurality of data items.
(page 6194 column 1 section B) “a) Encode the target training data using each of the source models truncated after the global average pooling layer. This is visually represented by the dashed rectangle in figure 1. b) Calculate the Mean Silhouette Coefficient (MSC) for each set of encodings using the cosine distance metric.”; (page 6194 column 1 last paragraph) “We suggest that if this is the case the encodings representing sequences belonging to the same target class will be close to each other and far from encodings representing sequences from the other classes, before fine-tuning the source model. In this sense we expect the encodings[*Examiner notes: using derived features] from the same class[*Examiner notes: one of a plurality of clusters] to be clustered together[*Examiner notes: clustering of data items].”; (page 6194 column 1 last paragraph) “We, therefore, calculate the MSC of all possible source models[*Examiner notes: based on a result of the clustering] to encode the target data and sort the source models by their MSC in decreasing order. We then use this list to select which source domain to transfer from.”
Vu, Meiseles, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transfer learning of Vu by selecting the model using the clustering technique as taught by Meiseles because (Meiseles page 6199 column 1 last paragraph) “We proved that when the MSC is 1, perfect classification can be achieved using the final layer of the target network alone. Our method achieves an MRR of 0.115 and top-1 accuracy of 7.1%, which is statistically significant compared to random model selection. The top-1 accuracy and MRR of our method are comparable to the state-of-the-art [6], despite not having access to the source training data.”
Regarding Claim 21
Vu in view of Meiseles teaches:
The transfer learning system of claim 1
(see rejection of claim 1)
Meiseles further teaches:
wherein the first part of each of the plurality of stored pre-trained models comprises one or more layers configured to perform at least one of a convolution operation or a pooling operation, and the second part of each of the plurality of stored pre-trained models comprises a fully connected layer.
(page 6194 column 1 3) “Fully connected output layer- The output layer con tains one unit for each of the classes in the dataset. A softmax activation is used in order to transform the raw outputs into a probability distribution.”; (page 6194 figure 1)
PNG
media_image3.png
345
519
media_image3.png
Greyscale
Regarding Claim 22
Vu in view of Meiseles teaches:
The transfer learning method of claim 12
(see rejection of claim 12)
Meiseles further teaches:
wherein the first part of each of the plurality of stored pre-trained models comprises one or more layers configured to perform at least one of a convolution operation or a pooling operation, and the second part of each of the plurality of stored pre-trained models comprises a fully connected layer.
(page 6194 column 1 3) “Fully connected output layer- The output layer con tains one unit for each of the classes in the dataset. A softmax activation is used in order to transform the raw outputs into a probability distribution.”; (page 6194 figure 1)
PNG
media_image3.png
345
519
media_image3.png
Greyscale
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Vu with Meiseles for the same reasons given in claim 12 above.
Claims 3-4 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Vu in view of Meiseles, and further in view of NPL reference Lu et al. “Neural Architecture Transfer” herein referred to as Lu.
Regarding Claim 3
Vu in view of Meiseles teaches:
The transfer learning system of claim
(see rejection of claim 1)
Vu in view of Meiseles does not explicitly teach:
further comprising: a user requirement input unit for inputting a user requirement, wherein the pre-trained model selecting unit selects a pre-trained model corresponding to the input user requirement or the input transfer learning data.
However, Lu teaches:
further comprising: a user requirement input unit for inputting a user requirement, wherein the pre-trained model selecting unit selects a pre-trained model corresponding to the input user requirement or the input transfer learning data.
(page 3 column 2 paragraph 2) “NAT starts with an archive A of architectures (subnets) created by uniform sampling from our search space. We evaluate the performance fi of each subnet (ai) using weights inherited from the supernet. The accuracy predictor is then constructed from (ai; fi) pairs which (jointly with any additional objectives provided by the user[*Examiner notes: user requirement]) drives the subsequent many-objective evolutionary search towards optimal architectures[*Examiner notes: selecting a model].”
Vu, Meiseles, Lu, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transfer learning of Vu in view of Meiseles with the user requirement of Lu because (Lu page 2 column 1 last paragraph) “Finally we demonstrate the scalability and utility of NAT across many objectives. Optimizing accuracy, model size and one of MAdds, CPU or GPU latency, NATNets dominates MobileNetv3 [20] across all objectives. We also consider a 12 objective problem of finding a common architecture across eleven datasets”
Regarding Claim 4
Vu in view of Meiseles and Lu teaches:
The transfer learning system of claim 3
(see rejection of claim 3)
Vu further teaches:
wherein the pre-trained model selecting unit derives feature of the transfer learning data based on an output of a first part or a second part of the pre-trained model corresponding to the user requirement
(page 6 column 1 last paragraph) “To begin, we fine-tune BERT on the training dataset of a given task; the model without the final task-specific layer forms our feature extractor[*Examiner notes: second part]. Next, we feed the entire training dataset into the model[*Examiner notes: transfer learning data] and compute the task embedding[*Examiner notes: feature of the transfer learning data] based on the Fisher of the feature extractor’s parameters (weights) θ, i.e., the expected covariance of the gradients of the log-likelihood with respect to θ:”
and selects a pre-trained model based on clustering for the transfer learning data using the feature of the transfer learning data
[*Examiner notes: Figure 2 below shows the clustering of transfer learning feature data]; (page 6 column 2 paragraph 3) “Our evaluation centers around the meta-task of selecting the best source task for a given target task[*Examiner notes: selects a pre-trained model]. Specifically, given a target task, we rank all the other source tasks in our library in descending order by the cosine similarity13 between their task embeddings and the target task’s embedding[*Examiner notes: based on clustering]”; Figure 2
PNG
media_image2.png
414
384
media_image2.png
Greyscale
Regarding Claim 14
Vu in view of Meiseles teaches:
The transfer learning method of claim 12
(see rejection of claim 12)
Vu in view of Meiseles does not explicitly teach:
further comprising: a user requirement input step of inputting a user requirement, wherein the pre-trained model selecting step includes selecting a pre-trained model corresponding to the input user requirement or the input transfer learning data.
However, Lu teaches:
further comprising: a user requirement input step of inputting a user requirement, wherein the pre-trained model selecting step includes selecting a pre-trained model corresponding to the input user requirement or the input transfer learning data.
(page 3 column 2 paragraph 2) “NAT starts with an archive A of architectures (subnets) created by uniform sampling from our search space. We evaluate the performance fi of each subnet (ai) using weights inherited from the supernet. The accuracy predictor is then constructed from (ai; fi) pairs which (jointly with any additional objectives provided by the user[*Examiner notes: user requirement]) drives the subsequent many-objective evolutionary search towards optimal architectures[*Examiner notes: selecting a model].”
Vu, Meiseles, Lu, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transfer learning of Vu in view of Meiseles with the user requirement of Lu because (Lu page 2 column 1 last paragraph) “Finally we demonstrate the scalability and utility of NAT across many objectives. Optimizing accuracy, model size and one of MAdds, CPU or GPU latency, NATNets dominates MobileNetv3 [20] across all objectives. We also consider a 12 objective problem of finding a common architecture across eleven datasets”
Regarding Claim 15
Vu in view of Meiseles and Lu teaches:
The transfer learning method of claim 14
(see rejection of claim 14)
Vu further teaches:
wherein the pre-trained model selecting step includes: generating an output of a first part or a second part of the pre-trained model […] as a feature of the transfer learning data;
(page 6 column 1 last paragraph) “To begin, we fine-tune BERT on the training dataset of a given task[*Examiner notes: plurality of stored pre-trained models]; the model without the final task-specific layer forms our feature extractor[*Examiner notes: second part]. Next, we feed the entire training dataset into the model[*Examiner notes: transfer learning data] and compute the task embedding[*Examiner notes: feature of the transfer learning data] based on the Fisher of the feature extractor’s parameters (weights) θ, i.e., the expected covariance of the gradients of the log-likelihood with respect to θ:”
and selecting a pre-trained model by performing clustering on the transfer learning data by using the feature of the transfer learning data.
[*Examiner notes: Figure 2 below shows the clustering of transfer learning feature data]; (page 6 column 2 paragraph 3) “Our evaluation centers around the meta-task of selecting the best source task for a given target task[*Examiner notes: selects a pre-trained model]. Specifically, given a target task, we rank all the other source tasks in our library in descending order by the cosine similarity13 between their task embeddings and the target task’s embedding[*Examiner notes: based on clustering]”; Figure 2
PNG
media_image2.png
414
384
media_image2.png
Greyscale
And Lu further teaches:
corresponding to the user requirement
(page 3 column 2 paragraph 2) “NAT starts with an archive A of architectures (subnets) created by uniform sampling from our search space. We evaluate the performance fi of each subnet (ai) using weights inherited from the supernet. The accuracy predictor is then constructed from (ai; fi) pairs which (jointly with any additional objectives provided by the user[*Examiner notes: user requirement]) drives the subsequent many-objective evolutionary search towards optimal architectures[*Examiner notes: selecting a model].”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Vu in view of Meiseles with Lu for the same reasons given in claim 14 above.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Vu in view of Meiseles and further in view of NPL reference Rama Rao et al. “EXPLORING THE IMPACT OF OPTIMAL CLUSTERS ON CLUSTER PURITY” herein referred to as Rama Rao.
Regarding Claim 6
Vu in view of Meiseles teaches:
The transfer learning system of claim 5
(see rejection of claim 5)
Vu in view of Meiseles does not teach:
wherein the pre-trained model selecting unit calculates the performance evaluation score by purity derived through the following equation for the result of performing the clustering: [Equation]
PNG
media_image4.png
91
304
media_image4.png
Greyscale
wherein N is a total number of data, Nkj is the number of j-th classes in a k-th cluster
However Rama Rao teaches
wherein the pre-trained model selecting unit calculates the performance evaluation score by purity derived through the following equation for the result of performing the clustering: [Equation]
PNG
media_image4.png
91
304
media_image4.png
Greyscale
wherein N is a total number of data, Nkj is the number of j-th classes in a k-th cluster(page 756 column 1 section D paragraph 1) “Purity and entropy are the similar functions which used as an exterior evaluation criteria to know the cluster quality, Let x be an object which predicts and gives the cluster memberships for each sample and let y be a factor that gives the true class labels of each sample and suppose if we have given l categories as an input and if the clustering method generates k clusters, then the purity of that given clusters with respect to the known categories is given as […equation…] Where n is the total no. of samples and njq is the number of samples in the cluster q, which belongs to the class j(1≤j≤l). Purity value lies between [0], [1]. The large the purity the better the clustering performance.”
PNG
media_image5.png
150
344
media_image5.png
Greyscale
Vu, Rama Rao, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transfer learning of Vu using the clustering purity of Rama Rao because (Rama Rao page 756 column 1 above section E) “Purity value lies between [0], [1]. The large the purity the better the clustering performance” and (Rama Rao page 757 column 1 last paragraph) “Purity and Silhouette index have served as better indicators for evaluating cluster formation of this dataset.”
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Vu in view of Meiseles, and further in view of NPL reference Newman et al. “Improved mutual information measure for clustering, classification, and community detection” herein referred to as Newman.
Regarding Claim 7
Vu in view of Meiseles teaches:
The transfer learning system of claim 5
(see rejection of claim 5)
Vu in view of Meiseles does not teach:
wherein the pre-trained model selecting unit calculates the performance evaluation score based on normalized mutual information (NMI) derived through the following equation for the result of performing the clustering: [Equation]
PNG
media_image6.png
93
342
media_image6.png
Greyscale
wherein Y is a random variable for a class label of the transfer learning data, K is a random variable for a cluster label, HO is entropy, and I(Y;K) is mutual information between Y and K
However Newman teaches:
wherein the pre-trained model selecting unit calculates the performance evaluation score based on normalized mutual information (NMI) derived through the following equation for the result of performing the clustering: [Equation]
PNG
media_image6.png
93
342
media_image6.png
Greyscale
wherein Y is a random variable for a class label of the transfer learning data, K is a random variable for a cluster label, HO is entropy, and I(Y;K) is mutual information between Y and K
(page 3 column 1 above equation 8) “In addition to increasing with the similarity of the labelings, the mutual information has the nice property of being symmetric in r and s. In some circumstances, it is convenient to normalize it so as to create a measure whose value runs between zero and one [3]. There are several ways to perform the normalization [13], but the most widely used normalizes by the mean of the entropies H(r) and H(s), which preserves the symmetry with respect to r and s:”
PNG
media_image7.png
59
411
media_image7.png
Greyscale
Vu, Newman, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transfer learning of Vu with the normalized mutual information of Newman because (Newman page 3 column 1 above equation 8) “In addition to increasing with the similarity of the labelings, the mutual information has the nice property of being symmetric in r and s. In some circumstances, it is convenient to normalize it so as to create a measure whose value runs between zero and one [3].”
Claims 8-11 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Vu in view of Meiseles, and further in view of NPL reference Mustafa et al. “Deep Ensembles for Low-Data Transfer Learning” herein referred to as Mustafa.
Regarding Claim 8
Vu in view of Meiseles teaches:
The transfer learning system of claim 1
(see rejection of claim 1)
Vu in view of Meiseles does not explicitly teach
further comprising: a transfer learning model output unit configured to calculate classification performance accuracy for the one or more generated transfer learning models and select and output one or more transfer learning models in the order of the highest classification performance accuracy.
However, Mustafa teaches:
further comprising: a transfer learning model output unit configured to calculate classification performance accuracy for the one or more generated transfer learning models and select and output one or more transfer learning models in the order of the highest classification performance accuracy.
(page 3 second to last paragraph) “Pre-trained model selection. Fine-tuning all experts on the new task would be prohibitively expensive. Following Puigcerver et al. (2020), we rank all the models by their kNN leave-one-out accuracy as a proxy for final fine-tuned accuracy, instead keeping the K best models (rather than 1).”
Vu, Meiseles, Mustafa, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transfer learning of Vu in view of Meiseles with the transfer learning ensemble taught by Mustafa because (Mustafa page 1 abstract) “When evaluated together with strong baselines on 19 different downstream tasks (the Visual Task Adaptation Benchmark), this achieves state-of-the-art performance at a much lower inference budget, even when selecting from over 2,000 pre-trained models. We also assess our ensembles on ImageNet variants and show improved robustness to distribution shift.”
Regarding Claim 9
Vu in view of Meiseles teaches:
The transfer learning system of claim 1
(see rejection of claim 1)
Vu in view of Meiseles does not teach:
wherein the transfer learning model output unit generates and outputs a final transfer learning model y(x) by configuring an ensemble by the following equation for the one or more selected transfer learning models:[Equation]
PNG
media_image8.png
97
226
media_image8.png
Greyscale
wherein y(x) is a final transfer learning model, yt(x) is any transfer learning model among T, and wt is an ensemble weight.
However, Mustafa teaches:
wherein the transfer learning model output unit generates and outputs a final transfer learning model y(x) by configuring an ensemble by the following equation for the one or more selected transfer learning models:[Equation]
PNG
media_image8.png
97
226
media_image8.png
Greyscale
wherein y(x) is a final transfer learning model, yt(x) is any transfer learning model among T, and wt is an ensemble weight.
(page 20 paragraph 1) “The extra benefit (in our view, which is not discussed in that work) is that by weighting ensembles according to the number of times they were picked, we can use a weighted average, downweighting less performant models and potentially improving performance.”; [*Examiner notes: Mustafa exactly describes the equation claimed in words]
Vu, Meiseles, Mustafa, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transfer learning of Vu in view of Meiseles with the transfer learning ensemble taught by Mustafa because (Mustafa page 1 abstract) “When evaluated together with strong baselines on 19 different downstream tasks (the Visual Task Adaptation Benchmark), this achieves state-of-the-art performance at a much lower inference budget, even when selecting from over 2,000 pre-trained models. We also assess our ensembles on ImageNet variants and show improved robustness to distribution shift” and (Mustafa page 20 paragraph 1) “The extra benefit (in our view, which is not discussed in that work) is that by weighting ensembles according to the number of times they were picked, we can use a weighted average, downweighting less performant models and potentially improving performance.”
Regarding Claim 10
Vu in view of Meiseles and Mustafa teaches:
The transfer learning system of claim 9
(see rejection of claim 9)
Mustafa further teaches:
wherein the transfer learning model output unit first selects a first transfer learning model having the highest classification performance accuracy,
(page 3 second to last paragraph) “Fine-tuning all experts on the new task would be prohibitively expensive. Following Puigcerver et al. (2020), we rank all the models by their kNN leave-one-out accuracy as a proxy for final fine-tuned accuracy, instead keeping the K best models (rather than 1).”
and selects a second transfer learning model having the greatest accuracy improvement when configuring an ensemble with the first transfer learning model, to configure the ensemble
(page 4 paragraph 1) “We use the greedy algorithm introduced by Caruana et al. (2004). At each step, we greedily pick the next model which minimises cross-entropy on the validation set when it is ensembled with already chosen models.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Vu and Meiseles with Mustafa for the same reasons given in claim 9 above.
Regarding Claim 11
Vu in view of Meiseles and Mustafa teaches:
The transfer learning system of claim 10
(see rejection of claim 10)
And Mustafa further teaches:
wherein the transfer learning model output unit configures the ensemble by adding a transfer learning mode having the highest accuracy improvement, when added to a previously configured ensemble, as an ensemble member.
(page 4 paragraph 1) “We use the greedy algorithm introduced by Caruana et al. (2004). At each step, we greedily pick the next model which minimises cross-entropy on the validation set when it is ensembled with already chosen models.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Vu and Meiseles with Mustafa for the same reasons given in claim 9 above.
Regarding Claim 16
Vu in view of Meiseles teaches:
The transfer learning method of claim 12
(see rejection of claim 12)
Vu in view of Meiseles does not explicitly teach:
further comprising: a transfer learning model output step of calculating classification performance accuracy for the plurality of generated transfer learning models, and selecting and outputting one or more transfer learning models in the order of the highest classification performance accuracy among the plurality of generated transfer learning models
However, Mustafa teaches:
further comprising: a transfer learning model output step of calculating classification performance accuracy for the plurality of generated transfer learning models, and selecting and outputting one or more transfer learning models in the order of the highest classification performance accuracy among the plurality of generated transfer learning models
(page 3 second to last paragraph) “Pre-trained model selection. Fine-tuning all experts on the new task would be prohibitively expensive. Following Puigcerver et al. (2020), we rank all the models by their kNN leave-one-out accuracy as a proxy for final fine-tuned accuracy, instead keeping the K best models (rather than 1).”
Vu, Meiseles, Mustafa, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transfer learning of Vu in view of Meiseles with the transfer learning ensemble taught by Mustafa because (Mustafa page 1 abstract) “When evaluated together with strong baselines on 19 different downstream tasks (the Visual Task Adaptation Benchmark), this achieves state-of-the-art performance at a much lower inference budget, even when selecting from over 2,000 pre-trained models. We also assess our ensembles on ImageNet variants and show improved robustness to distribution shift.”
Claims 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Vu in view of Meiseles and further in view of Hackett (PGPUB no. US20170344903A1) herein referred to as Hackett.
Regarding Claim 17
Vu teaches:
A transfer learning method for a deep neural network, the transfer learning method comprising: a pre-trained model selecting step of selecting a pre-trained model corresponding to transfer learning data from among a plurality of stored pre-trained models
(page 1 column 2 figure 1 caption) “Given a target task, we first compute its task embedding and then identify the most similar source task embedding (in this example, WikiHop) from a precomputed library via cosine similarity[*Examiner notes: select a pre-trained model].”; (page 7 column 2 last paragraph) “TASKEMB can substantially boost the quality of the rankings, frequently outperforming the other methods across different classes of problems, data regimes, and transfer scenarios. These results demonstrate that the task similarity between the computed embeddings is a robust predictor of effective transfer.”
the pre-trained model selecting step including, for each of a plurality of data items of the transfer learning data, generating an output of a first part or a second part of each of the plurality of stored pre-trained models as a feature of the data item,
(page 6 column 1 last paragraph) “To begin, we fine-tune BERT on the training dataset of a given task[*Examiner notes: plurality of stored pre-trained models]; the model without the final task-specific layer forms our feature extractor[*Examiner notes: second part]. Next, we feed the entire training dataset into the model[*Examiner notes: each data item of transfer learning data] and compute the task embedding based on the Fisher of the feature extractor’s parameters (weights) θ, i.e., the expected covariance of the gradients of the log-likelihood with respect to θ:”
a transfer learning step of generating a plurality of transfer learning models by performing transfer learning using the selected pre-trained model and the transfer learning data;
(page 2 column 1 last paragraph) “In each experiment, we follow the STILTs pipeline of Phang et al. (2018) by taking a pretrained BERT model,5 fine-tuning it on an intermediate source task, and then fine-tuning the resulting model on a target task[*Examiner notes: performing transfer learning using selected model and the transfer learning data].”; Figure 1
PNG
media_image1.png
501
943
media_image1.png
Greyscale
Vu does not teach:
and performing clustering of the plurality of data items by assigning each of the plurality of data items to one of a plurality of clusters using the features generated for the plurality of data items;
and a step of configuring an ensemble by selecting at least some of the plurality of transfer learning models
wherein the step of configuring the ensemble includes configuring an ensemble by adding a transfer learning model having the highest accuracy among the plurality of transfer learning models
and a transfer learning model that contributes the most to improvement of accuracy of the transfer learning model:
and sequentially adding a transfer learning model that contributes the most to the improvement of the accuracy of the pre-configured ensemble to the ensemble until a preset accuracy is satisfied
However, Meiseles teaches:
and performing clustering of the plurality of data items by assigning each of the plurality of data items to one of a plurality of clusters using the features generated for the plurality of data items;
(page 6194 column 1 section B) “a) Encode the target training data using each of the source models truncated after the global average pooling layer. This is visually represented by the dashed rectangle in figure 1. b) Calculate the Mean Silhouette Coefficient (MSC) for each set of encodings using the cosine distance metric.”; (page 6194 column 1 last paragraph) “We suggest that if this is the case the encodings representing sequences belonging to the same target class will be close to each other and far from encodings representing sequences from the other classes, before fine-tuning the source model. In this sense we expect the encodings[*Examiner notes: using derived features] from the same class[*Examiner notes: one of a plurality of clusters] to be clustered together[*Examiner notes: clustering of data items].”
Mustafa teaches:
and a step of configuring an ensemble by selecting at least some of the plurality of transfer learning models
(page 1 abstract) “The approach is simple: Use nearest-neighbour accuracy to rank pre-trained models, fine-tune the best ones with a small hyperparameter sweep, and greedily construct an ensemble to minimise validation cross-entropy.”
wherein the step of configuring the ensemble includes configuring an ensemble by adding a transfer learning model having the highest accuracy among the plurality of transfer learning models
(page 3 second to last paragraph) “Fine-tuning all experts on the new task would be prohibitively expensive. Following Puigcerver et al. (2020), we rank all the models by their kNN leave-one-out accuracy as a proxy for final fine-tuned accuracy, instead keeping the K best models (rather than 1).”
and a transfer learning model that contributes the most to improvement of accuracy of the transfer learning model:
(page 4 paragraph 1) “We use the greedy algorithm introduced by Caruana et al. (2004). At each step, we greedily pick the next model which minimises cross-entropy on the validation set when it is ensembled with already chosen models.”
Vu, Mustafa, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transfer learning of Vu with the transfer learning ensemble taught by Mustafa because (Mustafa page 1 abstract) “When evaluated together with strong baselines on 19 different downstream tasks (the Visual Task Adaptation Benchmark), this achieves state-of-the-art performance at a much lower inference budget, even when selecting from over 2,000 pre-trained models. We also assess our ensembles on ImageNet variants and show improved robustness to distribution shift.”
And Hackett teaches:
and sequentially adding a transfer learning model that contributes the most to the improvement of the accuracy of the pre-configured ensemble to the ensemble until a preset accuracy is satisfied.
(paragraph 32-33) “In response to a positive determination at block 614, the training of the ensemble 104 may be determined to be complete. That is, the ensemble 104 may be determined as having suitable accuracy to be used to classify a production dataset 112 to generate production data classification results 114. On the other hand, in response to a negative determination at block 614 indicating that the classification error is not within the error threshold, the method 600 may proceed to block 616 where the ensemble build engine may determine whether it is possible to add new ensemble members to the ensemble 104.”; [*Examiner notes: Limitation is met when the technique of Hackett is applied to the transfer learning models of Vu in view of Meiseles and Mustafa]
Vu, Meiseles, Mustafa, Hackett, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the transfer learning of Vu in view of Meiseles and Mustafa by sequentially adding models until an accuracy is reached as taught by Hackett because (Hackett paragraph 32) “That is, the ensemble 104 may be determined as having suitable accuracy to be used to classify a production dataset 112 to generate production data classification results 114”
Regarding Claim 18
Vu in view of Meiseles, Mustafa, and Hackett teaches:
The transfer learning method of claim 17
(see rejection of claim 17)
Mustafa further teaches:
wherein the step of configuring the ensemble includes: first selecting a first transfer learning model having a highest classification performance accuracy
(page 3 second to last paragraph) “Fine-tuning all experts on the new task would be prohibitively expensive. Following Puigcerver et al. (2020), we rank all the models by their kNN leave-one-out accuracy as a proxy for final fine-tuned accuracy, instead keeping the K best models (rather than 1).”
and selecting a second transfer learning model having the greatest accuracy improvement when configuring an ensemble with the first transfer learning model
(page 4 paragraph 1) “We use the greedy algorithm introduced by Caruana et al. (2004). At each step, we greedily pick the next model which minimises cross-entropy on the validation set when it is ensembled with already chosen models.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Vu and Meiseles with Mustafa for the same reasons given in claim 17 above.
Regarding Claim 19
Vu in view of Meiseles, Mustafa, and Hackett teaches:
The transfer learning method of claim 18
(see rejection of claim 18)
Mustafa further teaches:
wherein the step of configuring the ensemble includes configuring the ensemble by adding a transfer learning mode having the highest accuracy improvement, when added to a previously configured ensemble, as an ensemble member.
(page 4 paragraph 1) “We use the greedy algorithm introduced by Caruana et al. (2004). At each step, we greedily pick the next model which minimises cross-entropy on the validation set when it is ensembled with already chosen models.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Vu and Meiseles with Mustafa for the same reasons given in claim 17 above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ezra J Baker whose telephone number is (703)756-1087. The examiner can normally be reached Monday - Friday 10:00 am - 8:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/E.J.B./Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
