Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Detailed Action
The following action is in response to the communication(s) received on 02/07/2023.
As of the claims filed 02/07/2023:
Claims 1-18 are pending.
Claims 1, 6, and 7 are independent claims.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 02/16/2023, 06/01/2023, 08/16/2023, 12/21/2023, 04/30/2024, 11/21/2024, and 05/23/2025 were filed in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5, 11, and 17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 5 and 11 recite the limitation "the dataset". There is insufficient antecedent basis for this limitation in the claim.
Claims 17 is rejected for indefiniteness by virtue of dependency of its parent claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 13-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because the claim recites only A storage medium, storing a program file capable of realizing the method as recited in claims 1-6, respectively. In light of the specification, [00101], “The storage medium in the embodiments of the present application stores a program file 71 capable of implementing all the above-mentioned methods, where the program file 71 may be stored in the above-mentioned storage medium in the form of a software product...,” but is silent in excluding signals per se. Thus, Claims 13-18 are interpreted as encompassing embodiments that are no more than signals per se. Signals per se does not fall within any of the four statutory categories of patentable subject matter. Thus, Claims 13-18 are subject-matter ineligible.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Liu et al., “Sparse Deep Transfer Learning for Convolutional Neural Network” (hereinafter Liu).
Regarding Claim 1, Liu teaches:
A method for reusing parameters of a deep learning model, comprising: obtaining a target model based on a pre-configured data set, the pre-configured data set comprising a training set and a validation set; (Liu [Fig.1]
PNG
media_image1.png
494
1029
media_image1.png
Greyscale
[p.6 right ¶2] To make our results extensible, we choose the benchmark two-stream CNNs… as our Reference-SourceNet, where the spatial / temporal VGGNets are respectively pre-trained with RGB images / stacked optical flow from UCF101…Sparse-TargetNet is obtained by only pruning connections in the target domain, due to the limited data in both UCF101 and HMDB51, λ is set as two for the total training loss in Eq. (3), and the proportion of spatial/temporal stream is one/four for output fusion of two-stream net) (Note: the images and optical flow corresponds to a pre-configured data set; Sparse-TargetNet corresponds to the obtained target model)
obtaining a pre-trained original model, wherein part or all of network structures are identical between the target model and the original model; (Liu [p.6 right ¶2] To make our results extensible, we choose the benchmark two-stream CNNs… as our Reference-SourceNet, where the spatial / temporal VGGNets are respectively pre-trained with RGB images / stacked optical flow from UCF101.) (Note: the Reference-SourceNet pre-trained with the pre-configured data set corresponds to the pro-trained original model)
obtaining correspondences between layers of the target model and the original model that have identical network structures, and obtaining parameter correspondences between corresponding layers of the target model and the original model; (Liu [Fig.1]
PNG
media_image1.png
494
1029
media_image1.png
Greyscale
) (Note: Sparse-TargetNet corresponds to the target model; the fully connected layers and conv1-5 layers correspond to the obtained correspondences between layers)
extracting a plurality of original model parameters from the layers of the original model each having an identical network structure with the respective layer of the target model; (Liu [p.2 right last ¶] Hence, we exploit the iterative pruning network strategy… to reduce redundancy in the source domain, where the connections with low weights are proportionally removed in the prune-phase, and the remaining weights are fine-tuned with the source data in the retrain-phase.) (Note: using the connection from the source domain corresponds to extracting the plurality of original model parameters)
using the plurality of original model parameters to replace one by one the corresponding parameters of the target model based on the parameter correspondences, (Liu [p.3 left ¶2] (I) Main Branch is used to make prediction for target. It is our Sparse-SourceNet but with the target-domain-related modifications.... First, we change the output layer of Sparse-SourceNet to the target classes, in order to perform transfer learning from source to target. Furthermore, we propose to recover the top Nm layers of Sparse-SourceNet to be dense and re-initialize these layers randomly to increase transferability.)
validating the replaced target model with the validation set, and each time when the validation is passed, recording that the corresponding original model parameter is reusable; (Liu [p.4 left ¶3] Second, we propose to prune neurons of the inner-product layers in our main branch, due to the fact that many high-level features (neurons) of the transferred model may not be quite useful in the target domain, especially when the size of the target data is limited. Specifically, for an inner-product layer to be pruned, we compute the mean of the activation vectors over a data subset (randomly chosen from the training set), and proportionally delete the low-activated neurons...) (Note: Since low-activated neurons are not used, keeping connections from the original model with high-activated neurons correspond to recording that the parameter is reusable)
and using all reusable parameters of the original model to replace the corresponding parameters of the target model to obtain a new target model, and training the new target model. (Liu [p.4 left ¶3] Then, we perform re-training with our total loss... The resulting pruned main branch is denoted as Sparse-TargetNet which is a highly-compact, source-knowledge-integrated CNN for the target domain.) (Note: Sparse-TargetNet corresponds to the new target model)
Regarding Claim 2, Liu respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Liu further teaches:
The method as recited in claim 1, wherein the operations of validating the replaced target model, and each time when the validation is passed, recording that the corresponding original model parameter is reusable comprise: obtaining a first result derived from training the target model with the training set; (Liu [Fig.1]
PNG
media_image1.png
494
1029
media_image1.png
Greyscale
[p.6 right ¶2] To make our results extensible, we choose the benchmark two-stream CNNs… as our Reference-SourceNet, where the spatial / temporal VGGNets are respectively pre-trained with RGB images / stacked optical flow from UCF101…Sparse-TargetNet is obtained by only pruning connections in the target domain, due to the limited data in both UCF101 and HMDB51, λ is set as two for the total training loss in Eq. (3), and the proportion of spatial/temporal stream is one/four for output fusion of two-stream net) (Note: the pre-trained Reference-SourceNet corresponds to the first result)
validating the replaced target model with the validation set, and recording a second result of the validation; (Liu [p.5 right ¶2] Secondly, we prune neurons of FC6 and FC7 in our Sparse-TargetNet. This is mainly because high-level features from the transferred model are not quite important for the target domain, when the size of the target data is limited. Hence, we randomly choose 50 / 10 images per class from the training set of MIT Indoor67 / Flower102, and feed these images into our Sparse-TargetNet to compute the mean of activations (values after ReLU) for each neuron of FC6 and FC7. In each pruning iteration, we prune 512 lowest-activated neurons and perform retraining with our total loss in Eq. (3). This iterative procedure is stopped when the accuracy is the level of Standard-TransferNet. As shown in Table 2, almost half of the neurons in FC6 and FC7 are removed with little loss of accuracy.
determining whether a difference between the first result and the second result lies within a preset range; ) (Note: feeding the new domain MIT Indorr67 / Flower 102 corresponds to the validation set; evaluating each activated neuron at each pruning iteration corresponds to recording a second result of the validation)
and when the difference between the first result and the second result lies within a preset range, determining that the validation is passed and recording that the corresponding original model parameter is reusable. (Liu [p.5 right ¶2] Secondly, we prune neurons of FC6 and FC7 in our Sparse-TargetNet. This is mainly because high-level features from the transferred model are not quite important for the target domain, when the size of the target data is limited. Hence, we randomly choose 50 / 10 images per class from the training set of MIT Indoor67 / Flower102, and feed these images into our Sparse-TargetNet to compute the mean of activations (values after ReLU) for each neuron of FC6 and FC7. In each pruning iteration, we prune 512 lowest-activated neurons and perform retraining with our total loss in Eq. (3). This iterative procedure is stopped when the accuracy is the level of Standard-TransferNet. As shown in Table 2, almost half of the neurons in FC6 and FC7 are removed with little loss of accuracy.
determining whether a difference between the first result and the second result lies within a preset range; ) (Note: keeping the neurons which has high enough activations corresponds to recording that the parameters are reusable)
Regarding Claim 3, Liu respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Liu further teaches:
The method as recited in claim 1, wherein the operation of training the new target model comprises: training the new target model directly using the training set. (Liu [p.31 right ¶1] I) Source Domain: Sparse-SourceNet To make our results extensible, we choose AlexNet (trained by ImageNet) from Caffe model zoo… as our Reference-SourceNet. Then, we perform the iterative pruning strategy (Han et al. 2015) on our Reference-SourceNet to reduce model redundancy in the source domain. As a result, we obtain our Sparse-SourceNet which achieves a comparable result to (Han et al. 2015), i.e., total pruning rate (ours vs (Han et al. 2015)) is 88.0% vs 89.0%, top-1 accuracy (ours vs (Han et al. 2015)) is 57.4% vs 57.2%, top-5 accuracy (ours vs (Han et al. 2015)) is 80.4% vs 80.3%. Note that, our contribution on Sparse-SourceNet is not the pruning strategy itself but the fact that its model sparsity can reduce overfitting for transfer learning. Hence, we next design our Hybrid-TransferNet to show if our Sparse-SourceNet can help to improve transfer learning.) (Note: using ImageNet for training Reference-SourceNet corresponds to using the training set to train the new target model (Sparse-TransferNet)
Regarding Claim 4, Liu respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Liu further teaches:
The method as recited in claim 1, wherein the operation of training the new target model comprises: freezing reusable original model parameters in the new target model, and train the new target model using the training set. (Liu [p.2 right last ¶] Hence, we exploit the iterative pruning network strategy in (Han et al. 2015) to reduce redundancy in the source domain, where the connections with low weights are proportionally removed in the prune-phase, and the remaining weights are fine-tuned with the source data in the retrain-phase. We denote the original CNN as Reference-SourceNet, and denote the pruned CNN as our Sparse-SourceNet.) (Note: not removing high weights of Reference-SourceNet corresponds to freezing the original model parameters; since the parameters in Sparse-SourceNet is used to train the target model, retraining using the source data corresponds to using the training set to train the new target model)
Regarding Claim 5, Liu respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Liu further teaches:
The method as recited in claim 1, further comprising the following operation prior to training the target model with the pre-configured data set: preprocessing the dataset. (Liu [p.6 right ¶2] To make our results extensible, we choose the benchmark two-stream CNNs… as our Reference-SourceNet, where the spatial / temporal VGGNets are respectively pre-trained with RGB images / stacked optical flow from UCF101…Sparse-TargetNet is obtained by only pruning connections in the target domain, due to the limited data in both UCF101 and HMDB51, λ is set as two for the total training loss in Eq. (3), and the proportion of spatial/temporal stream is one/four for output fusion of two-stream net) (Note: the RGB images used to train Reference-SourceNet corresponds to a preprocessed dataset, thus corresponding to preprocessing the dataset prior to training the target model)
Regarding Claim 13, Liu respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Liu further teaches:
A storage medium, storing a program file capable of realizing the method as recited in claim 1. (Liu [p.2 right last ¶] …where the connections with low weights are proportionally removed in the prune-phase, and the remaining weights are fine-tuned with the source data in the retrain-phase.) (Note: fine-tuning the weights corresponds to training the neural network, which requires a program file capable of performing the method.)
Independent Claim 6 recites a target model with a pre-configured first data set, and training an original model with a pre-configured second data set (Liu [p.31 right ¶1] I) Source Domain: Sparse-SourceNet To make our results extensible, we choose AlexNet (trained by ImageNet))…
[p.5 right ¶2] …we randomly choose 50 / 10 images per class from the training set of MIT Indoor67 / Flower102, and feed these images into our Sparse-TargetNet to compute the mean of activations (values after ReLU) for each neuron of FC6 and FC7.) (Note: ImageNet corresponds to the second data set; MIT Indoor67 / Flower10 corresponds to the first data set) to perform precisely the methods of Claim 1. Thus, Claim 6 is rejected for reasons set forth in Claim 1.
Regarding Claim 12, Liu respectively teaches and incorporates the claimed limitations and rejections of Claim 6. Liu further teaches:
A terminal, comprising a processor and a memory coupled to the processor, wherein the memory stores program instructions for realizing the method as recited in claim 6; and wherein the processor is configured to execute the program instructions stored in the memory to realize parameter reuse across different deep learning models. (Liu [p.2 right last ¶] …where the connections with low weights are proportionally removed in the prune-phase, and the remaining weights are fine-tuned with the source data in the retrain-phase…
[fig.1; Reference-SourceNet; Hybrid-TransferNet; Sparse-TargetNet each correspond to a different deep learning model]) (Note: fine-tuning the weights corresponds to training the neural network, which requires a processor, memory, and a program file capable of performing the method.)
Independent Claim 7 recites A terminal, comprising a processor and a memory coupled to the processor, wherein the memory stores program instructions for realizing a method for reusing parameters of a deep learning model; and wherein the processor is configured to execute the program instructions stored in the memory to realize parameter reuse across different deep learning models; wherein the method comprises: (Liu [p.2 right last ¶] …where the connections with low weights are proportionally removed in the prune-phase, and the remaining weights are fine-tuned with the source data in the retrain-phase…[fig.1; Reference-SourceNet; Hybrid-TransferNet; Sparse-TargetNet each correspond to a different deep learning model])) (Note: fine-tuning the weights corresponds to training the neural network, which requires a processor, memory, and a program file capable of performing the method.) to perform precisely the methods of Claim 1. Thus, Claim 7 is rejected for reasons set forth in Claim 1.
Claims 8-11, dependent on Claim 7, also recite the system configured to perform precisely the methods of Claims 2-6, respectively. Thus, Claims 8-11 are rejected for reasons set forth in Claims 2-5, respectively.
Claims 14-18, dependent on claims 2-6, respectively, recite precisely the limitations of Claim 13. Thus, they are rejected for reasons set forth in Claim 13.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSEP HAN whose telephone number is (703)756-1346. The examiner can normally be reached Mon-Fri 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached on (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.H./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights