Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments filed 02/13/2026 on pages 10-11 of Remarks regarding the rejection under 35 U.S.C. 103 with respect to claims 1, 3-8, 10-15 and 17-23 have been fully considered but are moot. New reference Alain has been incorporated below to teach the newly presented limitations.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-8, 10-15 and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Baughman et al. (US20200175408A1); hereinafter Baughman in view of Vila Casado et al. (US20200210810A1); hereinafter Vila Casado in view of Kontschieder et al. (Deep Neural Decision Forests); hereinafter Kontschieder and in further view of Alain et al. (Understanding intermediate layers using linear classifier probes); hereinafter Alain
Claim 1 is rejected over Baughman, Vila Casado, Alain and Kontschieder.
Regarding claim 1, Baughman teaches a method comprising:
receiving, by one or more processors of a computer system, model data; (“During the training, the deep learning program module 420 performs partial domain adaptation on the core deep learning model 430 by adjusting the weights used in each of the layers from their initial values (taken from the external deep learning models) (model data) to improve the recognition performance (e.g., a confidence level) of each layer.”; [0076])
slicing, by the one or more processors of the computer system, each of the plurality of [supervised models] into individual layers of the plurality of layers; (“Still referring to step 500, in embodiments, the testing performed by the deep learning program module 420 includes feeding testing data including unseen exemplars (i.e., data/patterns not previously seen by the external deep learning models) into the layers of the external deep learning models and determining whether or not the external deep learning models are able to correctly recognize (classify) the testing data. In an example, a layer in an external deep learning model may classify images as “animals” or “not animals.”; [0070]; Note: The testing and selection of individual layers is slicing.)
Baughman does not teach training, by the one or more processors of the computer system, a plurality of supervised models using the model data, each of the plurality of supervised models including a plurality of layers;
However, Vila Casado teaches training, by the one or more processors of the computer system, a plurality of supervised models using the model data, each of the plurality of supervised models including a plurality of layers; (“Multiple neural net models are trained independently, such that the implemented embodiment can take advantage of model diversity, feature diversity, and team diversity. Model diversity comes from having different neural network architectures, feature diversity comes from having different types of inputs into the network (raw and expert engineered), and team diversity comes from having diverse teams that can optimize the different classifiers/networks in parallel. DLNNs 240 are trained based on raw data from labeled dataset 210, while shallow classifiers 230 are trained based on expert engineered features 220.”; [0024]; The models are supervised models because they are trained on a labeled dataset.)
It would have been obvious before the effective filing date to combine the concatenation of layers from a plurality of models Baughman and the supervised models of Vila Casado to improve classification accuracy (Vila Casado, [0020]). Baughman and Vila Casado are analogous art because they both concern layer selection and concatenation into a new core model.
Baughman does not teach inputting, by the one or more processors of the computer system, the individual layers into a shallow neural network to determine a plurality of feature classes that contribute to detection of an overall object;
However, Vila Casado teaches inputting, by the one or more processors of the computer system, the individual layers of the plurality of layers into a shallow neural network to determine one or more feature classes, from a plurality of feature classes, [predicted by each of the individual layers] of each of the plurality of supervised models, [wherein the plurality of feature classes correspond to features] that contribute to detection of an overall object; (“That new set of features, which includes the concatenated layers described above, is then fed through a new classifier to form a single final classifier that uses the best parts of each input classifier.”; [0019]; “FIG. 2B is an enlarged portion of FIG. 2A that shows feature fusion 250 and fusion learning classifier 260 (shallow neural network), according to an embodiment of the present invention. The last two neuron layers from kerasNN 231 and ResNeXt 241 are horizontally concatenated. The neural layers for kerasNN 231 are batch normalized. This input layer is connected to a first hidden dense layer with a rectified linear unit (ReLU) activation function. This first hidden layer is connected to a second hidden dense layer with a ReLU activation function. The output layer is a dense layer with a softmax activation function connected to the second hidden layer.”; [0030]; and “outputs 312 from the output layer and results 314 from the last intermediate layer for each original neural network 310 for all of the training data and training truth data 320 are provided to merging neural network 330 (shallow neural network).”; [0045]; Note: See Fig. 3 of Vila Casado to see that the features are processed through the Merging Neural Network to process and output the classifications;)
It would have been obvious before the effective filing date to combine the concatenation of layers from a plurality of models Baughman and the processing of individual layers of Vila Casado to improve classification accuracy (Vila Casado, [0020]). Baughman and Vila Casado are analogous art because they both concern layer selection and concatenation into a new core model.
Baughman does not appear to explicitly teach [inputting, by the one or more processors of the computer system, the individual layers of the plurality of layers into a shallow neural network to determine one or more feature classes, from a plurality of feature classes,] predicted by each of the individual layers [of each of the plurality of supervised models,] wherein the plurality of feature classes correspond to features [that contribute to detection of an overall object];
However, Alain teaches [inputting, by the one or more processors of the computer system, the individual layers of the plurality of layers into a shallow neural network to determine one or more feature classes, from a plurality of feature classes,] predicted by each of the individual layers [of each of the plurality of supervised models,] wherein the plurality of feature classes correspond to features [that contribute to detection of an overall object]; (Alain [page 4, section 3.2 Linear classifier probes]: “At every layer we can take the features Hk from that layer and try to predict the correct labels y using a linear classifier parameterized as”; and [page 1, 1 Introduction]: “we take the features of each layer separately and we fit a linear classifier to predict the original classes.”
It would have been obvious before the effective filing date to combine the concatenation of layers from a plurality of models of Baughman with the individual layers of Alain to improve test prediction error (Alain, page 6). Baughman and Alain are analogous art because they both concern evaluating individual layers of a deep neural network.
Baughman does not teach calculating, by the one or more processors of the computer system, prediction accuracies corresponding to the one or more feature classes predicted by each of the individual layers of each [of the plurality of supervised models; and]
However, Alain teaches calculating, by the one or more processors of the computer system, prediction accuracies corresponding to the one or more feature classes predicted by each of the individual layers of each [of the plurality of supervised models; and] (Alain [page 5]: “we chose to report the classification Error”; [page 4, 3.2 Linear classifier probes]: “At every layer we can take the features Hk from that layer and try to predict the correct labels y using a linear classifier parameterized as”; and [page 1, 1 Introduction]: “we take the features of each layer separately and we fit a linear classifier to predict the original classes.” Note: Alain shows prediction error, the inverse of prediction accuracy)
It would have been obvious before the effective filing date to combine the concatenation of layers from a plurality of models of Baughman with the individual layers of Alain to improve test prediction error (Alain, page 6). Baughman and Alain are analogous art because they both concern evaluating individual layers of a deep neural network.
Baughman teaches combining, by the one or more processors of the computer system, a sequence of the individual layers selected from different models of the plurality of [supervised] models into a composite model of each of the plurality of [supervised] models. (“concatenate (combine) the selected layers from the plurality of external deep learning models to form a core deep learning model (composite model)”; [0005])
Baughman does not appear to explicitly teach based on the calculated prediction accuracies,
However, Alain teaches based on the calculated prediction accuracies, (Alain [page 5]: “we chose to report the classification Error”; [page 4, 3.2 Linear classifier probes]: “At every layer we can take the features Hk from that layer and try to predict the correct labels y using a linear classifier parameterized as”; and [page 1, 1 Introduction]: “we take the features of each layer separately and we fit a linear classifier to predict the original classes.” Note: Alain shows prediction error, the inverse of prediction accuracy)
It would have been obvious before the effective filing date to combine the concatenation of layers from a plurality of models of Baughman with the individual layers of Alain to improve test prediction error (Alain, page 6). Baughman and Alain are analogous art because they both concern evaluating individual layers of a deep neural network.
Baughman does not teach wherein a first plurality of layers of the sequence of the plurality of layers corresponds to a convolutional neural network and a second plurality of layers of the sequence of the plurality of layers corresponds to a decision tree network.
However, Kontschieder teaches wherein a first plurality of layers of the sequence of the plurality of layers corresponds to a convolutional neural network and a second plurality of layers of the sequence of the plurality of layers corresponds to a decision tree network. (“Fig. 2 provides a schematic illustration of this idea, showing how decision nodes can be implemented by using typically available fully-connected (or inner-product) and sigmoid layers in DNN frameworks like Caffe or MatConvNet. Easy to see, the number of split nodes is determined by the number of output nodes of the preceding fully-connected layer.”; page 1470-1471; Note: See Figure 2 to see the illustration of the top being a Deep CNN (first plurality of layers) and the second plurality of layers being the decision nodes that come with the decision tree.)
It would have been obvious before the effective filing date to combine the concatenation of layers from a plurality of models Baughman and the convolutional decision tree structure of Kontschieder for improved prediction performance (Kontschieder, page 1472, 5.2 Improving Performance with dNDF). Baughman and Kontschieder are analogous art because they both concern learning using convolutional neural networks.
Claim 3 is rejected over Baughman, Vila Casado, Alain and Kontschieder with the incorporation of claim 1.
Regarding claim 3, Baughman teaches concatenating, by the one or more processors of the computer system, the sequence of the plurality of layers (“At step 520, the computer server 410 concatenates the selected layers to form the core deep learning model 430. In embodiments, the deep learning program module 420 concatenates the layers from the external deep learning models selected at step 510 to form the core deep learning model 430. In particular, in concatenating the selected layers to form the core deep learning model 430, the deep learning program module 420 maintains the ordering of layers from the external deep learning models and also reuses the weights from the external deep learning models.”; [0073]) in an order that begins with low level features and ends with high level features. (“In an example, each of the external deep learning models may have three layers, including a first layer that performs coarse-grained recognition, a second layer that performs medium-grained recognition, and a third layer that performs fine-grained recognition. (sequential ordering of layers from low to high level features)”; [0074])
Claim 4 is rejected over Baughman, Vila Casado, Alain and Kontschieder with the incorporation of claim 1.
Regarding claim 4, Baughman does not teach wherein the plurality of supervised models comprises different types of models.
However, Vila Casado teaches wherein the plurality of supervised models comprises different types of models. (“DLNNs 240 are trained based on raw data from labeled dataset 210, while shallow classifiers 230 are trained based on expert engineered features 220.”; [0024]; Note: Because the models are trained on labeled data, they are supervised models.)
It would have been obvious before the effective filing date to combine the concatenation of layers from a plurality of models Baughman and the different types of supervised models of Vila Casado to improve classification accuracy (Vila Casado, [0020]). Baughman and Vila Casado are analogous art because they both concern layer selection and concatenation into a new core model.
Claim 5 is rejected over Baughman, Vila Casado, Alain and Kontschieder with the incorporation of claim 1.
Regarding claim 5, Baughman does not teach wherein an output of a first layer of the sequence is not connected directly to an input of a second layer of the sequence during training, and
wherein the output of the first layer is connected to the input of the second layer after the training.
However, Vila Casado teaches wherein an output of a first layer of the sequence is not connected directly to an input of a second layer of the sequence during training, and
wherein the output of the first layer is connected to the input of the second layer after the training. (“Multiple neural net models are trained independently, such that the implemented embodiment can take advantage of model diversity, feature diversity, and team diversity. Model diversity comes from having different neural network architectures, feature diversity comes from having different types of inputs into the network (raw and expert engineered), and team diversity comes from having diverse teams that can optimize the different classifiers/networks in parallel. DLNNs 240 are trained based on raw data from labeled dataset 210, while shallow classifiers 230 are trained based on expert engineered features 220.”; [0024]; Note: The models are trained separately, therefore the layers are also trained separately before being combined.).
It would have been obvious before the effective filing date to combine the concatenation of layers from a plurality of models Baughman and independently trained supervised models of Vila Casado to improve classification accuracy (Vila Casado, [0020]). Baughman and Vila Casado are analogous art because they both concern layer selection and concatenation into a new core model.
Claim 7 is rejected over Baughman, Vila Casado, Alain and Kontschieder with the incorporation of claim 1.
Regarding claim 7, Baughman does not teach comparing, by the one or more processors of the computer system, accuracy of the composite model relative to the accuracy of individual models of the plurality of supervised models.
However, Vila Casado teaches comparing, by the one or more processors of the computer system, accuracy of the composite model relative to the accuracy of individual models of the plurality of supervised models. (“The accuracy score of DLNNs (individual models) 240 was 74.693. However, the accuracy score of fusion learning classifier 260 after feature fusion 250 (composite model) was 76.422, which was better than SLNNs 230 or DLNNs 240 alone.”; [0027])
It would have been obvious before the effective filing date to combine the concatenation of layers from a plurality of models Baughman and comparison of models of Vila Casado to improve classification accuracy (Vila Casado, [0020]). Baughman and Vila Casado are analogous art because they both concern layer selection and concatenation into a new core model.
Claim 8 is rejected over Baughman, Vila Casado, Alain and Kontschieder.
Regarding claim 8, Baughman teaches a computer system, comprising:
one or more processors; and
one or more memory devices coupled to the one or more processors, wherein the one or more processors are configured to:
(“there is a system that includes: a hardware processor, a computer readable memory, and a computer readable storage medium associated with a computing device; program instructions to select layers from a plurality of external deep learning models; program instructions to concatenate the selected layers from the plurality of external deep learning models to form a core deep learning model; [0006])
The remainder of claim 8 is claim 1 in the form of a computer system and is rejected for the same reasons as claim 1 stated above.
Dependent claim 10 is claim 3 in the form of a computer system and is rejected for the same reasons as claim 3 stated above. For the rejection of the limitations specifically pertaining to the computer system of claim 8, see the rejection of claim 8 above.
Dependent claim 11 is claim 4 in the form of a computer system and is rejected for the same reasons as claim 4 stated above. For the rejection of the limitations specifically pertaining to the computer system of claim 8, see the rejection of claim 8 above.
Dependent claim 12 is claim 5 in the form of a computer system and is rejected for the same reasons as claim 5 stated above. For the rejection of the limitations specifically pertaining to the computer system of claim 8, see the rejection of claim 8 above.
Dependent claim 13 is claim 6 in the form of a computer system and is rejected for the same reasons as claim 6 stated above. For the rejection of the limitations specifically pertaining to the computer system of claim 8, see the rejection of claim 8 above.
Dependent claim 14 is claim 7 in the form of a computer system and is rejected for the same reasons as claim 7 stated above. For the rejection of the limitations specifically pertaining to the computer system of claim 8, see the rejection of claim 8 above.
Claim 15 is rejected over Baughman, Vila Casado, Alain and Kontschieder.
Regarding claim 15, Baughman teaches a non-transitory computer-readable medium storing a set of instructions for wireless communication, the set of instructions comprising: one or more instructions that, when executed by one or more processors of a device, cause the device to: (“there is a system that includes: a hardware processor, a computer readable memory, and a computer readable storage medium associated with a computing device; program instructions to select layers from a plurality of external deep learning models; program instructions to concatenate the selected layers from the plurality of external deep learning models to form a core deep learning model; [0006])
The remainder of claim 15 is claim 1 in the form of a non-transitory computer-readable medium and is rejected for the same reasons as claim 1 stated above.
Dependent claim 17 is claim 3 in the form of a non-transitory computer-readable medium and is rejected for the same reasons as claim 3 stated above. For the rejection of the limitations specifically pertaining to the non-transitory computer-readable medium of claim 15, see the rejection of claim 15 above.
Dependent claim 18 is claim 5 in the form of a non-transitory computer-readable medium and is rejected for the same reasons as claim 5 stated above. For the rejection of the limitations specifically pertaining to the non-transitory computer-readable medium of claim 15, see the rejection of claim 15 above.
Dependent claim 20 is claim 7 in the form of a non-transitory computer-readable medium and is rejected for the same reasons as claim 7 stated above. For the rejection of the limitations specifically pertaining to the non-transitory computer-readable medium of claim 15, see the rejection of claim 15 above.
Claims 6 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Baughman, Vila Casado, Kontschieder and Alain in view of Nagae et al. (“Automatic layer selection for transfer learning and quantitative evaluation of layer effectiveness”); hereinafter Nagae
Claim 6 is rejected over Baughman, Vila Casado, Kontschieder, Alain and Nagae with the incorporation of claim 1.
Regarding claim 6, Baughman does not teach using at least one transfer learning technique, by the one or more computer processors of the computer system, to determine best features from the individual layers to produce an output class with high accuracy.
However, Nagae teaches using at least one transfer learning technique, by the one or more computer processors of the computer system, to determine best features from the individual layers to produce an output class with high accuracy. (“We propose a method to evaluate the update layer’s effectiveness in transfer learning by quantitatively assessing how much each layer (individual layers) of the pretrained model detects common features in the source and target datasets. The pretrained model’s convolutional filter detects a specific feature from the input image and generates an activation feature map, as shown in Fig. 5. The degree of feature detection of each layer is evaluated quantitatively using the OTDD for the activation feature map of the convolutional filter of each layer of the pretrained model for the target datasets.”; page 155, section 3.2)
It would have been obvious before the effective filing date to combine the concatenation of layers from a plurality of models Baughman and the layer selection based on the feature detection of Nagae to improve transfer learning performance (Nagae, page 160, 6. Conclusion, last paragraph). Baughman and Nagae are analogous art because they both concern layer selection and transfer learning.
Dependent claim 19 is claim 6 in the form of a non-transitory computer-readable medium and is rejected for the same reasons as claim 6 stated above. For the rejection of the limitations specifically pertaining to the non-transitory computer-readable medium of claim 15, see the rejection of claim 15 above.
Claims 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over Baughman Vila Casado, Kontschieder and Alain in further view of Zagoruyko et al. (“PAYING MORE ATTENTION TO ATTENTION: IMPROVING THE PERFORMANCE OF CONVOLUTIONAL NEURAL NETWORKS VIA ATTENTION TRANSFER”); hereinafter Zagoruyko
Claim 21 is rejected over Baughman, Vila Casado, Kontschieder, Alain and Zagoruyko with the incorporation of claim 1.
Regarding claim 21, Baughman does not teach wherein the slicing is performed at each layer of the plurality of layers taking output from attention layers.
However, Zagoruyko teaches wherein the slicing is performed at each layer of the plurality of layers taking output from attention layers. (“we consider attention as a set of spatial maps that essentially try to encode on which spatial areas of the input the network focuses most for taking its output decision (e.g., for classifying an image), where, furthermore, these maps can be defined w.r.t. various layers of the network so that they are able to capture both low-, mid-, and high-level representation information. More specifically, in this work we define two types of spatial attention maps: activation-based and gradient-based”; page 1, Introduction; and “more, attention maps focus on different parts for different layers in the net”; page 4, paragraph 1)
It would have been obvious before the effective filing date to combine the concatenation of layers from a plurality of models Baughman and the attention maps of Zagoruyko to effectively focus on different parts for different layers in the network (Zagoruyko, page 4, paragraph 1). Baughman and Zagoruyko are analogous art because they both concern focusing on different layers of a neural network.
Dependent claim 22 is claim 21 in the form of a computer system and is rejected for the same reasons as claim 21 stated above. For the rejection of the limitations specifically pertaining to the computer system of claim 8, see the rejection of claim 8 above.
Dependent claim 23 is claim 21 in the form of a non-transitory computer-readable medium and is rejected for the same reasons as claim 21 stated above. For the rejection of the limitations specifically pertaining to the non-transitory computer-readable medium of claim 15, see the rejection of claim 15 above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zhang et al., “Interpreting CNNs via Decision Trees”
Frosst et al., “Distilling a Neural Network Into a Soft Decision Tree”
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID H TRAN whose telephone number is (703)756-1525. The examiner can normally be reached M-F 9:30 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID H TRAN/Examiner, Art Unit 2147
/VIKER A LAMARDO/Supervisory Patent Examiner, Art Unit 2147
Prosecution Insights