DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This office action is in response to the amendment filed on 07/29/2025. Claims remain pending in the application. Claims 1, 7, and 13 are independent.
Drawings
Applicant's amendment to specification does not fully resolve the issues raised in the previous Office Action. However, as pointed out in Page 9 of the Remarks that FIG. 7 is in fact indicating replacing information related to any of network layers (e.g., 704, 708, 710, 712) in some sense, hence the previous Drawing objection is withdrawn and instead Specification objection is raised to correct remaining issue.
Specification
The disclosure is objected to because of the following informalities:
in ¶ [0060], "FIG. 7 shows an example neural network model 700, experimented with replacing some network layers of a neural network model with deep forest tree layers … In this example experiment, information related to network layer 1 704 is replaced with deep forest tree layers (generated)1 706. Input 702 is then provided to deep forest 1 706 as input, and the output from deep forest 1 706 is then provided to network layer 2 708 (instead of output from network layer 1 704) … … Subsequent experiments are then executed, with each experiment corresponding to replacing information relating to a different layer of neural network 700 with a corresponding deep forest layer (information), with their resulting outputs recorded and evaluated as in the first experiment …" appears to be "FIG. 7 shows an example neural network model 700, experimented with replacing some network layers of a neural network model with deep forest tree layers … In first example experiment, information related to network layer 1 704 is replaced with deep forest 1 706. Input 702 is then provided to deep forest 1 706 as input, and the output from deep forest 1 706 is then provided to network layer 2 708 (instead of output from network layer 1 704) … Subsequent experiments are then executed, with each experiment corresponding to replacing information relating to a different layer of neural network 700 with a corresponding deep forest layer (information), with their resulting outputs recorded and evaluated as in the first experiment …" for naming consistency in 706 and indicating "information related to network layer 1 704 is replaced with deep forest tree 1 706" described in the specification is not the example experiment shown in FIG. 7.
Appropriate correction is required.
Claim Objections
Claims 1-5 and 7-18 are objected to because of the following informalities:
in Claim 1, lines 7-8; Claim 7, lines 11-12; Claims 13, lines 12-13, "… creating an associated training data …" appears to be "… creating associated training data …";
in Claim 2, lines 2-3; Claim 8, lines 3-4; Claim 14, lines 3-4, "… ranking the experimental deep learning model of the set of experimental deep learning models based, at least in part, on an accuracy metric" appears to be "… ranking said each experimental deep learning model of the set of experimental deep learning models based, at least in part, on an accuracy metric" or "… ranking the set of experimental deep learning models based, at least in part, on an accuracy metric";
in Claim 3, lines 1-3; Claim 9, lines 1-3; Claim 15, lines 1-3, "… comparing output of the experimental deep learning models with output of the deep learning model …" appears to be "… comparing output from the set of experimental deep learning models with output of the deep learning model …";
in Claim 4, lines 2-3; Claim 10, lines 3-4; Claim 16, lines 3-4, "… generating a modified deep learning model with at least one network layer replaced by a tree model from the set of tree models" appears to be … generating a modified deep learning model with at least one network layer replaced by one tree model from the set of tree models" to distinguish with "… a copy of the deep learning model having a different network layer replaced with a tree model from the set of tree models" recited in their respective based claim;
in Claim 5, lines 1-3; Claim 11, lines 1-3; Claim 17, lines 1-3, "… wherein replacing the at least one network layer is based, at least in part, on the ranking of experimental deep learning models, where experimental deep learning models corresponding to high accuracy metrics indicate …" appears to be "… wherein replacing the at least one network layer is based, at least in part, on the ranking of the set of experimental deep learning models, where experimental deep learning models corresponding to high accuracy metrics indicate …";
in Claim 7, lines 4-5 and Claim 13, lines 5-6, "… the computer code including instructions for causing a processor(s) set to perform operations including the following: …" appears to be "… the computer code including instructions for causing a processor(s) set to perform operations including following: …";
in Claim 8, lines 1-2; Claim 10, lines 1-2; Claim 12, lines 1-2; Claim 14, lines 1-2; Claim 16, lines 1-2; and Claim 18, lines 1-2, "… wherein the computer code further includes instructions for causing the processor(s) set to perform the following operations: …" appears to be "… wherein the computer code further includes instructions for causing the processor(s) set to perform following operations: …".
Appropriate correction is required.
Claim Rejections - 35 USC § 112
Applicant's amendment to claims corrects previous rejections; therefore, the previous rejections are withdrawn. Applicant's amendment to claims also raises the following new rejections.
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 1, 7, and 13 recite "determining when any of the network layers require performance improvements" in lines 9,13, and 14 respectively. The specification in ¶ [0045] only describes "… determine which network layers of the trained neural network are most suitable for replacement with tree models". However, there is no description found in the specification to support the afore-mentioned limitation. If the Examiner has overlooked the portion of the original Specification that describes these features of the present invention, the Applicant should point it out (by paragraph number or page number with line number) in response to the Office Action. For examination purposes, "determining which network layers require performance improvements" is considered.
Claims 2-6, 8-12, and 14-18 are rejected for fully incorporating the deficiency of their respective base claims.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 7, and 13 recite the limitation "... extracting output tensors from network layers of the deep learning model, wherein said network layers belong to a central processing unit (CPU) of a computing environment; for at least some network layers of the deep learning model, training a set of tree models, where a given tree model corresponds to an individual network layer and creating an associated training data for each of the network layers; determining when any of the network layers require performance improvements; providing information relating to training the set of tree models to each layer of network determined to require an improvement in performance by ..." in lines 4-12, 8-16, and 9-17 respectively, which rendering these claims indefinite because (1) it is unclear which "network layers" (1st instance of "network layers" or 3rd instance of "at least some network layers") is referred by the 4th and 5th instances of the "network layers"; and (2) it is unclear whether "each of the network layers" and "each layer of network" are the same or different. For examination, "... extracting output tensors from network layers of the deep learning model, wherein said network layers belong to a central processing unit (CPU) of a computing environment; for at least some network layers of the deep learning model, training a set of tree models, where a given tree model corresponds to an individual network layer and creating associated training data for each of the at least some network layers; determining which network layers require performance improvements; providing information relating to training the set of tree models to each network layer determined to require an improvement in performance by ..." is considered (see also Claim Objections and 112(a) Rejections to Claims 1, 7, and 13).
Claims 1, 7, and 13 recite the limitation "… for at least some network layers of the deep learning model, training a set of tree models … providing information relating to training the set of tree models to each layer of network determined to require an improvement in performance by providing and storing an input vector and generating a related output vector …" in lines 6-12, 10-16, and 11-17 respectively, which rendering these claims indefinite because it is unclear what are "an input vector" and "a related output vector"? ("an input vector" and "a related output vector" of "deep learning model", "a set of tree models", or "each layer of network determined to require an improvement in performance"?) Clarification is required.
Claims 2-6, 8-12, and 14-18 are rejected for fully incorporating the deficiency of their respective base claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5, 7-11, and 13-17 are rejected under 35 U.S.C. 101 because
the claimed invention is directed to abstract idea without significantly more.
Independent Claims 1, 7, and 13
Step 1: Claim 1 is a process claim, Claim 7 is a claim for a product including a machine/computer readable storage device (excluding transitory signal per se. in ¶ [0022]), and Claim 13 is a claim for a system including a machine/computer readable storage device (excluding transitory signal per se. in ¶ [0022]). These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) "extracting output tensors from network layers of the deep learning model", "a given tree model corresponds to an individual network layer", "creating an associated training data for each of the network layers", "determining when any of the network layers require performance improvements", "each layer of network determined to require an improvement in performance", and "generating a set of experimental deep learning models, using said related output vector, with each experimental deep learning model corresponding to a copy of the deep learning model having a different network layer replaced with a tree model from the set of tree models" (i.e., human mind is capable of replacing "a different network layer" in "a copy of the deep learning model" with "a tree model" from "the set of tree models" to generate "a set of deep learning experimental models" with different architectures) which can be reasonably considered as mental processes (i.e., which "can be performed in the human mind, or by a human using a pen and paper") or mathematical concepts/calculations/algorithms.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) recite(s) additional elements/limitations of "a machine readable storage device" (Claims 7 and 13), "a processor(s) set" Claims 7 and 13), "a central processing unit (CPU) of a computing environment", "receiving a deep learning model with corresponding training input datasets and training labels", "said network layers belong to a central processing unit (CPU) of a computing environment", "for at least some network layers of the deep learning model, training a set of tree models", "providing information relating to training the set of tree models to each layer of network", "providing and storing an input vector", and generating a related output vector" which only amount to "apply it" with the use of generic computer components or insignificant extra solution activity. None of the additional elements/limitations, taken alone or in combination, integrate the abstract idea into a practical application.
Step 2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because (a) the additional .
Claims 2, 8, and 14
Step 1: Claim 2 is a process claim, Claim 8 is a claim for a product including a machine/computer readable storage device (excluding transitory signal per se. in ¶ [0022]), and Claim 14 is a claim for a system including a machine/computer readable storage device (excluding transitory signal per se. in ¶ [0022]). These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) "ranking the experimental deep learning model of the set of experimental deep learning models based, at least in part, on an accuracy metric" which can be reasonably considered as mental processes (i.e., which "can be performed in the human mind, or by a human using a pen and paper") or mathematical concepts/calculations/algorithms.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 3, 9, and 15
Step 1: Claim 3 is a process claim, Claim 9 is a claim for a product including a machine/computer readable storage device (excluding transitory signal per se. in ¶ [0022]), and Claim 15 is a claim for a system including a machine/computer readable storage device (excluding transitory signal per se. in ¶ [0022]). These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) "the accuracy metric is determined by comparing output of the experimental deep learning models with output of the deep learning model using the training input datasets and the training labels" which can be reasonably considered as mental processes (i.e., which "can be performed in the human mind, or by a human using a pen and paper") or mathematical concepts/calculations/algorithms.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 4, 10, and 16
Step 1: Claim 4 is a process claim, Claim 10 is a claim for a product including a machine/computer readable storage device (excluding transitory signal per se. in ¶ [0022]), and Claim 16 is a claim for a system including a machine/computer readable storage device (excluding transitory signal per se. in ¶ [0022]). These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) "generating a modified deep learning model with at least one network layer replaced by a tree model from the set of tree models" (i.e., human mind is capable of replacing "at least one network layer" in "a deep learning model" with "a tree model" from "the set of tree models" to generate "a modified deep learning model" with a different architecture) which can be reasonably considered as mental processes (i.e., which "can be performed in the human mind, or by a human using a pen and paper") or mathematical concepts/calculations/algorithms.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 5, 11, and 17
Step 1: Claim 5 is a process claim, Claim 11 is a claim for a product including a machine/computer readable storage device (excluding transitory signal per se. in ¶ [0022]), and Claim 17 is a claim for a system including a machine/computer readable storage device (excluding transitory signal per se. in ¶ [0022]). These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) "replacing the at least one network layer is based, at least in part, on the ranking of experimental deep learning models, where experimental deep learning models corresponding to high accuracy metrics indicate which network layers of the deep learning model are replaceable by tree models determined not to be affected by output accuracy of the modified deep learning model relative to the deep learning model" which can be reasonably considered as mental processes (i.e., which "can be performed in the human mind, or by a human using a pen and paper") or mathematical concepts/calculations/algorithms.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 6, 12, and 18
Step 1: Claim 6 is a process claim, Claim 12 is a claim for a product including a machine/computer readable storage device (excluding transitory signal per se. in ¶ [0022]), and Claim 18 is a claim for a system including a machine/computer readable storage device (excluding transitory signal per se. in ¶ [0022]). These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) does/do not further recite(s) elements/limitations which can be reasonably considered as mental processes (i.e., which "can be performed in the human mind, or by a human using a pen and paper") or mathematical concepts/algorithms/calculations.
Step 2A Prong 2: This judicial exception is integrated into a practical application because the claim(s) further recite(s) additional element/limitation of "outputting the modified deep learning model to a CPU reliant environment" (i.e., deploying the modified deep learning model in a CPU reliant environment) which integrates with other elements/limitations in a meaningful way so that the improvement of a practical application indicating in ¶¶ [0051]-[0052] of the specification (e.g., in order to improve the performance of the deep learning model in the CPU environment, some network layers in the deep neural network need to be replaced with a decision tree type model) is reflected in these claims as a whole.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-18 are rejected under 35 U.S.C. 103 as being unpatentable over Nunes Coelho, Jr. et al. (US 2024/0220867 A1, filed on 05/10/2021), hereinafter Nunes Coelho in view of Wan et al. ("NBDT: NEURAL-BACKED DECISION TREE", arXiv:2004.00221v3, Jan. 28, 2021, pp. 1-19), hereinafter Wan.
Independent Claims 1, 7, and 13
Nunes Coelho discloses a computer-implemented method (CIM) comprising: receiving a deep learning model with corresponding training input datasets mislabeling, or a log loss representing information gain based on entropy theory, or any other suitable losses for training a decision tree; ¶ [0073] with 310 in FIG. 3: receives data representing a neural network comprising a plurality of layers arranged in a sequence 310; ¶ [0080] in 340 in FIG. 3: train the new machine learning model based on training data for the original neural network 340);
extracting output tensors from network layers of the deep learning model, wherein said network layers belong to a central processing unit (CPU) of a computing environment (Nunes Coelho, ¶ [0003]: each neural network layer includes a plurality of nodes, and each layer represents a set of operations defined by the neural network; in general, these operations are arithmetic operations that can include linear operations, e.g., additions and multiplications, and nonlinear operations, e.g., non-linear activation functions like "Relu" or "Sigmoid" functions; the linear operations combine layer inputs and weights for the layer; the linear operations for each layer can be implemented using tensor operations, in which the weights of a layer are presented in a matrix or tensor form, and the layer inputs for the layer are presented in a vector form; ¶ [0046] with FIG. 1: quantize an activation tensor, a weight tensor of a neural network layer, or the layer outputs from a precision of 8-bit to 4-bit or even 1-bit ( e.g., one bit for the mantissa); ¶¶ [0053]-[0054] and [0106] with FIG. 7: the training engine 140 used for training at least a portion of the new machine learning models 130 can include central processing units (CPUs); computing units such as MUX units or arithmetic logic (ALC) units can be used to perform operations in decision trees, a neural network inference engine or system can replace large, expensive computing units such as TPUs or GPUs with smaller and cheaper programmable cores with one or more MUX units or ALC units to perform inference computations for decision trees of the new machine learning model 180; ¶¶ [0113]-[0118]: the neural network output may comprise a feature representation, which may then be further processed to generate a system output; the neural network is configured to receive an input data item and to process the input data item to generate a feature representation of the input data item in accordance with the network parameters; generally, a feature representation of a data item is an ordered collection of numeric values, e.g., a vector, that represents the data item as a point in a multi-dimensional feature space; in other words, each feature representation may include numeric values for each of a plurality of features of the input data item (i.e., each feature representation of a data item is a vector, and representation for a plurality of features of the data item is a tensor); features extracted from the input images can include one or more pixels, each having a respective intensity value);
for at least some network layers of the deep learning model, training a set of tree models, where a given tree model corresponds to an individual network layer and creating an associated training data for each of the network layers (Nunes Coelho, ¶¶ [0034]-[0035]: replacing one or more layers of a neural network by respective decision trees; replacing one or more groups of network layers of a neural network by one or more decision trees to reduce the computational cost and improve efficiency; ¶ [0037] with FIG. 1: a training engine 140 configured for training the new machine learning model using the training data 150, wherein the new machine learning model 130 is a hybrid of the original input neural network model with one or more neural network layers replaced by one or more decision trees; ¶¶ [0041]-[0042] with FIG. 1: select one or more groups of layers of the neural network, and replace each group of layers with a respective decision tree to output a new machine learning model 130; each decision tree that replaces a respective group of layers; ¶ [0045] with FIG. 1: quantize at least a portion of the training data 150, and use the quantized training data to train respective decision trees or at least a portion of the new machine learning model 130, or both; ¶ [0047] with FIG. 1: provide the new machine learning model 130 to the training engine 140 which can then train at least a portion of the new machine learning model 130 based on the training data 150 used for training the original neural network 110; train the remaining layers of the neural network that have not been replaced by decision trees for a time period using the training data 150 (i.e., train the layers of the neural network that have been replaced by decision trees using the quantized training data and train the remaining layers of the neural network that have not been replaced by decision trees using original training data 150); alternatively or in addition, train the entire new machine learning model 130 based on quantized training data, assuming a respective gradient for each decision tree; ¶¶ [0049]-[0051] with FIG. 1: train the decision trees replacing one or more layers of the original neural network; train the decision trees before replacing the network layers of the original neural network with the decision trees; fine-tune the decision trees in the new machine learning model 130; to train the decision trees, use as training data corresponding portions of the same but quantized training data 150 for training the original neural network; the quantized version of layer inputs and outputs are associated with layer inputs and outputs for corresponding training data 150 that has been used for training the group of layers in the original neural network; ¶ [0054]: one or more layers of the neural network layers have been replaced by shallow decision trees with fewer nodes; ¶ [0065] with FIG. 2A: generate a new (or a new portion of) machine learning model 295 by replacing the group of network layers 290 of the original portion of neural network 200 by a decision tree 220, which has a tree depth based as least on the number of layers in the group of network layers 290; ¶ [0068] with FIG. 2A: before replacing the group of network layers 290 by a decision tree 220, train the decision tree 220 using quantized version of corresponding portions of training examples; set the quantized version of inputs as the inputs to the decision tree 220, set the quantized version of outputs as the outputs from the decision tree 220, and train the decision 220 using the quantized version of inputs and outputs; ¶¶ [0070]-[0071] with FIG. 2B: replace a different group of layers 285 in a portion of a neural network 250 by a different decision tree 270, and quantize the inputs and outputs for the decision tree using ternary quantization; the number of group of network layers to be replaced by a decision tree can be any suitable value determined by the system 100; e.g., the number can be one, ten, or fifty; ¶¶ [0074]-[0075] with 320-330 in FIG. 3: select one or more groups of layers from the plurality of layers, each group of layers comprising one or more layers adjacent to each other in the sequence 320; generates a new machine learning model that corresponds to the neural network by replacing each of the selected one or more groups of layers by a respective decision tree 330 (i.e., when each selected group of layers comprising one layer and replace each selected group of layers with a respective decision tree to output a new machine learning model, a given decision tree is corresponding to or associated with an individual network layer); ¶¶ [0078]-[0081]: for each of the one or more groups of layers, the respective decision tree receives as input a quantized version of the inputs to a respective first layer in the group and generates as output a quantized version of the outputs of a respective last layer in the group; obtain quantized versions of inputs and outputs to respective decision trees based on respective scaling factors; generate a quantized version of inputs to a decision tree by multiplying the quantized inputs with a respective scaling factor; a scaling factor is obtained based at least on a measure of similarity between the quantized layer and the original layer; train at least a portion of layers in the original neural network that were not replaced by respective decision trees; train the layers of the neural network that succeed the one or more group of layers of the neural network in the sequence; train the entire new machine learning model using the same but quantized training samples for the original neural network; use the quantized inputs and outputs for the forward propagation during training, and floating-type inputs and outputs for updating weights during backward propagation; ¶ [0091]: train the rest of layers starting from the last layer of the group to the last layer of the neural network; fine tunes the weights of the rest of layers using corresponding portions of the same training examples used for training the original neural network);
determining when any of the network layers require performance improvements (Nunes Coelho, ¶¶ [0015]-[0018]: reduce the computational cost and increase efficiency for performing inference computations for a large neural network; replacing one or more network layers of the large neural network by decision trees can reduce the amount of operations when a system performs inference computations for the neural network; error introduced by quantizing inputs and outputs to layers replaced by decision trees, is minimal compared to the efficiency gains; ¶¶ [0034]-[0035]: efficiently perform one or more inference computations for a large neural network using quantization and decision trees; by replacing one or more groups of network layers of a neural network by one or more decision trees to reduce the computational cost and improve efficiency for performing inference computations using economic hardware; ¶ [0043]: automatically determine a respective decision tree to replace a group of layers; ¶ [0056] with FIGS. 1 and 2A: a portion of an original neural network 200 represented by input dada 110 includes a plurality of network layers which can include a group of network layers 290 determined by the system 100 to be replaced by a corresponding decision tree; ¶¶ [0074]-[0075] with 320-330 in FIG. 3: select one or more groups of layers from the plurality of layers, each group of layers comprising one or more layers adjacent to each other in the sequence 320; generates a new machine learning model that corresponds to the neural network by replacing each of the selected one or more groups of layers by a respective decision tree 330; ¶¶ [0082]-[0090]: choose any suitable algorithm to select a group of layers and replace the group of layers by a respective decision tree; update the layer information for the group of layers and measures a performance (e.g., inference accuracy) of the modified network; after determining the group of layers, replace the group of layers with a respective decision tree; ¶¶ [0092]-[0096] with FIG. 4: select a respective initial layer in the neural network 410; select an initial layer for a group of layers randomly or based on the layer sequence; generates a respective plurality of candidate groups that each has the respective initial layer as the first layer in the candidate group 420; the candidate groups can have consecutive numbers of layers; for each of the respective plurality of candidate groups, determine a respective performance measure for the candidate group that measures a performance of a corresponding new machine learning model that has the layers in the candidate group replaced by a respective decision tree 430; generates a plurality of new machine learning models, each of which including a respective candidate group of layers replaced by a respective decision tree; perform inference computations for each of the new machine learning models using the same input data, and obtains a respective performance score for each of the new machine learning models; the respective performances cores can be obtained based on inference accuracy; select, as the group of layers to be replaced by a decision tree, one of the candidate groups of layers based on respective performance measures for the respective plurality of candidate groups 440; determine a maximum performance measure among the respective performance measures; and select, as the group, a candidate group associated with the maximum performance measure from the respective plurality of candidate groups; alternatively, select a candidate group which has a relatively high performance score but is the fastest for performing inference computations);
providing information relating to training the set of tree models to each layer of network determined to require an improvement in performance by providing and storing an input vector and generating a related output vector (Nunes Coelho, ¶ [0045] with FIG. 1: quantize at least a portion of the training data 150, and use the quantized training data to train respective decision trees or at least a portion of the new machine learning model 130, or both; ¶ [0047] with FIG. 1: provide the new machine learning model 130 to the training engine 140 which can then train at least a portion of the new machine learning model 130 based on the training data 150 used for training the original neural network 110; train the remaining layers of the neural network that have not been replaced by decision trees for a time period using the training data 150 (i.e., train the layers of the neural network that have been replaced by decision trees using the quantized training data and train the remaining layers of the neural network that have not been replaced by decision trees using original training data 150); alternatively or in addition, train the entire new machine learning model 130 based on quantized training data, assuming a respective gradient for each decision tree; ¶ [0051] with FIG. 1: to train the decision trees, use as training data corresponding portions of the same but quantized training data 150 for training the original neural network; the quantized version of layer inputs and outputs are associated with layer inputs and outputs for corresponding training data 150 that has been used for training the group of layers in the original neural network; ¶ [0068] with FIG. 2A: before replacing the group of network layers 290 by a decision tree 220, train the decision tree 220 using quantized version of corresponding portions of training examples; set the quantized version of inputs as the inputs to the decision tree 220, set the quantized version of outputs as the outputs from the decision tree 220, and train the decision 220 using the quantized version of inputs and outputs; ¶ [0078]: for each of the one or more groups of layers, the respective decision tree receives as input a quantized version of the inputs to a respective first layer in the group and generates as output a quantized version of the outputs of a respective last layer in the group; ¶¶ [0078]-[0081]: for each of the one or more groups of layers, the respective decision tree receives as input a quantized version of the inputs to a respective first layer in the group and generates as output a quantized version of the outputs of a respective last layer in the group; obtain quantized versions of inputs and outputs to respective decision trees based on respective scaling factors; generate a quantized version of inputs to a decision tree by multiplying the quantized inputs with a respective scaling factor; a scaling factor is obtained based at least on a measure of similarity between the quantized layer and the original layer; train at least a portion of layers in the original neural network that were not replaced by respective decision trees; train the layers of the neural network that succeed the one or more group of layers of the neural network in the sequence; train the entire new machine learning model using the same but quantized training samples for the original neural network; use the quantized inputs and outputs for the forward propagation during training, and floating-type inputs and outputs for updating weights during backward propagation; ¶ [0091]: train the rest of layers starting from the last layer of the group to the last layer of the neural network; fine tunes the weights of the rest of layers using corresponding portions of the same training examples used for training the original neural network); and
generating a set of experimental deep learning models, using said related output vector, with each experimental deep learning model corresponding to a copy of the deep learning model having a different network layer replaced with a tree model from the set of tree models (Nunes Coelho, ¶¶ [0092]-[0096] with FIG. 4: select a respective initial layer in the neural network 410; select an initial layer for a group of layers randomly or based on the layer sequence; generates a respective plurality of candidate groups that each has the respective initial layer as the first layer in the candidate group 420; the candidate groups can have consecutive numbers of layers; for each of the respective plurality of candidate groups, determine a respective performance measure for the candidate group that measures a performance of a corresponding new machine learning model that has the layers in the candidate group replaced by a respective decision tree 430; generates a plurality of new machine learning models, each of which including a respective candidate group of layers replaced by a respective decision tree; perform inference computations for each of the new machine learning models using the same input data, and obtains a respective performance score for each of the new machine learning models; the respective performances cores can be obtained based on inference accuracy; select, as the group of layers to be replaced by a decision tree, one of the candidate groups of layers based on respective performance measures for the respective plurality of candidate groups 440; determine a maximum performance measure among the respective performance measures; and select, as the group, a candidate group associated with the maximum performance measure from the respective plurality of candidate groups; alternatively, select a candidate group which has a relatively high performance score but is the fastest for performing inference computations).
Nunes Coelho further discloses a computer program product (CPP) comprising: a machine readable storage device (Nunes Coelho, ¶¶ [0037]and [0042] with 160 in FIG. 1: a memory 160 configured to store and provide data (e.g., training and output data for a machine learning model, and data defining a machine learning model) for the training engine 140; each decision tree that replaces a respective group of layers can be stored in memory 160 and is accessible for the modification engine 120; ¶ [0156] with FIG. 7: the programmable core 700 can include a plurality of computing components, e.g., MUX units, ALU units, and static random-access memory (SRAMs)); and computer code stored on the machine readable storage device (Nunes Coelho, ¶ [0124]: one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus; ¶ [0131]: computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices), with the computer code including instructions for causing a processor(s) set (Nunes Coelho, ¶ [0053]: central processing units (CPUs), graphical processing units (GPUs), tensor processing units (TPUs), or any other computing units suitable for performing operations of a neural network; ¶ [0125]: the term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers; the apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit); ¶ [0130]: computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit) to perform operations described above (Nunes Coelho, ¶¶ [0124]-[0125]: one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus; ¶¶ [0129]-[0130]: one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output; a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data)
Nunes Coelho also discloses a computer system (CS) comprising: a processor(s) set (Nunes Coelho, ¶ [0125]: the term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers; the apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit); ¶ [0130]: computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit); a machine readable storage device (Nunes Coelho, ¶¶ [0037]and [0042] with 160 in FIG. 1: a memory 160 configured to store and provide data (e.g., training and output data for a machine learning model, and data defining a machine learning model) for the training engine 140; each decision tree that replaces a respective group of layers can be stored in memory 160 and is accessible for the modification engine 120; ¶ [0156] with FIG. 7: the programmable core 700 can include a plurality of computing components, e.g., MUX units, ALU units, and static random-access memory (SRAMs)); and computer code stored on the machine readable storage device (Nunes Coelho, ¶ [0124]: one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus; ¶ [0131]: computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices), with the computer code including instructions for causing the processor(s) set to perform operations described above (Nunes Coelho, ¶¶ [0124]-[0125]: one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus; ¶¶ [0129]-[0130]: one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output; a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data).
Nunes Coelho fails to explicitly disclose receiving training labels.
Wan teaches a system and a method relating to neural networks and decision trees (Wan, ABSTRACT I Page 1), wherein receiving training labels (Wan, ABSTRACT I Page 1: jointly improving accuracy and interpretability using Neural-Backed Decision Trees (NBDTs); NBDTs replace a neural network’s final linear layer with a differentiable sequence of decisions and a surrogate loss; this forces the model to learn high-level concepts and lessens reliance on highly uncertain decisions, yielding (1) accuracy of NBDTs to match or outperform modern neural networks and better generalize to unseen classes by up to 16%, and furthermore, the surrogate loss improves the original model’s accuracy by up to 2%; NBDTs also afford (2) interpretability for improving human trust by clearly identifying model mistakes and assisting in dataset debugging; Section I in Pages 1-2: propose Neural-Backed Decision Trees (NBDTs) to jointly improve both (1) accuracy and (2) interpretability of modern neural networks, utilizing decision rules that preserve (3) properties like sequential, discrete decisions; pure leaves; and non-ensembled prediction; NBDTs replace the final linear layer of a neural network with a differentiable oblique decision tree and, unlike its predecessors (i.e. decision trees, hierarchical classifiers), uses a hierarchy derived from model parameters, does not employ a hierarchical softmax, and can be created from any existing classification neural network without architectural modifications; these improvements tailor the hierarchy to the network rather than overfit to the feature space, lessens the decision tree’s reliance on highly uncertain decisions, and encourages accurate recognition of high-level concepts; propose a tree supervision loss, yielding NBDTs that match/outperform and out-generalize modern neural networks; propose alternative hierarchies for oblique decision trees – induced hierarchies built using pre-trained neural network weights – that outperform both data-based hierarchies (e.g. built with information gain) and existing hierarchies (e.g. WordNet), in accuracy; show NBDT explanations are more helpful to the user when identifying model mistakes, preferred when using the model to assist in challenging classification tasks, and can be used to identify ambiguous ImageNet labels; Section 3.3 in Page 4 with FIG. 14 in Page 19: Labeling Decision Nodes with WordNet; to assign WordNet meaning to nodes, compute the earliest common ancestor for all leaves in a subtree; e.g., say Dog and Cat are two leaves that share a parent; to find WordNet meaning for the parent, find all ancestor concepts that Dog and Cat share, like Mammal, Animal, and Living Thing; the earliest shared ancestor is Mammal, so we assign Mammal to the parent of Dog and Cat.; use WordNet to assign meaning to intermediate decision nodes; Section 3.4 in Pages 4-5 with Tables 3 and 6 in Page 7: even though standard cross entropy loss separates representatives for each leaf, it is not trained to separate representatives for each inner node (Table 3, “None”); to amend this, add a tree supervision loss, a cross entropy loss over the class distribution of path probabilities with time-varying weights
β
t
,
ω
t
where t is the epoch count:
L
=
β
t
L
original
+
ω
t
L
soft
=
β
t
CrossEntropy
D
pred
,
D
label
+
ω
t
CrossEntropy
D
nbdt
,
D
label
; the tree supervision loss
L
soft
requires a pre-defined hierarchy; fine-tuning with
L
soft
only when the original model accuracy is not reproducible; unlike hierarchical softmax, the path-probability cross entropy loss
L
soft
disproportionately upweights decisions earlier in the hierarchy, encouraging accurate high-level decisions; this is reflected our out-generalization of the baseline neural network by up to 16% to unseen classes (Table 6); Section 5.4 with FIGS. 5-6 in Pages 8-9: there are several types of ambiguous labels (Figure 5), any of which could hurt model performance for an image classification dataset like ImageNet; to find these images, use entropy in NBDT decisions, which is a much stronger indicator of ambiguity than entropy in the original neural network prediction; if all intermediate decisions have high certainty except for a few decisions, those decisions are deciding between multiple equally plausible cases; using this intuition, one can identify ambiguous labels by finding samples with high "path entropy" – or highly disparate entropies for intermediate decisions on the NBDT prediction path; Per Figure 6, the highest "path entropy" samples in ImageNet contain multiple objects, where each object could plausibly be used for the image class; in contrast, samples that induce the highest entropy in the baseline neural network do not suggest ambiguous labels, which suggests NBDT entropy is more informative compared to that of a standard neural network).
Nunes Coelho and Wan are analogous art because they are from the same field of endeavor, a system and a method relating to neural networks and decision trees. It is also well-known in the art that training labels are provided in supervised learning for evaluate loss function. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Wan to Nunes Coelho. Motivation for doing so would improves the original model’s .
Claims 2, 8, and 14
Nunes Coelho in view of Wan discloses all the elements as stated in Claims 1, 7, and 13 respectively and further discloses ranking the experimental deep learning model of the set of experimental deep learning models based, at least in part, on an accuracy metric (Nunes Coelho, ¶¶ [0092]-[0096] with FIG. 4: select a respective initial layer in the neural network 410; select an initial layer for a group of layers randomly or based on the layer sequence; generates a respective plurality of candidate groups that each has the respective initial layer as the first layer in the candidate group 420; the candidate groups can have consecutive numbers of layers; for each of the respective plurality of candidate groups, determine a respective performance measure for the candidate group that measures a performance of a corresponding new machine learning model that has the layers in the candidate group replaced by a respective decision tree 430; generates a plurality of new machine learning models, each of which including a respective candidate group of layers replaced by a respective decision tree; perform inference computations for each of the new machine learning models using the same input data, and obtains a respective performance score for each of the new machine learning models; the respective performances cores can be obtained based on inference accuracy; select, as the group of layers to be replaced by a decision tree, one of the candidate groups of layers based on respective performance measures for the respective plurality of candidate groups 440; determine a maximum performance measure among the respective performance measures; and select, as the group, a candidate group associated with the maximum performance measure from the respective plurality of candidate groups; alternatively, select a candidate group which has a relatively high performance score but is the fastest for performing inference computations; i.e., ranking is performed in order to determine which candidate has a maximum performance measure).
Claims 3, 9, and 15
Nunes Coelho in view of Wan discloses all the elements as stated in Claims 2, 8, and 14 respectively and further discloses wherein the accuracy metric is determined by comparing output of the experimental deep learning models with output of the deep learning model using the training input datasets and the training labels (Nunes Coelho, ¶¶ [0092]-[0096] with FIG. 4: select a respective initial layer in the neural network 410; select an initial layer for a group of layers randomly or based on the layer sequence; generates a respective plurality of candidate groups that each has the respective initial layer as the first layer in the candidate group 420; the candidate groups can have consecutive numbers of layers; for each of the respective plurality of candidate groups, determine a respective performance measure for the candidate group that measures a performance of a corresponding new machine learning model that has the layers in the candidate group replaced by a respective decision tree 430; generates a plurality of new machine learning models, each of which including a respective candidate group of layers replaced by a respective decision tree; perform inference computations for each of the new machine learning models using the same input data, and obtains a respective performance score for each of the new machine learning models; the respective performances cores can be obtained based on inference accuracy; select, as the group of layers to be replaced by a decision tree, one of the candidate groups of layers based on respective performance measures for the respective plurality of candidate groups 440; determine a maximum performance measure among the respective performance measures; and select, as the group, a candidate group associated with the maximum performance measure from the respective plurality of candidate groups; alternatively, select a candidate group which has a relatively high performance score but is the fastest for performing inference computations; i.e., ranking is performed in order to determine which candidate has a maximum performance measure) (Wan, Section 3.4 in Pages 4-5 with Tables 3 and 6 in Page 7: even though standard cross entropy loss separates representatives for each leaf, it is not trained to separate representatives for each inner node (Table 3, “None”); to amend this, add a tree supervision loss, a cross entropy loss over the class distribution of path probabilities with time-varying weights
β
t
,
ω
t
where t is the epoch count:
L
=
β
t
L
original
+
ω
t
L
soft
=
β
t
CrossEntropy
D
pred
,
D
label
+
ω
t
CrossEntropy
D
nbdt
,
D
label
; the tree supervision loss
L
soft
requires a pre-defined hierarchy; fine-tuning with
L
soft
only when the original model accuracy is not reproducible; unlike hierarchical softmax, the path-probability cross entropy loss
L
soft
disproportionately upweights decisions earlier in the hierarchy, encouraging accurate high-level decisions; this is reflected our out-generalization of the baseline neural network by up to 16% to unseen classes (Table 6); ∆ is the accuracy difference between our soft loss and hierarchical softmax).
Claims 4, 10, and 16
Nunes Coelho in view of Wan discloses all the elements as stated in Claims 2, 8, and 14 respectively and further discloses generating a modified deep learning model with at least one network layer replaced by a tree model from the set of tree models (Nunes Coelho, ¶¶ [0034]-[0035]: replacing one or more layers of a neural network by respective decision trees; replacing one or more groups of network layers of a neural network by one or more decision trees to reduce the computational cost and improve efficiency; ¶ [0037] with FIG. 1: the new machine learning model 130 is a hybrid of the original input neural network model with one or more neural network layers replaced by one or more decision trees; ¶¶ [0041]-[0042] with FIG. 1: select one or more groups of layers of the neural network, and replace each group of layers with a respective decision tree to output a new machine learning model 130; each decision tree that replaces a respective group of layers; ¶ [0054]: one or more layers of the neural network layers have been replaced by shallow decision trees with fewer nodes; ¶ [0065] with FIG. 2A: generate a new (or a new portion of) machine learning model 295 by replacing the group of network layers 290 of the original portion of neural network 200 by a decision tree 220, which has a tree depth based as least on the number of layers in the group of network layers 290; ¶¶ [0070]-[0071] with FIG. 2B: replace a different group of layers 285 in a portion of a neural network 250 by a different decision tree 270; the number of group of network layers to be replaced by a decision tree can be any suitable value determined by the system 100; e.g., the number can be one, ten, or fifty; ¶¶ [0074]-[0075] with 320-330 in FIG. 3: select one or more groups of layers from the plurality of layers, each group of layers comprising one or more layers adjacent to each other in the sequence 320; generate a new machine learning model that corresponds to the neural network by replacing each of the selected one or more groups of layers by a respective decision tree 330 (i.e., when each group of layers comprising one layer and replace each group of layers with a respective decision tree to output a new machine learning model, a given decision tree is corresponding to or associated with an individual network layer); ¶¶ [0095]-[0096] with FIG. 4: for each of the respective plurality of candidate groups, determine a respective performance measure for the candidate group that measures a performance of a corresponding new machine learning model that has the layers in the candidate group replaced by a respective decision tree 430; generates a plurality of new machine learning models, each of which including a respective candidate group of layers replaced by a respective decision tree; perform inference computations for each of the new machine learning models using the same input data, and obtains a respective performance score for each of the new machine learning models; the respective performances cores can be obtained based on inference accuracy; select, as the group of layers to be replaced by a decision tree, one of the candidate groups of layers based on respective performance measures for the respective plurality of candidate groups 440; determine a maximum performance measure among the respective performance measures; and select, as the group, a candidate group associated with the maximum performance measure from the respective plurality of candidate groups; alternatively, select a candidate group which has a relatively high performance score but is the fastest for performing inference computations).
Claims 5, 11, and 17
Nunes Coelho in view of Wan discloses all the elements as stated in Claims 4, 10, and 16 respectively and further discloses wherein replacing the at least one network layer is based, at least in part, on the ranking of experimental deep learning models, where experimental deep learning models corresponding to high accuracy metrics indicate which network layers of the deep learning model are replaceable by tree models determined not to be affected by output accuracy of the modified deep learning model relative to the deep learning model (Nunes Coelho, ¶¶ [0015]-[0018]: reduce the computational cost and increase efficiency for performing inference computations for a large neural network; replacing one or more network layers of the large neural network by decision trees can reduce the amount of operations when a system performs inference computations for the neural network; error introduced by quantizing inputs and outputs to layers replaced by decision trees, is minimal compared to the efficiency gains; ¶¶ [0095]-[0096] with FIG. 4: for each of the respective plurality of candidate groups, determine a respective performance measure for the candidate group that measures a performance of a corresponding new machine learning model that has the layers in the candidate group replaced by a respective decision tree 430; generates a plurality of new machine learning models, each of which including a respective candidate group of layers replaced by a respective decision tree; perform inference computations for each of the new machine learning models using the same input data, and obtains a respective performance score for each of the new machine learning models; the respective performances cores can be obtained based on inference accuracy; select, as the group of layers to be replaced by a decision tree, one of the candidate groups of layers based on respective performance measures for the respective plurality of candidate groups 440; determine a maximum performance measure among the respective performance measures; and select, as the group, a candidate group associated with the maximum performance measure from the respective plurality of candidate groups; alternatively, select a candidate group which has a relatively high performance score but is the fastest for performing inference computations).
Claims 6, 12, and 18
Nunes Coelho in view of Wan discloses all the elements as stated in Claims 4, 10, and 16 respectively and further discloses outputting the modified deep learning model to a CPU reliant environment (Nunes Coelho, ¶¶ [0053]-[0054] and [0106] with FIG. 7: the training engine 140 used for training at least a portion of the new machine learning models 130 can include central processing units (CPUs); computing units such as MUX units or arithmetic logic (ALC) units can be used to perform operations in decision trees, a neural network inference engine or system can replace large, expensive computing units such as TPUs or GPUs with smaller and cheaper programmable cores with one or more MUX units or ALC units to perform inference computations for decision trees of the new machine learning model 180; perform inference computations for a decision tree using a programmable core 700; the programmable core 700 can receive tree input 710 and generate an inference output 720 for the tree input; the programmable core 700 can include a plurality of computing components, e.g., MUX units, ALU units, and static random-access memory (SRAMs); the programmable core 700 can perform nodal operations by merely add, select, and switch functions configured in the computing components; the programmable core 700 therefore does not need to include any MAC units for performing multiplication and additions as nodal operations for generating inference output for a decision tree).
Response to Arguments
Applicant's arguments filed on 07/29/2025 have been fully considered but they are not persuasive.
Applicant comments on Pages 8-10 of the Remarks regarding to FIG. 7 that "Input 702 is manipulated and information is then replaced with the information relating to the network layer 704. The drawing is reflecting that information is being replaced. Therefore, it is more appropriate to amend the specification relating to paragraph [0060] rather than changing the drawings. The deep forest generated layer is in fact replacing network layer 704 in some sense. The specification, rather than the drawings, is now amended as follows as indicated earlier: FIG. 7 shows an example neural network model 700, experimented with replacing some network layers of a neural network model with deep forest tree, including: (i) input 702; (ii) network layer 1 704; (iii) deep forest 706; (iv) network layer 2 708; (v) network layer 3 710; (vi) network layer n 712; and (vii) output 714. In this example experiment, information relating to network layer 1 704 is replaced with deep forest tree layers (generated) 1 706 … Subsequent experiments are then executed, with each experiment corresponding to replacing information relating to a different layer of neural network 700 with a corresponding deep forest layer (information), with their resulting outputs recorded and evaluated as in the first experiment …".
In response, although FIG.7 is a general representation of replacing information related to any of network layers (e.g., 704, 708, 710, 712) via Input 702 in some sense, however, current amended specification "FIG. 7 … In this example experiment …" still indicates that this example experiment "information relating to network layer 1 704 is replaced with deep forest tree layers (generated) 1 706" is the example experiment shown in FIG. 7 which is inconsistent with a general representation of replacing information related to any of network layers in FIG. 7. Therefore, to resolve the issue, Examiner suggested to change "this example experiment " to "first example experiment " because later in the paragraph describes "Subsequent experiments are then executed, with each experiment corresponding to replacing information relating to a different layer of neural network 700 with a corresponding deep forest layer (information), with their resulting outputs recorded and evaluated as in the first experiment".
Applicant argues on Pages 10-17 of the Remarks regarding 101 Rejections that claimed invention integrate "additional elements" with any "judicial exception" elements into a practical application to resolve the issues with poor performance of the deep learning model (especially in the CPU environment) (see ¶¶ [0051]-[0052] in the specification of the present invention).
In response, examiner respectfully disagrees. In order for a claim recites any "judicial exception" elements without subject to "judicial exception", "additional elements" (i.e., non-judicial exception elements) must be integrated with other "judicial exception" elements in a meaningful way (i.e., not just "apply it") so that the improvement of a practical application indicated in the specification (e.g., ¶¶ [0051]-[0052] in the specification of the present invention) is reflected in the claim as a whole. However, in the independent claims, the improvement of a practical application indicated in ¶¶ [0051]-[0052] of the specification is not reflected in the claim because the claim stop at experimental stage (i.e., generate a set of experimental deep learning models with each experimental deep learning model corresponding to a copy of the deep learning model having a different network layer replaced with a tree model from the set of tree models which is either mental process or common human activities for performing experimental testing) and no indication how to utilize these experimental deep learning models in a practical application. In order for the improvement of a practical application indicated in ¶¶ [0051]-[0052] of the specification to be reflected in the claim, a modified deep learning model with at least one network layer replaced by a tree model from the set of tree models must be generated and deployed in a CPU environment. Since "deploy the modified deep learning model in a CPU environment" is an "additional element" which integrates with other "judicial exception" elements in a meaningful way (e.g., generating a modified deep learning model with at least one network layer replaced by a tree model from the set of tree models based on the ranking in the set of experimental deep learning models generated earlier in the claim) so that the improvement of a practical application indicated in ¶¶ [0051]-[0052] of the specification is reflected in the claim as a whole. Therefore, Claims 6, 12, and 18 are not subject to "judicial exception".
Applicant further argues on Pages 17-18 of the Remarks regarding 103 Rejections that (1) "Nunes does not apply because while it seemingly provides a decision tree, it concentrates on a quantized version of the inputs to a respective first layer in the group and generates as output a quantized version of the outputs of a respective last layer in the group. This is not the problem or the solution of the present amended claims." and (2) "Wan only provides the basic teachings about machine learning applications, especially in specialized fields like finance and medicine. It uses decision trees to replace a neural network's final linear layer with a differentiable sequence of decisions and a surrogate loss. This is very different than the amended claims".
In response, examiner respectfully disagrees. Nunes Coelho is not only discloses "a quantized version of the inputs" to "a tree model" used to replace a group of network layers in a neural network but also teaches in ¶¶ [0041]-[0042] with FIG, 1 and ¶¶ [0074]-[0075] with FIG. 3 that (1) select one or more groups of layers of the neural network, and replace each group of layers with a respective decision tree to output a new machine learning model 130; (2) each decision tree that replaces a respective group of layers; (3) select one or more groups of layers from the plurality of layers, each group of layers comprising one or more layers adjacent to each other in the sequence 320; and (4) generate a new machine learning model that corresponds to the neural network by replacing each of the selected one or more groups of layers by a respective decision tree 330. In other words, when each selected group of layers comprising one layer and replace each selected group of layers with a respective decision tree to output a new machine learning model, a given decision tree is corresponding to or associated with an individual network layer as recited. Nunes Coelho further discloses in ¶¶ [0015]-[0018], [0053]-[0054], and [0095]-[0096] with FIG. 4 that the reasons for replacing each group of layers with a respective decision tree to output a new machine learning model are to reduce the computational cost and increase efficiency for performing inference computations for a large neural network with minimum error impacts in non-GPU/TPU environment (e.g., CPU environment). Therefore, Nunes Coelho teaches all the limitations as recited in independent Claims 1, 7, and 13 except "receiving training labels". Wan teaches this missing limitation by using "training labels" to calculate tree supervision loss and original loss during training when replacing a neural network's final linear layer with a differentiable oblique decision tree (Wan, Section 3.4 in Pages 4-5). Therefore, the combination of Wan and Nunes Coelho teaches all the limitations as recited in independent Claims 1, 7, and 13.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
SUN et al. (CN112085157A, pub. date: 12/15/2020) discloses in ¶¶ [0018]-[0019] that the neural network tree model is obtained by replacing the neuron nodes of the output layer and several neuron nodes of each hidden layer in the preset neural network model with the preset tree model, wherein the preset tree model includes a decision tree model, a GBDT model, an XGBoost model, and an AdaBoost model. SUN further discloses in ¶¶ [0058]-[0069] with FIGS. 2-3 that (1) the neuron nodes of the output layer and several neuron nodes of each hidden layer in the preset neural network model NN shown in STEP1 in Figure 2 and STEP1 in Figure 3 are replaced with the preset tree model Tree (the neuron nodes of the input layer do not need to be replaced), resulting in the neural network tree model (NNT) shown in STEP2 in Figure 2 and STEP2 in Figure 3, wherein the preset tree model includes decision tree model, gradient boosting decision tree (GBDT), extreme gradient boosting (XGBoost), and adaptive boosting (AdaBoost); (2) a neural network random forest model (NNRF) is constructed based on the neural network tree model (NNT), as shown in STEP3 in Figure 2 and STEP3 in Figure 3; (3) the neural network random forest model (NNRF) consists of n neural network tree models (NNT), where n is an integer greater than 0 and NNTn represents the nth neural network tree model (NNT); (4) outputs of the n neural network tree models (NNT) are combined using the voting/averaging method to obtain the output of the neural network random forest model (NNRF); (5) it should be noted that the preset tree models for replacing neuron nodes in the preset neural network model may be the same or different; and (6) similarly, the several neural network tree models in the neural network random forest model (NNRF) may be the same or different, and the selection should be made according to actual needs.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HWEI-MIN LU whose telephone number is (313)446-4913. The examiner can normally be reached Mon - Fri: 9:00 AM - 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela D. Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HWEI-MIN LU/Primary Examiner, Art Unit 2142
Prosecution Insights