Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendments filed 02/09/2026 have been entered. Claims 1, 5-10 remain pending in the application.
Applicant’s arguments and amendments, with respect to claim rejections of claims 1-10 under 35 U.S.C 101 filed 10/09/2025 have been considered and they are persuasive. Therefore, the previous rejections as set forth in the previous office action has been withdrawn.
Applicant’s arguments amendments, with respect to claim rejections of claims 1-10 under 35 U.S.C 103 filed 10/09/2025 have been considered and some of them are persuasive. However, upon further consideration, new ground(s) of rejections have been raised (See Below.)
The applicant argues that the Harmon in view of Pardo does not teach the claims as amended. Applicant argues that Harmon does not disclose the amended configuration in which the mixing coefficients are applied to the same input vector element before the activation functions are applied. According to applicant, Harmon teaches applying an input to multiple activation functions and then multiplying/summing the outputs of those activation function using coefficients, which is the reverse of amended claim 1. Applicant further argues that Pardo only teaches vector-based input/output processing and does not cure this deficiency, and that Apicella and Yaguchi were cited only dependent limitations, and likewise do not disclose or suggests the claimed ensemble activation function. Therefore, applicant contends that claim 1, and the dependent claims relying on claim 1, are patentable over the cited references.
Applicant’s arguments are acknowledged. However, Harmon still teaches or at least suggests significant portions of amended claim 1 because Harmon teaches an activation ensemble in which a plurality of activation functions is applied and the outputs of the activation functions are applied and the outputs of the activation functions are combined/weighted to produce an output value. This corresponds to the claimed ensemble activation function having a plurality of activation functions and a mixing function arranged to receive outputs of the activation functions. This interpretation is also consistent with applicant’s own specification, which explains that “different activation functions are applied to the same data” and that the output values of the activation functions are integrated “in accordance with the mixing coefficients” to output an integrated output value. See, e.g., specification [0030]-[0031]. The specification further states at [0033] “A mixing coefficient is a weighted parameter provided for each of the activation functions that are to be integrated” In the embodiment of paragraph [0036]-[0037], the input vector element is inserted into activation functions to generate output values, and mixing function receive those outputs values and integrates them in accordance with the mixing coefficient to obtain output value. Thus, Harmon’s disclosure of applying multiple activation functions and weighting/combining their activation outputs remains relevant to the claimed ensemble/mixing-function structure.
The examiner recognizes that amended claim now recites a coefficient-before-activation arrangement, i.e., calculating first output values by applying mixing coefficients to the same input vector element and then calculating second output values by applying activation functions to the first output values. This amended ordering corresponds to a modified example in applicant’s specification, rather than the earlier embodiment of paragraph [0030]-[0037]. Accordingly, Harmon is relied upon for the activation ensemble and mixing-function structure, while Faraone is relied upon for teaching or suggesting applying a coefficient/scaling value to a pre-activation value of a neural network before activation processing is performed.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5-7, 10 are rejected under 35 U.S.C. 103 as being unpatentable over Harmon et.al (NPL: Activation Ensembles for Deep Neural Networks) in view of Pardo et.al (US 20230110285 A1), further in view of Faraone et.al (US 11615300 B1)
Regarding claim 1,
Harmon teaches the limitation “inference apparatus comprising a processing circuit” (page 5 section 4 column 1 ”In addition, we use Theano and Titan X GPU’s for all experiments.” Harmon discloses performing the experiment which comprises of the neural network with ensemble layers through a device configured with a GPU, which is known by one of ordinary skilled the art as a graphical processing unit that comprise various processing circuit and a processor.)
Harmon teaches the limitation “obtain processing target data” (Page 3 section 3 column 1 “The first naive approach is to simply take the input, use a variety of activation functions,” and Page 5 section 4 column 1 “For our experiments, we use the datasets MNIST, ISOLET, CIFAR-100, and STL-10. For MNIST and ISOLET, we use small designed feed forward and convolutional neural networks”. Harmon discloses a method of implementing the activation function ensembles on a variety of datasets. Within the disclosure, Harmon discloses using datasets such as MNIST and ISOLET as input data for experiments, which analogous to the target data.)
Harmon teaches the limitation “calculate inference data by applying a trained neural network to the processing target data,” (Page 3 section 3 column 1 “Ensemble Layers were created with the idea of allowing a network to choose its own activation for each neuron and for each layer of the network. Overall, the network takes the output of a previous layer, for example from a convolutional step, applies its various activations”, Page 6 section 4.1 column 1 Figure 2 “Next we compare the functions that the ensembles create. Figure 2 depicts the various sets of activations for our ensemble. ... the final image is taken from the second layer of the same model trained on the ISOLET dataset”, and page 7 section 4.3 column 2 “For training and testing, we split the data into 70% for training and 30% for testing”. Harmon discloses a neural network model such as convolutional neural networks with layers that allow the network to choose an activation function for each neuron for each layer to calculate a value using the activation function. The neural network model is trained on example data set such as the ISOLET dataset, such that the dataset goes through the calculation of the activation function to obtain an output. Figure 2 represents the function values (calculated inference data) that are illustrated on the graph based on applying the activation functions from the neural network model. The ISOLET dataset suggest the processing target data, such that the dataset is split into training data set and testing dataset, and after the model is trained on the training dataset, the trained model is applied onto the testing dataset for testing.)
Harmon teaches a part of the limitation “wherein the trained neural network includes an ensemble activation function for each of a plurality of unit network structures ... and a mixing function arranged to receive outputs of the activation function” (Page 1 section 1 column 2 “Each activation function, from ReLU to hyperbolic tangent contain advantages in learning. We propose to use the best parts of each in a dynamic way decided by variables configuring contributions of each activation function. These variables put weights on each activation function under consideration and are optimized via backpropagation” and Page 3 section 3 column 1-2 “Ensemble Layers were created with the idea of allowing a network to choose its own activation for each neuron and for each layer of the network. Overall, the network takes the output of a previous layer, for example from a convolutional step, applies its various activations, normalizes these activations, and places weights on each activation function ...” Harmon discloses the neural network model includes ensemble layers that allow the network to choose its own activation for each neuron and for each layer of the network, which corresponds to the ensemble activation function for each of a plurality of unit network structures, as claimed. Harmon further discloses the various activations which corresponds to the claimed outputs of the activation function, and a weighted summation of each activation at function yi(z) at page 3, which corresponds to the claimed mixing function arrange to receive outputs of the activation function, because the summation receives the outputs of the plurality of activation functions and combines them using weights to produce a single output value for the neuron/layer.)
Harmon does not teach part of the limitation “... activation function ... configured to convert an input vector element to an output vector element”. However, Pardo teaches this limitation (paragraph 525 “In some embodiments, the approximation circuitry 3548 may receive an input vector 3920 at the input circuit(s) 3904 and then provide the input vector to the approximation circuit(s) 3908. The approximation circuit(s) 3908 may then apply an activation function to the input vector by performing a hardware approximation of the activation function in a vector manner. The output circuit(s) 3912 may then generate an output vector 3924 based on the activation function.” Harmon discloses a processing unit described as including circuitry to receive input vector and applies an activation function to generate an output vector. Within the disclosure, Pardo discloses the neural network pipeline as well as the architecture and hardware devices to perform the method.)
Before the effective filing date, it would have been obvious to one of ordinary skilled in the art to combine the teaching of activation ensembles for deep neural networks by Harmon with the teaching of vector activation function to support machine learning inference by Pardo. The motivation to do so is referred to in Pardo’s disclosure (paragraph 2 “Some activation functions are simple and can be performed by some vector primitives. Other activation functions are less simple. Improvements to processor architectures that implement these types of functions (e.g., non-linear functions, activation functions, etc.) can improve the overall performance of the processor from a computational efficiency standpoint, a reduced processing time standpoint, and/or a reduced energy consumption standpoint”, paragraph 451 “In at least one embodiment, hardware 3022 may include GPUs, CPUs, graphics cards, an AI/deep learning system (e.g., an AI supercomputer, such as NVIDIA's DGX supercomputer system), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 3022 may be used to provide efficient, purpose-built support for software 3018 and services 3020 in deployment system 3006. In at least one embodiment, use of GPU processing may be implemented for processing locally (e.g., at facility 3002), within an AI/deep learning system, in a cloud system, and/or in other processing components of deployment system 3006 to improve efficiency, accuracy, and efficacy of image processing, image reconstruction, segmentation, MM exams, stroke or heart attack detection (e.g., in real-time), image quality in rendering, etc”, and paragraph 522 “In some embodiments, the circuitry 3804 may facilitate a vector add instruction with a vector size of eight elements ... In some embodiments, the circuitry 3804, 3808 may be provided as part of the approximation circuitry 3548 to support performance of vector activation functions defined in the vector instruction list 3544. Because all operations can be performed on the data using the approximation circuitry 3548, it becomes possible to perform vector activation functions without requiring access to separate memory 3516, thereby increasing the speed of processing.” Pardo discloses the benefits of the invention, which provide improvement to processor architectures to handle less simple activation function and improve the overall performance of the processor from a computational efficiency standpoint. Pardo then discloses the hardware to carry out the neural network model with an activation function can be implemented using a GPU, which is similar to Harmon. Pardo then discloses the circuitry of hardware that support the input in vector format and the capability of the activation function to process the input vector. While Harmon introduces ensemble of activation functions through ensemble layers, wherein one of ordinary skilled in the art would recognize that an activation function can output vector data in response to given input vector data, Harmon does not disclose such method to receive input vector for activation function. Therefore, the teaching by Harmon can further incorporate the teaching by Pardo based on the similarity of the hardware configuration of GPU to carry out the activation functions while also obtain the improvement in application of input vector for processing and computational efficiency as disclosed by Pardo.)
Harmon/Pardo does not teach a part of the limitation “... the ensemble activation function being configured to apply mixing coefficients to the input vector element, and having a plurality of activation functions configured to apply activation processing to the input vector element to which the mixing coefficients are applied, respectively, ...” However, Faraone in view of Harmon teaches or at least suggest this part of the limitation (Column 2 lines 41-53 “a method includes providing an input layer and one or more hidden layers following the input layer; wherein a first layer of the one or more hidden layers includes a first weight space including one or more subgroups; receiving an input from a layer preceding the first layer; generating a first subgroup weighted sum using the input and a first plurality of weights associated with a first subgroup; providing a first scaling coefficient associated with the first subgroup; applying the first scaling coefficient to the first subgroup weighted sum to generate a first subgroup scaled weighted sum; and generating an activation based on the first subgroup scaled weighted sum and provide the activation to a layer following the first layer.” Faraone discloses applying a coefficient to a neural network value before activation processing. Specifically, Faraone teaches receiving an input, generating a first subgroup weighted sum using the input and a first plurality of weights associated with a first subgroup, applying the first scaling coefficient to the first subgroup weighted sum to generate a first subgroup scaled weighted sum, and then generating an activation based on the first subgroup scaled weighted sum. The first scaling coefficient corresponds to the claimed mixing coefficient because it is a coefficient applied to neural network value to scale/weight that value before activation processing. The first subgroup weighted sum corresponds to the claimed input vector element supplied to the activation function, and the first subgroup scaled weighted sum corresponds to the claimed first output value because it is generated by applying the scaling coefficient to that input. Thus, Faraone discloses teaches or at least suggests applying a coefficient to a pre-activation input value to generate a scaled value, and subsequently generating activation based on the scale value, which corresponds to the claimed arrangement in which a mixing coefficient is applied before activation processing is performed. The activation processing may be performed in view of the ensemble of activation function as disclosed by Harmon above, wherein the motivation to combine is disclosed below.)
Harmon/Pardo does not teach the limitation “wherein the ensemble activation function calculates a plurality of first output values by applying the mixing coefficients to the same input vector element, calculates a plurality of second output values by applying the activation functions to the first output values, and calculates the output vector element based on the second output values” However, Faraone in view of Harmon teaches or at least suggest this limitation (Column 2 lines 41-53 “a method includes providing an input layer and one or more hidden layers following the input layer; wherein a first layer of the one or more hidden layers includes a first weight space including one or more subgroups; receiving an input from a layer preceding the first layer; generating a first subgroup weighted sum using the input and a first plurality of weights associated with a first subgroup; providing a first scaling coefficient associated with the first subgroup; applying the first scaling coefficient to the first subgroup weighted sum to generate a first subgroup scaled weighted sum; and generating an activation based on the first subgroup scaled weighted sum and provide the activation to a layer following the first layer.” Faraone discloses applying a coefficient to a neural network value before activation processing. Specifically, Faraone teaches receiving an input, generating a first subgroup weighted sum using the input and a first plurality of weights associated with a first subgroup, applying the first scaling coefficient to the first subgroup weighted sum to generate a first subgroup scaled weighted sum, and then generating an activation based on the first subgroup scaled weighted sum. The first scaling coefficient corresponds to the claimed mixing coefficient because it is a coefficient applied to neural network value to scale/weight that value before activation processing. The first subgroup weighted sum corresponds to the claimed input vector element supplied to the activation function, and the first subgroup scaled weighted sum corresponds to the claimed first output value because it is generated by applying the scaling coefficient to that input. The activation based on the first subgroup scaled weighted sum corresponds to the second output values, as claimed, and the result activation provided to a layer following the first layer corresponds to the output vector element, as claimed. Thus, Faraone discloses teaches or at least suggests applying a coefficient to a pre-activation input value to generate a scaled value, and subsequently generating activation based on the scale value, which corresponds to the claimed arrangement in which a mixing coefficient is applied before activation processing is performed. The activation processing may be performed in view of the ensemble of activation function as disclosed by Harmon above, wherein the motivation to combine is disclosed below.)
Before the effective filing date, it would have been obvious to one of ordinary skilled in the art to combine the teaching of activation ensembles for deep neural networks by Harmon, and the teaching of vector activation function to support machine learning inference by Pardo, with the teaching of applying coefficient to input before the activation processing by Faraone. The motivation to do so is referred to in Faraone’s disclosure (Column 9, lines 11-26 “As shown in FIGS. 2 through 12, to improve accuracy without incurring much hardware costs, a method for neural network training and inference is described. During the neural network training, by using subgroups for weights determined based on locality of the weights, irregular (different) scaling coefficients are only used on one or two dimensions of all the dimensions (e.g., a total of four dimensions) for a weight space of a particular a layer. This achieves better prediction accuracy than having one scaling coefficient for all dimensions, without incurring much hardware costs. Further, it reduces the required hardware compared to a method that uses multiple scaling coefficients on each dimension. This method enables high data parallelism in hardware, and achieves high accuracy for both binary and ternary networks.” Faraone teaches that using coefficients in neural network training/inference improves accuracy while avoiding the increased hardware cost and complexity associated with applying multiple scaling coefficients across all weight-space dimensions. Thus, Faraone provides a reason to use scaling coefficients in neural network computations, namely to improve accuracy while maintaining efficient hardware implementation. In view of Faraone’s teaching, it would have been obvious to apply coefficient-based scaling to Harmon’s activation-ensemble branches before activation processing to provide additional control over activation inputs and improve accuracy, while reducing the hardware cost in implementing the machine learning process.)
Regarding claim 5 depends on claim 1, thus the rejection of claim 1 is incorporated.
Pardo teaches the limitation “a medical imaging apparatus configured to perform medical imaging upon a subject” (paragraph 458 “In at least one embodiment, system 3100 may include a multi-layer platform that may include a software layer (e.g., software 3018) of diagnostic applications (or other application types) that may perform one or more medical imaging and diagnostic functions.”, and paragraph 467 “In at least one embodiment, AI services 3118 may be leveraged to perform inferencing services for executing machine learning model(s) associated with applications (e.g., tasked with performing one or more processing tasks of an application). In at least one embodiment, AI services 3118 may leverage AI system 3124 to execute machine learning model(s) (e.g., neural networks, such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and/or other inferencing tasks.” Pardo discloses a platform with application to perform medical imaging (medical imaging apparatus), wherein the application may execute machine learning model for inference task such as object detection of medical image which is analogous to medical imaging upon a subject. The machine learning model may incorporate the neural network with ensemble layers technique as disclosed above.)
Pardo teaches the limitation “wherein the processing circuit obtains, as the processing target data, medical data obtained by the medical imaging apparatus” (paragraph 467 “In at least one embodiment, AI services 3118 may be leveraged to perform inferencing services for executing machine learning model(s) associated with applications (e.g., tasked with performing one or more processing tasks of an application). In at least one embodiment, AI services 3118 may leverage AI system 3124 to execute machine learning model(s) (e.g., neural networks, such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and/or other inferencing tasks” Pardo discloses inference data from the executing the machine learning model, wherein one of ordinary skilled in the art would recognize that the inference data can be medical image data from performing one or more medical imaging using the apparatus with neural network model and ensemble layers as mentioned above.)
Regarding claim 6, the applicant is directed to the rejection of claim 1 above, because the claim is rejected under the same rationale as the claim recites similar limitations and processing steps.
Regarding claim 7,
Harmon teaches a part of the limitation “obtaining training data that includes input training data ...” (Page 5 section 4 column 1 “For our experiments, we use the datasets MNIST, ISOLET, CIFAR-100, and STL-10. For MNIST and ISOLET, we use small designed feed forward and convolutional neural networks ... We train on our three activation sets ...” Harmon discloses using datasets such as MNIST, ISOLET, etc., as training data input to train the neural network model on activation layers.)
Pardo teaches a part of the limitation “obtaining ... output training data” (paragraph 440 “In at least one embodiment, imaging data 3008 generated by imaging device(s), sequencing devices, and/or other device types may be received. In at least one embodiment, once imaging data 3008 is received, AI-assisted annotation 3010 may be used to aid in generating annotations corresponding to imaging data 3008 to be used as ground truth data for a machine learning model.” Pardo discloses using imaging devices to generate imaging data, which can be processed by AI-assisted annotation to obtain ground truth data for a machine learning model, wherein the ground truth data is analogous to the output training data.)
Harmon teaches the limitation “the learning includes updating the training parameter and the mixing coefficients based on the input training data in such a manner as to minimize a discrepancy between the output data of the neural network and the output training data” (Page 4 section 3 column 2 “We next provide derivatives for the new parameters. Let l denote the cost function for our neural network. We backpropagate through our loss function
l
with the chain rule to find the following:
∂
l
l
α
j
…
∂
l
l
η
j
…
∂
l
l
δ
j
” and Page 8 section 4.5 column 1 “The reproduced images by the network are then very easy for comparison by using the sigmoid function and mean-squared error” Harmon discloses partial derivatives of the loss function
l
with respect to each parameter α, η and δ as demonstrated by the three expressions. One of ordinary skilled in the art would recognize that the back propagation step through the loss function to find these derivatives imply that these parameters are updated through back propagation. Harmon further discloses the loss function
l
which represent the loss in calculating output data and analogous to the discrepancy as understood by one of ordinary skilled in the art. while Harmon does not explicitly explain the loss function
l
to minimize the loss (discrepancy) between the output data of the neural network and the output training data, Harmon discloses in the experiment 4.5 that the reproduced images (output data of the neural network) are very easy to compared and using mean-squared error, which suggest the incorporation of a set of ground truth data (the ground truth data as disclosed by Pardo which is analogous to the output training data) for comparison as understood by one of ordinary skilled in the art when the prior art discloses using mean-squared error and comparison with reproduced images (output data of the neural network). The mean-squared error function would represent the loss function
l
that is not disclosed in prior section of Harmon.
The applicant is directed to the rejection of claim 1 above, because the claim is further rejected under the same rationale as the claim recites similar limitations and processing steps.
Regarding claim 10 depends on claim 7, thus the rejection of claim 7 is incorporated.
“The method for generating the trained neural network according to claim 7, wherein the learning includes training the neural network based on the training data, using a training parameter and a mixing coefficient of a trained neural network as initial values, the trained neural network being generated through training the neural network based on training data outside an objective.” (Page 5 section 4 column 1-2 “For our experiments, we use the datasets MNIST, ISOLET, CIFAR-100, and STL-10. For MNIST and ISOLET, we use small designed feed forward and convolutional neural networks ... We train on our three activation sets”, Page 2 section 1 column 1 “The second are a set of “offset” parameters, η and δ, which we use to dynamically offset normalization range for each function. Training of these new parameters occurs during typical model training and is done through backpropagation.”, and Page 1 section 1 column 2 “Our work can be seen as a layer or neuron ensemble that can be applied to any variety of deep neural network via activation ensembles. We do not focus on creating yet another unique activation function, but rather ensemble a number of proven activation functions in a compelling way. The end result is a novel activation function that is a combination of existing functions” Harmon discloses a training of the neural network model based on the training data as disclosed above in claim 7. Harmon also discloses additional “offset” parameters η and δ, which are used to dynamically offset normalization range for each function as well as the weight value α that is placed on each activation function, wherein these parameters are analogous to the training parameter and the mixing coefficient of the trained neural network. Harmon further discloses their work can be seen as a layer or neuron ensemble that can be applied to any variety of deep neural network via activation ensembles, suggesting that the activation ensembles can be applied to any neural network with any training data, which is demonstrated through the experiment process with various different datasets and analogous to the capability of the training the neural network on training data outside an objective.)
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Harmon et.al (NPL: Activation Ensembles for Deep Neural Networks) in view of Pardo et.al (US 20230110285 A1) further in view of Faraone et.al (US 11615300 B1), further in view of Apicella et.al (NPL: A survey on modern trainable activation functions)
Regarding claim 8 depends on claim 7, thus the rejection of claim 7 is incorporated.
Harmon/Pardo does not teach the limitation “neural network according to claim 7, wherein the learning includes determining mixing coefficients corresponding to the same activation function to have a same value in the ensemble activation functions included in the neural network”. However, Apicella teaches this limitation (Page 22 Figure 8 “Figure 8: Several trainable activation functions represented as feed forward layers with constrained weights. The connections having the same labels have shared weights, that is the same value. The connection having numeric values as labels must have the same fixed value during the training.” Apicella discloses a technique of weight sharing for multiple activation functions as demonstrated via Figure 8, such that several activation functions are represented as feed forward layers with constrained weights, and edges with similar label are constrained to use the same weight value, which corresponds to the mixing coefficients corresponding to the same activation function to have a same value in the claim. The shared weights are analogous to the mixing coefficients, and one of ordinary skilled in the art would have been able to configure a similar label for similar activation function.)
Before the effective filing date, it would have been obvious to one of ordinary skilled in the art to combine the teaching of activation ensembles for deep neural networks by Harmon, the teaching of vector activation function to support machine learning inference by Pardo, the teaching of applying coefficient to input before the activation processing by Faraone, with the teaching of a technique of weight sharing for multiple activation functions by Apicella. The motivation to do so is referred to in Apicella’s disclosure (page 15 section 4.2.1 “Harmon & Klabjan Activation Ensembles Some studies try to define activation functions using different available activation functions rather than creating an entirely new function. For example, in [Klabjan and Harmon, 2019] the authors allow the network to choose the best activation function from a predefined set ...”, and Page 22 Figure 8 “Figure 8: Several trainable activation functions represented as feed forward layers with constrained weights. The connections having the same labels have shared weights, that is the same value.” Apicella discloses a survey on modern trainable activation functions, which include a recitation of the technique of ensemble layers by Harmon above. Apicella later discloses the technique of shared weights for activation functions, wherein one of ordinary skilled in the art would recognize that the shared weights may be applied for the teaching by Harmon to optimize the ensemble layers as well as cut down the number of parameters the network needs to learn and less likely to overfit the training data.)
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Harmon et.al (NPL: Activation Ensembles for Deep Neural Networks) in view of Pardo et.al (US 20230110285 A1), further in view of Faraone et.al (US 11615300 B1), further in view of Yaguchi et.al (US 20200012945 A1).
Regarding claim 9 depends on claim 7, thus the rejection of claim 7 is incorporated.
Harmon/Pardo does not teach the limitation “The method for generating the trained neural network according to claim 7, wherein the learning includes adding a cost function to a loss function, where a value of the cost function decreases as a discrepancy between the mixing coefficients is reduced”. However, Yaguchi teaches this limitation (paragraph 43 “Further, the update unit 38 updates each of a plurality of weight coefficients included in the neural network 20 so as to minimize an objective function obtained by adding a basic loss function and an L2 regularization term multiplied by a regularization strength. ... The L2 regularization term is a sum of squares of all the weight coefficients. Further, the regularization strength is a non-negative value. The objective function obtained by adding the basic loss function and the L2 regularization term obtained by multiplying the regularization strength is also referred to as a cost function into which the L2 regularization term is introduced” Yaguchi discloses a learning method of optimizing a neural network, include updating a plurality of weight coefficients by adding a basic loss function and an L2 regularization term multiplied by a regularization strength. In particular, Yaguchi discloses the L2 regularization obtained by multiplying the regularization strength as the cost function in addition to the loss function. One of ordinary skilled in the art would recognize that applying the L2 cost function technique with regard to the weight coefficients would allow the decrease of the cost as the weight coefficients is reduced, otherwise known as ‘weight decay’ as large weight is penalized to encourage the model to learn smaller weight that capture the underlying patterns in the data.)
Before the effective filing date, it would have been obvious to one of ordinary skilled in the art to combine the teaching of activation ensembles for deep neural networks by Harmon, the teaching of vector activation function to support machine learning inference by Pardo, the teaching of applying coefficient to input before the activation processing by Faraone, with the teaching of a learning method that include updating a plurality of weight coefficients by adding a basic loss function and an L2 regularization term multiplied by a regularization strength by Yaguchi. The motivation to do so is referred to in Yaguchi’s disclosure (paragraph 20 “learning device 10 according to the present embodiment updates a plurality of weight coefficients included in a neural network 20 through a learning process. Accordingly, the learning device 10 can optimize the neural network 20 for a predetermined application and reduce the size of the neural network 20”, and paragraph 109 “The learning devices 10 according to the first to third embodiments solve the optimization problem for minimizing the objective function obtained by adding the basic loss function and the L2 regularization term multiplied by the regularization strength and optimize the neural network 20. Accordingly, the learning device 10 can optimize the neural network 20 for a predetermined application and reduce the size while suppressing the accuracy deterioration.” Yaguchi discloses the benefit of the method in providing a regularization term to optimize a neural network and reduce the size while suppressing the accuracy deterioration. Harmon mentioned above that using mean-squared error in one of the examples to minimize the loss, which corresponds to the loss function by Yaguchi. Yaguchi further introduces L2 regularization term as a sum of squares of all the weight coefficients, which is represented as the cost function in combination with the loss function to minimize an objective function. One of ordinary skilled would draw the connection to the weight parameter α of Harmon such that the cost function can penalize the large weight values and encourage the model to learn smaller weight that capture the underlying patterns in the data. Therefore, the teaching combination by Harmon/Pardo may further incorporate the teaching by Yaguchi for further improvement using L2 regularization term reduce the size.)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUY TU DIEP whose telephone number is (703)756-1738. The examiner can normally be reached M-F 8-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUY T DIEP/ Examiner, Art Unit 2123
/ALEXEY SHMATOV/ Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights