DETAILED ACTION
This action is in response to the original filing on April 20, 2023 and the Remarks and Amendments filed on June 1, 2026. Claims 1-25 are pending and have been considered below. Claims 1, 10, 18, 21, and 25 are independent claims. Claims 1-3, 5-19, 21, and 23-25 are amended.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
In review of Applicant’s amendments, filed June 1, 2026, the objections to the claims, specification, and drawing made in the previous office action have been withdrawn.
The rejections of claims 3, 5-6, 14, 16-17, and 24 under 35 U.S.C. 112(b) set forth in the previous office action are withdrawn in view of the amendments to the claims.
The rejections of claims 1-20 under 35 U.S.C. 101 set forth in the previous office action are withdrawn in view of the amendments made to independent claims 1, 10, 18, 21, and 25.
Response to Arguments
Applicant’s arguments, filed June 1, 2026 regarding the rejections from the previous office action made under 35 U.S.C. 103 have been fully considered but are moot as they do not apply to the reference Merity being used in the current rejections of claims 1, 10, 18, 21, and 25 and their associated dependent claims 2-9, 11-17, 19-20, and 22-24, respectively, to teach the amended claim limitation directed to the regularization range loss term.
Specifically, Merity teaches the amended claim limitation wherein loss processing comprises using a loss function that includes a regularization range loss term which causes model weights of the neural network to produce smaller activation values, or the “activation regularization (AR)” term of Merity (Page 2, Section 2.1).
With the addition of the reference Merity teaching the subject matter introduced in the amendments, the rejections under 35 U.S.C. 103 stand.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7, 10, 13-15, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over No et al. (US 20240211738 A1, hereinafter No) in view of Obla (“Effective Activation Functions for Homomorphic Evaluation of Deep Neural Networks,” hereinafter Obla) and further in view of Merity et al. (“Revisiting Activation Regularization for Language RNNs,” 2017, hereinafter Merity).
Regarding claim 1:
No teaches a computer-implemented method (¶14 “a processor-implemented method”) comprising: performing a training process on a neural network resulting in a trained neural network (Fig. 1 – 10, ¶43 “computing apparatus 10 may be configured to perform a neural network operation of a neural network model. In an example, the computing apparatus 10 may perform training… of a machine leaning model”), wherein the neural network includes a non-polynomial layer (NPL) that is incompatible with homomorphic encryption (HE) (Fig. 3 – “Set approximation region… of ReLU function of each layer,” ¶39 “A machine learning model (e.g., a neural network model) may be utilized for homomorphic encrypted data… it is found that a polynomial with a low degree may be used, e.g., as an activation function, to perform a neural network operation using the homomorphic encrypted data,” ¶41 “in such neural network operation for the fully homomorphic encrypted data using the high degree polynomial, the typical neural network model needs to be retrained based on the high degree polynomial having replaced the existing activation function of the typical neural network model, e.g., having replaced a standard ReLU activation function in the typical neural network model,” wherein to replace an “existing activation function” with a “polynomial” approximation in order to process “homomorphic encrypted data” implies that the original non-polynomial layer, or “ReLU activation function” of the original layer, is incompatible with homomorphic encryption).
No further teaches the training process comprising: performing a plurality of training iterations on the neural network (Fig. 1 – 200, Fig. 4 – 430-470, 490, ¶88 “if the high accuracy threshold is not achieved through the set approximation region, the processor 200 may repeat the operation of newly setting the approximation region using the maximum and minimum values in the obtained approximation region. When the predetermined accuracy threshold is achieved, the processor 200 may generate/output the set approximation region for use with the approximate polynomial that replaces a ReLU unit/function of a ReLU layer of the neural network”).
Regarding the limitation and performing, between each of the plurality of training iterations, loss processing that (i) minimizes a loss of the neural network and (ii) reduces a range of values to the NPL, No further teaches and performing, between each of the plurality of training iterations, loss processing that (i) minimizes a loss of the neural network (¶46 “The neural network model may be trained to infer a result from an input by incrementally adjusting weights of the nodes through training… Each of such nodes of the plural layers may also include respective biases that may be determined or set during training,” Fig. 1 – 200, Fig. 4 – 430-470, 490, ¶88 “if the high accuracy threshold is not achieved… the processor 200 may repeat the operation of newly setting the approximation region”) and a range of values to the NPL (Fig. 3, ¶59 “a maximum value and a minimum value of respective input data to one or more layers (e.g., ReLU layers)”). However, No fails to teach and (ii) reduces a range of values to the NPL.
Obla, in the same field of endeavor, teaches and (ii) reduces a range of values to an NPL (Page 24 Figure 3.2 depicts polynomial approximations of ReLU, a non-polynomial activation function, Page 24 Table 3.2 depicts distribution ranges for input values to an activation function, Page 24 ¶1 “we analysed the… inputs to the activation function,” Page 25 ¶1 “Depending on the complexity of the dataset, we observe that at least 98% of the data lies between the range [-3, 3], but the ranges can be as large as [-30, 21],” Page 25 ¶4 “our first observation was the significant increase in accuracy for the polynomial approximated between [-7, 7]… the activation layer is able to process more inputs accurately,” Page 26 Table 3.3 depicts the accuracy of using smaller ranges of input values, Page 33 ¶1 “since the polynomial approximation will be accurate only within their approximation range, it is necessary to restrict the range of inputs,” Page 34 Table 4.1 depicts the maximum input ranges recorded for every activation layer; “to restrict the range of inputs” to smaller ranges such as “[-3, 3]” or “[-7, 7]” as opposed to the larger ranges depicted in Tables 3.2 and 4.1 encompasses reduces a range of values to a non-polynomial layer (NPL)).
No and Obla are analogous to the claimed invention as both are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the iterative training and reducing a range of values of Obla with the methodology of No. The motivation to do so is to introduce a method of approximating activation functions using polynomial approximations that “outperforms other methods and is robust regardless of the approximation method, degree, dataset, or activation function approximated” (Obla, Page 52 ¶1).
The combination of No and Obla fails to teach wherein the performing of the loss processing comprises minimizing the loss of the neural network using a loss function that includes a regularization loss term. However, Merity, in the same field of endeavor, teaches this limitation (Abstract: “We revisit traditional regularization techniques, specifically L2 regularization on RNN activations… to improve the performance of RNNs on the task of language modeling,” Page 1, Col. 2, ¶5 “we revisit L2 regularization in the form of activation regularization (AR),” one of ordinary skill in the art would recognize that an “L2 regularization” is a term added to a standard loss function which penalizes large model weights, hence minimizing the loss of the neural network using a loss function that includes a regularization loss term is implicit).
Regarding the limitation and the regularization range loss term causes model weights of the neural network to produce smaller values for input values of the NPL, Obla teaches to produce smaller values for input values of the NPL (Page 33, ¶1 “since the polynomial approximation will be accurate only within their approximation range, it is necessary to restrict the range of inputs”). However, the combination of No and Obla fails to teach and the regularization range loss term causes model weights of the neural network to produce smaller values…
Merity teaches and the regularization range loss term causes model weights of the neural network to produce smaller values (Page 2, Col. 1, Section 2.1, ¶1 “While L2 regularization is traditionally used on the weights of machine learning models (L2 weight decay), it could also be used on the activations. We define AR as
PNG
media_image1.png
56
153
media_image1.png
Greyscale
where… ht is the output of the RNN at timestep t, and α is a scaling coefficient,” ¶2 “When applied to the output of a dense layer, AR penalizes activations that are substantially away from 0, encouraging the activations to remain small,” wherein the “activation regularization (AR),” or the regularization range loss term, when applied to an implicit loss function as a penalty term, is also implied to cause weights of the neural network to produce smaller activation values, Page 3, Col. 1, ¶1 “experiments use a model containing a two layer RNN. The AR… only applied to the output of the final RNN layer”).
No further teaches estimating a range of input values to the NPL of the trained neural network (Fig. 1 – 200, Fig. 3, ¶59 “The processor 200 may calculate a maximum value and a minimum value of respective input data to one or more layers (e.g., ReLU layers) … based on data input to the neural network”).
No further teaches forming a replacement layer for the NPL, wherein: the replacement layer comprises a polynomial approximation of an operation performed by the NPL (Fig. 1 – 200, ¶66 “The processor 200 may be configured to implement the neural network using the target approximate polynomial instead of an original neural network operation… the neural network may have a ReLU layer that uses a typical ReLU unit or function for an insertion of non-linearity… the typical ReLU of the neural network may be replaced (updated) with the target approximate polynomial, and act on input data from another layer of the neural network”).
No further teaches and the polynomial approximation of the operation performed by the NPL is HE-compatible (Fig. 1 – 200, ¶72 “when approximating a rectified linear unit (ReLU unit) (or a ReLU function) used in typical neural networks for fully homomorphic encrypted data, a processor (e.g., the processor 200 of FIG. 1) may effectively generate a target approximate polynomial that approximates the ReLU unit/function of a ReLU layer”).
No further teaches generating a revised neural network by replacing the NPL of the trained neural network with the replacement layer (Fig. 2 – 220, Fig. 3 – 310, ¶76 “the generated neural network 310 may correspond to the trained neural network 220 except that the ReLU layers now apply the respective target approximate polynomials instead of the ReLU units/functions of the ReLU layers in the trained neural network 220”).
No further teaches and processing encrypted data using the revised neural network (Fig. 1 – 200, Fig. 7 – 770, ¶121 “the processor 200 may configured to generate a homomorphic encrypted data operation result by inputting the homomorphic encrypted data to a resultant neural network and performing each of the layers of the neural network, including the one or more ReLU layers that respectively apply their target approximate polynomials instead of a typical ReLU operation”).
No, Obla, and Merity are analogous to the claimed invention as both are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the smaller range of input values to an NPL layer of Obla and the loss function including the regularization range loss term of Merity with the methodology of No. The motivation to do so is to introduce a method of approximating activation functions using polynomial approximations that “outperforms other methods and is robust regardless of the approximation method, degree, dataset, or activation function approximated” (Obla, Page 52 ¶1) while performing regularization that “require minimal modification to… architectures and result in performance improvements comparable to superior or more complicated regularization techniques or custom cell architectures” (Merity, Abstract).
Regarding claim 2, No in view of Obla and further in view of Merity teaches the computer-implemented method of claim 1 (and thus the rejection of claim 1 is incorporated).
No fails to teach wherein: the performing of the training process on the neural network comprises training the neural network using a first training dataset. However, Obla teaches this limitation (Page 42 ¶3 “The model is initially trained using a traditional activation function like Softplus… The training on MNIST using the original activation is conducted”).
No further teaches and the estimating of the range of input values to the NPL comprises inputting instances of a second training dataset to the trained neural network and determining corresponding training input values to the NPL (Fig. 1 – 200, Fig. 2 – 230, 250, Fig. 4 – 410, ¶81 “The processor 200 may calculate maximum and minimum values 250 of the input values of the ReLU function based on the pre-trained deep learning model 220 and the sample 230 of the trained data set”).
No and Obla are analogous to the claimed invention as both are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the training process and first training dataset of Obla with the second training dataset and estimating of the range of input values of No. The motivation to do so, as stated by Obla, is to introduce a method of approximating activation functions using polynomial approximations that “outperforms other methods and is robust regardless of the approximation method, degree, dataset, or activation function approximated” (Obla, Page 52 ¶1).
Regarding claim 3, No in view of Obla and further in view of Merity teaches the computer-implemented method of claim 2 (and thus the rejection of claim 2 is incorporated).
Regarding the limitation wherein the estimating of the range of input values to the NPL further comprises performing a statistical analysis using the range of input values to the NPL, No teaches the estimating of the range of input values to the NPL (Fig. 3, ¶59, as discussed above with respect to claim 1). However, No fails to teach wherein the estimating of the range… further comprises performing a statistical analysis using the range of input values to the NPL.
Obla wherein the estimating of the range… further comprises performing a statistical analysis using the range of input values to the NPL (Page 24 ¶1 “we analysed the… inputs to the activation function,” Page 24 Table 3.2 depicts standard deviation, or statistical analysis, Page 25 ¶1 “we observe that at least 98% of the data lies between the range [-3, 3],” Page 25 ¶3 “we construct two polynomials of degree 4 between the bounds [-7, 7] and [-25, 25]. In this experiment we also compare the performance with a degree 4 polynomial approximated between [-3, 3],” Page 26 ¶2 “From our experiments in this section, we can conclude that it is necessary to strike a balance between maintaining an acceptable error of approximation between [-3, 3], while covering a range larger than [-3, 3]. This is true, especially for complex datasets that have large input distributions to the activation layer,” Page 26 Table 3.3 depict the results of polynomial approximations using different input ranges such as “[-3, 3]” and “[-7, 7]” including their respective accuracies and approximation errors, which are calculated through performing a statistical analysis using the training input range to the NPL).
No and Obla are analogous to the claimed invention as both are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the statistical analysis of Obla with the estimating of the range of input values of No. The motivation to do so, as stated by Obla, is to introduce a method of approximating activation functions using polynomial approximations that “outperforms other methods and is robust regardless of the approximation method, degree, dataset, or activation function approximated” (Obla, Page 52 ¶1).
Regarding claim 4, No in view of Obla and further in view of Merity teaches the computer-implemented method of claim 3 (and thus the rejection of claim 3 is incorporated).
No teaches determining the polynomial approximation of the operation performed by the NPL (Fig. 6, ¶110 “A solid line represents an accurate typical ReLU and a dotted line represents an approximate polynomial approximated in an approximation region”), wherein the determining of the polynomial approximation comprises determining a degree of the polynomial approximation based at least in part on the estimated range of input values to the NPL (Fig. 3, ¶59, as explained above with respect to claim 1, Fig. 1 – 200, Fig. 2 – 220, Fig. 3 – 310, ¶75 “The processor 200 may adjust the approximation region using the obtained minimum and maximum values. Through this operation, the processor 200 may effectively set a target approximation region for an interim approximate polynomial with the same degree (e.g., without having to change a degree of the interim approximate polynomial),” ¶76 “The processor 200 may generate the neural network 310… while using a low polynomial degree, by effectively setting the approximation region of the approximate polynomial based on the values of the input data… the generated neural network 310 may correspond to the trained neural network 220 except that the ReLU layers now apply the respective target approximate polynomials instead of the ReLU units/functions of the ReLU layers in the trained neural network 220”).
Regarding claim 7, No in view of Obla teaches the computer-implemented method of claim 1 (and thus the rejection of claim 1 is incorporated).
No teaches wherein: the neural network comprises a plurality of NPLs including the NPL (Fig. 1 – 200, Fig. 3 depicts a plurality of NPLs including the NPL).
Regarding the limitation and the performing of the loss processing comprises minimizing NPL input values to each of the plurality of NPLs, No teaches and the performing of the loss processing comprises adjusting NPL input values to each of the plurality of NPLs (¶62 “The processor 200 may set the approximation region for respective approximate polynomials for each of the ReLU layers of the neural network based on the respective maximum value and the respective minimum value for each input to each ReLU layer. In one example, the processor 200 may set the approximation region for respective ReLU layers based on the number of ReLU layers included in the neural network model and the total number of respective input data to the neural network model”). However, No fails to teach minimizing NPL input values to each of the plurality of NPLs.
Obla teaches minimizing NPL input values to each of a plurality of NPLs (Page 34 Table 4.1 depicts a plurality of NPLs, each with differing ranges of inputs, Page 34 ¶1 “the range of inputs differs from layer to layer,” Page 34 ¶2 “We can take advantage of this phenomenon by approximating a polynomial for every layer… the approximation for every layer would be able to accept most of the inputs while having the lowest error of approximation… layers with a smaller input range will not be constrained by a single polynomial catering to the largest range observed,” wherein approximating a polynomial for every layer that “would be able to accept most of the inputs,” but not all inputs within the range of inputs, while having “the lowest error of approximation,” implies minimizing NPL input values to each of the plurality of an NPLs).
No and Obla are analogous to the claimed invention as both are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the minimizing NPL input values of Obla with the plurality of NPLs of No. The motivation to do so, as stated by Obla, is to introduce a method of approximating activation functions using polynomial approximations that “outperforms other methods and is robust regardless of the approximation method, degree, dataset, or activation function approximated” (Obla, Page 52 ¶1).
Regarding claim 10:
No teaches a computer program product (¶124 “Instructions or software to control computing hardware… to… perform the methods as described above may be written as computer programs”) comprising one or more computer-readable storage media (¶125 “instructions or software to control computing hardware… may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media”).
No further teaches and program instructions stored on the one or more computer-readable storage media to perform operations (Fig. 1 – 200, 300, ¶125 “The instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above, and any associated data… may be… stored… in or on one or more non-transitory computer-readable storage media”) comprising: performing a training process on a neural network resulting in a trained neural network (Fig. 1 – 10, ¶43 as described above with respect to claim 1), wherein the neural network includes a non-polynomial layer (NPL) that is incompatible with homomorphic encryption (HE) (Fig. 3, ¶¶39, 41 all as explained above with respect to claim 1)…
Claim 10 recites a system that parallels the method of claim 1. Therefore, the analysis discussed above with respect to claim 1 applies to claim 10. Accordingly, claim 10 is rejected based on substantially the same rationale as set forth above with respect to claim 1.
Claims 13-15 recite a computer program product that parallels the method claims of 2-4, respectively. Therefore, the analysis discussed above with respect to claims 2-4 also applies to claims 13-15, respectively. Accordingly, claims 13-15 are rejected based on substantially the same rationale as set forth above with respect to claims 2-4, respectively.
Claim 18 recites a system that parallels the product of claim 10. Therefore, the analysis discussed above with respect to claim 10 applies to claim 18. Accordingly, claim 18 is rejected based on substantially the same rationale as set forth above with respect to claim 10.
Claims 5-6 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over No in view of Obla and further in view of Merity, and further in view of Zhu et al. (US 20210365710 A1, hereinafter Zhu).
Regarding claim 5, No in view of Obla and further in view of Merity teaches the computer-implemented method of claim 1 (and thus the rejection of claim 1 is incorporated).
Regarding the limitation wherein the NPL is selected from the group consisting of: an activation layer, an instance normalization layer, a maximum pooling layer, and a softmax layer, No teaches wherein the NPL is selected from the group consisting of: an activation layer (Fig. 1 – 200, Fig. 2 – 220, Fig. 3 depicts a plurality of ReLU, or the group consisting of: an activation layer, Fig. 7 – 730, ¶114 “information is generated and processed through (with) the forward pass of the neural network, when each ReLU layer is reached, the processor 200 may be configured to respectively calculate a maximum value and a minimum value of the corresponding input data generated up to that point of the corresponding ReLU layer… of a neural network”). However, No fails to teach the group consisting of: an activation layer, an instance normalization layer, a maximum pooling layer, and a softmax layer.
Zhu, in the same field of endeavor, teaches an instance normalization layer, a maximum pooling layer, and a softmax layer (Fig. 3, ¶63 “The first layer of each ladder from left to right in FIG. 3 (except the last ladder) is the max pooling layer, the first three layers from left to right in the last ladder in FIG. 3 are full connection layer (fully connected+Relu), the last layer from left to right in the last ladder in FIG. 3 is the activation layer (softmax), and the remaining layers in FIG. 3 are convolution layers (convolution+Relu),” ¶47 “Instance Normalization layer is used to normalize the feature image output from the convolutional layer,” ¶65 “Optionally, the normalizing network includes an Adaptive Instance Normalization (AdaIN) processing layer, and the coding full connection layer is connected to the AdaIN processing layer”).
No and Zhu are analogous to the claimed invention as both are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the layers of Zhu with the NPL and group consisting of an activation layer of No. The motivation to do so, as stated by Zhu, is to design an iterative training method that “can train a style transfer network model adapted to multiple types of images” (Zhu, ¶105).
Regarding claim 6, No in view of Obla and further in view of Merity, and further in view of Zhu teaches the computer-implemented method of claim 5 (and thus the rejection of claim 5 is incorporated).
No teaches wherein the polynomial approximation is of an activation function (Fig. 2 – 220, Fig. 3 – 310, ¶76 “the generated neural network 310 may correspond to the trained neural network 220 except that the ReLU layers now apply the respective target approximate polynomials instead of the ReLU units/functions of the ReLU layers in the trained neural network 220”).
Claims 16-17 recite a computer program product that parallels the method claims of 5-6, respectively. Therefore, the analysis discussed above with respect to claims 5-6 also applies to claims 16-17, respectively. Accordingly, claims 16-17 are rejected based on substantially the same rationale as set forth above with respect to claims 5-6, respectively.
Claims 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over No in view of Obla and further in view of Merity, and further in view of Teig et al (US 12112254 B1, hereinafter Teig).
Regarding claim 8, No in view of Obla and further in view of Merity teaches the computer-implemented method of claim 7 (and thus the rejection of claim 7 is incorporated).
Regarding the limitation wherein the performing of the loss processing comprises minimizing NPL input values to each of the plurality of NPLs during respective separate training iterations, No teaches the performing of the loss processing (¶46, as explained above with respect to claim 1). However, No fails to teach wherein the performing of the loss processing comprises minimizing NPL input values to each of the plurality of NPLs during respective separate training iterations.
Obla teaches wherein loss processing comprises minimizing NPL input values to each of the plurality of NPLs (Page 34 Table 4.1, Page 34 ¶1, Page 34 ¶2, all as explained above with respect to claim 7). However, Obla fails to teach during respective separate training iterations.
Teig, in the same field of endeavor, teaches performing loss processing during respective separate training iterations (Col. 7 Lines 43-52, “some embodiments… iteratively trains the MT network by progressively adding data to the inputs used to train the network at each iteration. Between iterations, the hyperparameters are optimized by determining the error of the network as trained from the prior iteration when using a set of validation inputs, and modifying the hyperparameters to decrease this error. The set of validation inputs… are then added to the training inputs for the next iteration”).
No, Obla, and Teig are analogous to the claimed invention as all are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the minimizing of NPL input values of Obla and the respective separate training iterations of Teig with the loss processing and each of the plurality of NPLs of No. The motivation to do so is to introduce a method of approximating activation functions using polynomial approximations that “outperforms other methods and is robust regardless of the approximation method, degree, dataset, or activation function approximated” (Obla, Page 52 ¶1) and to produce a training method that “reduces overfitting by preventing the network from learning the noise in the training set” (Teig, Col. 16 Lines 1-3).
Regarding claim 9, No in view of Obla and further in view of Merity teaches the computer-implemented method of claim 1 (and thus the rejection of claim 1 is incorporated).
Regarding the limitation wherein the performing of the loss processing comprises minimizing the loss of the neural network using a loss function that further includes a standard loss term, No teaches wherein the performing of the loss processing comprises minimizing the loss of the neural network (¶46, as explained above with respect to claim 1). However, No fails to teach using a loss function that further includes a standard loss term.
Teig teaches using a loss function that further includes a standard loss term (Col. 7 Lines 17-25 “the training process typically (1) forward propagates the input value set through the network's nodes to produce a computed output value set and then (2) backpropagates a gradient (rate of change) of a loss function (output error) that quantifies in a particular way the difference between the input set's known output value set and the input set's computed output value set, in order to adjust the network's configurable parameters (e.g., the weight values),” Col. 20 Lines 53-54 “One option is to use the standard cross-entropy loss function to train the decoder network,” wherein “standard cross-entropy loss function” encompasses a standard loss term).
No and Teig are analogous to the claimed invention as both are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the loss function including a standard loss term of Teig with the loss processing of No. The motivation to do so is to produce a training method that “reduces overfitting by preventing the network from learning the noise in the training set” (Teig, Col. 16 Lines 1-3).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over No in view of Obla and further in view of Merity, and further in view of Beaty et al. (US 20140136707 A1, hereinafter Beaty).
Regarding claim 11, No in view of Obla and further in view of Merity teaches the computer-implemented method of claim 10 (and thus the rejection of claim 10 is incorporated).
No teaches wherein: the program instructions are stored in a computer-readable storage device in a data processing system (¶125 “Examples of a non-transitory computer-readable storage medium include… magneto-optical data storage devices, optical data storage devices… and any other device that is configured to store the instructions or software,” Fig. 1 – 100-300, ¶57 “The processor 200 may include one or more data processing devices… the execution of the instructions by the one or more data processing devices may configure the processor 200 to perform any one or any combinations of the operations/methods described herein,” wherein a configuration of devices including “data processing devices” encompasses a data processing system, ¶52 “The receiver 100 may include a receiving interface, through which various data are received by the receiver 100. The receiver 100 may receive data from an external device”).
No fails to teach and the program instructions are received over a network from a remote data processing system. However, Beaty, in the same field of endeavor, teaches this limitation (¶23 “The program code may be run… entirely on the remote computer… the remote computer may be connected to the user's computer through any type of network,” Fig. 10 – 1000, 1004, 1018, 1020, ¶112 “Program code 1018 is located in a functional form on computer readable media 1020 that… may be loaded onto or transferred to data processing system 1000 for processing by processor unit 1004,” ¶116 “The data processing system providing program code 1018 may be… a remote data processing system”).
No and Beaty are analogous to the claimed invention as both are from the same field of endeavor of receiving data for a data processing system. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the transferring of instructions over a network from a remote data processing system of Beaty with the data processing system of No. The motivation to do so is to design “a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources… that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service” (Beaty, ¶29).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over No in view of Obla and further in view of Merity, and further in view of Beaty, and further in view of Cohen et al. (US 20140040446 A1, hereinafter Cohen).
Regarding claim 12, No in view of Obla and further in view of Merity teaches the computer-implemented method of claim 10 (and thus the rejection of claim 10 is incorporated).
Regarding the limitation wherein the program instructions are stored in a computer-readable storage device in a server data processing system and the program instructions are downloaded responsive to a request over a network to a remote data processing system for use in a computer readable storage device associated with the remote data processing system, and wherein the operations further comprise: metering use of the program instructions associated with the request, No teaches wherein the program instructions are stored in a computer-readable storage device in a server data processing system (¶125 “Examples of a non-transitory computer-readable storage medium include… magneto-optical data storage devices, optical data storage devices… and any other device that is configured to store the instructions or software”, Fig. 1 – 1, 10, 200, 300, ¶48 “The computing apparatus 10 may be… a data server… or the electronic device 1 may be… the data server”). However, No fails to teach and the program instructions are downloaded responsive to a request over a network to a remote data processing system for use in a computer readable storage device associated with the remote data processing system, and wherein the operations further comprise: metering use of the program instructions associated with the request.
Beaty teaches and the program instructions are downloaded (Fig. 10 – 1000, 1008, 1018, 1026, ¶116 “program code 1018 may be downloaded over a network to persistent storage 1008 from another device or data processing system through computer readable signal media 1026 for use within data processing system 1000. The data processing system providing program code 1018 may be… a remote data processing system”)… for use in a computer readable storage device associated with the remote data processing system (Fig. 4 – 400, 404, 410, ¶23 “The program code may be run… entirely on the remote computer… the remote computer may be connected to the user's computer through any type of network,” ¶75 “set of resources 410 can be software… set of resources 410 may be services on computer system 404 or on another computer system in data processing environment 400,” Fig. 10 – 1018-1024, ¶112 “Program code 1018 and computer readable media 1020 form computer program product 1022 in these examples. In one example, computer readable media 1020 may be computer readable storage media 1024,” ¶113 “computer readable storage media 1024 is a physical or tangible storage device used to store program code 1018”), and wherein the operations further comprise: metering use of the program instructions associated with a request (Fig. 4 – 404-413, 432-436, 446-448, ¶82 “user 412 may use client 413 to make request 446 to verify information about the use of set of resources 410 by service 408… Responsive to receiving request 446, computer system 404 generates timeline graph 448 using signed information 432 such as timestamps 436 in metering data 434,” Fig. 4 –414, 418, 422, Fig. 7, ¶95 “FIG. 7 may be implemented in software… the steps may be implemented by metering components, such as metering 414 in provenance service 416, metering 418 in hypervisor management 420, and metering 422 in service 408”). However, the combination of No, Obla, Merity, and Beaty fails to teach and the program instructions are downloaded responsive to a request over a network to a remote data processing system and the request.
Cohen, in the same field of endeavor, teaches downloading data responsive to a request over a network to a remote data processing system (Fig. 1 – 110, 120, 124, 140, 150, Fig. 2 – S210-230, ¶24 “in response to receiving a request from computing machine 110, for accessing or manipulating target data, service 150 dynamically invokes one or more instances of remote software 124 on remote host 120 (S210). If it is determined that the location of the target data is remote to the remote host 120 (e.g., if the target data is stored on local computing machine 110 or on remote storage 140), then service 150 seamlessly transfers the data to remote host 120 where the remote software 124 is hosted”) and the request (24 “in response to receiving a request…”).
Beaty further teaches and generate an invoice based on the metered use (Fig. 4 – 408, 410, 412, 430, 444, ¶81 “Resource usage report 444 may also include one more summaries… The summaries may describe the use of set of resources 410 by service 408 over… time periods of a particular event of interest. Events of interest… may include times of high use of set of resources 410 by service 408, times of low use… and times specified by metering policy 430… metering policy 430 may include instructions to generate a summary in resource usage report 444 describing the use of set of resources 410 by service 408… resource usage report 444 may also be used in the form of an invoice or bill that is sent to user 412”).
No, Beaty, and Cohen are analogous to the claimed invention as all are from the same field of endeavor of data processing. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the downloaded program instructions and remote data processing system of Beaty and the response to the request of Cohen with the server data processing system of No. The motivation to do so, as stated by both Beaty and Cohen, is to design “a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources… that [may/can] be rapidly provisioned and released with minimal management effort or interaction with a provider of the service” (Beaty, ¶29 and Cohen, ¶60).
Claims 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over No in view of Obla and further in view of Merity, and further in view of Teig, and further in view of Teig et al. (US 20220405591 A1, hereinafter Teig (2022)).
Regarding claim 19, No in view of Obla and further in view of Merity teaches the computer system of claim 18 (and thus the rejection of claim 18 is incorporated).
No further teaches wherein the neural network further comprises a first layer (Fig. 2 – 220, Fig. 3, ¶74 “The ReLU functions may be functions of respective ReLU layers, which may each follow another layer of the neural network 220 that may perform a different neural network operation (e.g., a convolution layer…), as non-limiting examples”).
Regarding the limitation and for an iteration in the plurality of training iterations: the neural network generates a weighted output value by applying a first weight value to an output of the first layer, No teaches the first layer (Fig. 2 – 220, ¶74 “a convolution layer…”). However, the combination of No, Obla, and Merity fails to teach and for an iteration in the plurality of training iterations: the neural network generates a weighted output value by applying a first weight value to an output of the first layer.
Teig teaches and wherein, for an iteration in the plurality of training iterations: the neural network generates a weighted output value by applying a first weight value to an input of a first layer (Col. 7 Lines 53-56 “for a particular iteration, a first set of training inputs are used to train the parameters of the MT network (e.g., the weight values for a neural network) using a first set of hyperparameters,” Col. 5 Lines 36-39 “The weight coefficients W(l) are parameters that are adjusted during the network's training in order to configure the network to solve a particular problem,” Fig. 2 – 200-210, Col. 6 Lines 19-21 “FIG. 2 conceptually illustrates a representation of a convolutional layer of a convolutional neural network,” Col. 6 Lines 38-42 “the layer includes six filters 205… Each value in one of the filters is a weight value that is trained… each filter includes 27 trainable weight values,” wherein “a weight value that is trained,” or a first weight value, is implied to be trained after “a particular iteration” or in a certain training iteration in which “inputs are used to train the parameters of the… network (e.g., the weight values for a neural network),” Col. 6 Lines 54-60 “To generate the output activations, each of the filters 205 is applied to numerous subsets of the input activation values… and the dot product between the 27 activations in the current subset and the 27 weight values in the filter is computed”). However, Teig fails to teach applying a first weight value to an output of the first layer.
Teig (2022), in the same field of endeavor, teaches applying weight values to an output of a layer (Fig. 2 – 210, ¶30 “the linear component 210 of each input neuron of some embodiments computes a dot product of a vector of weight coefficients and a vector of input values,” Fig. 5, ¶57 “FIG. 5 illustrates a simple feed-forward neural network 500 with one hidden layer having two nodes, and a single output layer with one output node… The output layer node C receives its inputs from the outputs of nodes A and B, and uses weight values wCA and wCB respectively for its linear component,” wherein the “linear component” of “output layer node C” which “computes a dot product of a vector of weight coefficients” and “a vector of input values,” or the “outputs of nodes A and B,” encompasses applying weight values to an output of a layer).
Regarding the limitation and the NPL receives the weighted output value as an input value in the range of input values, No teaches and the NPL receives input data as an input value in the range of input values (Fig. 1 – 200, Fig. 2 – 220, 230, 250, ¶66 “the typical ReLU of the neural network may… act on input data from another layer of the neural network,” ¶81 “The processor 200 may calculate maximum and minimum values 250 of the input values of the ReLU function based on the pre-trained deep learning model 220 and the sample 230 of the trained data set”). However, No fails to teach and the NPL receives the weighted output value as an input value in the range of input values.
Teig teaches wherein a layer receives the weighted output value as an input value (Fig. 2 – 210, Col 7 Lines 8-10 “These output activation values 210 are then the input activation values for the next layer of the neural network”).
No, Teig, and Teig (2022) are analogous to the claimed invention as all are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the plurality of training iterations, weighted output value, and first weight value of Teig and the applying of a weight value to an output of Teig (2022) with the first and second layers of the neural network of No. The motivation to do so is to produce a training method that “reduces overfitting by preventing the network from learning the noise in the training set” (Teig, Col. 16 Lines 1-3) while designing “Techniques that allow for lower power consumption without a reduction in the effectiveness of a neural network” (Teig (2022), ¶3).
Regarding claim 20, No in view of Obla and further in view of Merity, and further in view of Teig, and further in view of Teig (2022) teaches the computer system of claim 19 (and thus the rejection of claim 19 is incorporated).
Regarding the limitation wherein the loss processing comprises adjusting the first weight value, No teaches wherein the loss processing comprises adjusting weight values (¶46, as explained above with respect to claim 1). However, No fails to teach wherein the loss processing comprises adjusting the first weight value.
Teig teaches adjusting the first weight value (Col. 7 Lines 53-56, Col. 5 Lines 36-39, Fig. 2 – 200-210, Col. 6 Lines 39-42 “Each value in one of the filters is a weight value that is trained,” all as explained above with respect to claim 19, Col. 7 Lines 13-25 “the network is put through a supervised training process that adjusts the network's configurable parameters (e.g., the weight coefficients of its linear components)… the training process typically… backpropagates a gradient (rate of change) of a loss function (output error) that quantifies in a particular way the difference between the input set's known output value set and the input set's computed output value set, in order to adjust the network's configurable parameters (e.g., the weight values)”).
No and Teig are analogous to the claimed invention as both are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the first weight value of Teig with the loss processing of No. The motivation to do so is to produce a training method that “reduces overfitting by preventing the network from learning the noise in the training set” (Teig, Col. 16 Lines 1-3).
Claims 21-25 are rejected under 35 U.S.C. 103 as being unpatentable over No in view of Teig and further in view of Obla, and further in view of Merity.
Regarding claim 21:
No teaches a computer-implemented method (¶14) comprising: performing, on a neural network comprising at least a first layer and a second layer, wherein the second layer is a non-polynomial layer (NPL) (Fig. 2 – 220, Fig. 3, ¶74 “The ReLU functions may be functions of respective ReLU layers, which may each follow another layer of the neural network 220 that may perform a different neural network operation (e.g., a convolution layer…), as non-limiting examples”) that is incompatible with homomorphic encryption (HE) (Fig. 3, ¶¶39, 41 all as explained above with respect to claim 1), a training process on a neural network resulting in a trained neural network (Fig. 1 – 10, ¶43).
Regarding the limitation the training process comprising: performing a plurality of training iterations on the neural network, wherein in a certain training iteration a weighted output value is generated by using a first weight value to generate an output of the first layer and the weighted output value is sent as an input value in the NPL, No further teaches the training process comprising: performing a plurality of training iterations on the neural network (Fig. 1 – 200, Fig. 4 – 430-470, 490, ¶88, as explained above with respect to claim 1) and the first layer (¶74 “(e.g., a convolution layer…)”) and the NPL (¶74 “respective ReLU layers”). However, No fails to teach wherein in a certain training iteration a weighted output value is generated by using a first weight value to generate an output of the first layer and the weighted output value is sent as an input value in the NPL.
Teig teaches wherein in a certain training iteration a weighted output value is generated by using a first weight value to generate an output of a first layer and the weighted output value is sent as an input value in a second layer (Col. 7 Lines 53-56, Col. 5 Lines 36-39, Fig. 2 – 200-210, Col. 6 Lines 19-21, Col. 6 Lines 38-42, Col. 6 Lines 54-60, Fig. 2 – 210, Col. 7, Lines 8-10, all as explained above with respect to claim 19).
Regarding the limitation, and performing, between a selected training iteration and another training iteration after the selected training iteration, loss processing that (i) minimizes a loss of the neural network and (ii) reduces a range of values to the NPL, No further teaches loss processing that (i) minimizes a loss of the neural network (¶46, Fig. 1 – 200, Fig. 4 – 430-470, 490, ¶88, as described above with respect to claim 1) and a range of values to the NPL (Fig. 3, ¶59 as described above with respect to claim 1). However, No fails to teach and performing, between a selected training iteration and another training iteration after the selected training iteration, loss processing that… (ii) reduces a range of values to the NPL.
Obla teaches and performing, between a selected training iteration and another training iteration after the selected training iteration (Page 42 ¶3 “the first phase of training is performed for 450 epochs… After replacing the activation function with an approximation, the training continues… for 300 epochs”), loss processing that (ii) reduces a range of values to an NPL (Page 24 Figure 3.2, Page 24 Table 3.2, Page 24 ¶1, Page 25 ¶1, Page 25 ¶4, Page 26 Table 3.3, Page 33 ¶1, Page 34 Table 4.1 all as explained above with respect to claim 1).
No fails to teach wherein the performing of the loss processing comprises minimizing the loss of the neural network using a loss function that includes a regularization loss term. However, Teig teaches this limitation (Col. 7 Lines 17-25 “the training process typically (1) forward propagates the input value set through the network's nodes to produce a computed output value set and then (2) backpropagates a gradient (rate of change) of a loss function (output error) that quantifies in a particular way the difference between the input set's known output value set and the input set's computed output value set, in order to adjust the network's configurable parameters (e.g., the weight values),” Col. 3 Line 66-Col. 4 Line 5 “optimize the training of the parameters of a machine-trained (MT) network by optimizing the tuning of a set of hyperparameters that define how the training of the MT network is performed. These hyperparameters, in various embodiments, may include coefficients in the loss function used to train the network (e.g., L1 and L2 regularization parameters),” wherein “L1 and L2 regularization parameters” encompasses a regularization range loss term).
Regarding the limitation and the regularization range loss term causes model weights of the neural network to produce smaller values for input values of the NPL, Teig teaches and the regularization range loss term (Col. 3 Line 66-Col. 4 Line 5 “optimize the training of the parameters of a machine-trained (MT) network by optimizing the tuning of a set of hyperparameters… These hyperparameters… may include coefficients in the loss function used to train the network (e.g., L1 and L2 regularization parameters)”). However, Teig fails to teach and the regularization range loss term causes model weights of the neural network to produce smaller values for input values of the NPL.
Obla teaches to produce smaller values for input values of the NPL (Page 33, ¶1). However, the combination of No, Teig, and Obla fails to teach and the regularization range loss term causes model weights of the neural network to produce smaller values…
Merity teaches a regularization range loss term causes model weights of the neural network to produce smaller values (Page 2, Col. 1, Section 2.1, ¶¶1-2, Page 3, Col. 1, ¶1 all as explained above with respect to claim 1).
No further teaches estimating a range of input values to the NPL of the trained neural network (Fig. 1 – 200, Fig. 3, ¶59).
No further teaches forming a replacement layer for the NPL, wherein: the replacement layer comprises a polynomial approximation of an operation performed by the NPL (Fig. 1 – 200, ¶66 as explained above with respect to claim 1).
No further teaches and the polynomial approximation of the operation performed by the NPL is HE-compatible (Fig. 1 – 200, ¶72).
No further teaches generating a revised neural network by replacing the NPL of the trained neural network with the replacement layer (Fig. 2 – 220, Fig. 3 – 310, ¶76).
No further teaches and processing encrypted data using the revised neural network (Fig. 1 – 200, Fig. 7 – 770, ¶121).
No, Teig, Obla, and Merity are analogous to the claimed invention as all are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the iterative training, weighted output, and the first weight value of Teig, the reducing a range of values and selected training iteration of Obla, and the loss function including the regularization range loss term of Merity with the methodology of No. The motivation to do so is to introduce a method of approximating activation functions using polynomial approximations that “outperforms other methods and is robust regardless of the approximation method, degree, dataset, or activation function approximated” (Obla, Page 52, ¶1) while producing a training method that “reduces overfitting by preventing the network from learning the noise in the training set” (Teig, Col. 16 Lines 1-3) and performing regularization that “require minimal modification to… architectures and result in performance improvements comparable to superior or more complicated regularization techniques or custom cell architectures” (Merity, Abstract).
Regarding claim 22, No in view of Teig and further in view of Obla, and further in view of Merity teaches the computer-implemented method of claim 21 (and thus the rejection of claim 21 is incorporated).
Regarding the limitation wherein the loss processing comprises adjusting the first weight value, No teaches wherein the loss processing comprises adjusting weight values (¶46, as explained above with respect to claim 1). However, No fails to teach wherein the loss processing comprises adjusting the first weight value.
Teig teaches wherein lost processing comprises adjusting the first weight value (Col. 7 Lines 53-56, Col. 5 Lines 36-39, Fig. 2 – 200-210, Col. 6 Lines 39-42 “Each value in one of the filters is a weight value that is trained,” all as explained above with respect to claim 19, Col. 7 Lines 13-25, all as explained above with respect to claim 20).
No and Teig are analogous to the claimed invention as both are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the first weight value of Teig with the loss processing of No. The motivation to do so is to produce a training method that “reduces overfitting by preventing the network from learning the noise in the training set” (Teig, Col. 16 Lines 1-3).
Claims 23-24 recite a computer-implemented method that parallels the method claims of 2-3, respectively. Therefore, the analysis discussed above with respect to claims 2-3 also applies to claims 23-24, respectively. Accordingly, claims 23-24 are rejected based on substantially the same rationale as set forth above with respect to claims 2-3, respectively.
Regarding claim 25:
No teaches a computer program product (¶124) comprising: one or more computer-readable storage media (¶125).
No further teaches and program instructions stored on the one or more computer-readable storage media, the program instructions to perform operations (Fig. 1 – 200, 300, ¶125) comprising: performing, on a neural network comprising at least a first layer and a second layer, wherein the second layer is a non-polynomial layer (NPL) (Fig. 2 – 220, Fig. 3, ¶74) that is incompatible with homomorphic encryption (Fig. 3, ¶¶39, 41 all as explained above with respect to claim 1), a training process on a neural network resulting in a trained neural network (Fig. 1 – 10, ¶43 as described above with respect to claim 1)…
Claim 25 recites a product that parallels the method of claim 21. Therefore, the analysis discussed above with respect to claim 21 applies to claim 25. Accordingly, claim 25 is rejected based on substantially the same rationale as set forth above with respect to claim 21.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM M LEE whose telephone number is (571)272-4761. The examiner can normally be reached Mon-Fri. 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM M LEE/
Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights