Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 09/23/2025 has been entered.
Response to Remarks
Claim Rejections – 35 U.S.C. 103
Applicant’s prior art arguments have been fully considered but they are not persuasive.
Applicant argues (pgs. 7-8) that Soceanu and Chen, alone or in any combination, do not teach the amended limitations regarding:
extend one or more layers of the machine learning network to include additional neurons based on analysis of their respective list of HE packings, wherein the analysis comprises detecting a list of HE packings having one or more empty slots and occupying each of the one or more empty slots with an additional neuron weight, as recited by amended claim 1.
In particular, Applicant asserts that Chen, at most, discloses the performance of a grid search to tune hyperparameters and increase a number of hidden nodes of a model in order to increase model accuracy and depicts how model accuracy increases as the number of hidden layer nodes increases. Applicant asserts that Chen does not teach the amended limitation as cited above.
Examiner respectfully disagrees. First, refer to Page 6, of Figure “Example CryptoNets neural network”, where Chen teaches code that creates a CryptoNets model. On the last three lines of the code in this figure, there are two dense layers of different size (one has 100 neurons while the other has 10 neurons). Furthermore, there is an activation function that is created. Next, refer to Page 8, of Figure “Modified shallow model”, where Chen teaches code that creates a modified shallow model. Chen elaborates, “We can define a new model that takes out the convolutional and first activation and replaces it with a fully connected layer” (Chen, Page 8, Paragraph 2). Since a fully connected layer is one where every neuron is connected to the preceding and next layer, when a fully connected layer is introduced in lieu of an existing one, it usually comes with more neurons. Indeed, in the aforementioned “Modified shallow model” figure, the new layer has 400 neurons. Thus, if a layer with fewer neurons is analogous to one with many neurons but with many deactivated or set to zero, it follows that the operation that Chen teaches is one that detects a layer with empty slots (akin to layer with fewer neurons) and fill it with neurons (akin to fully connected layer). Note that a layer with few neurons is mathematically equivalent to that of many neurons with the corresponding non-existent neurons to be set to zero. If there is any doubt about this, refer to Page 6, Paragraph 2 of Chen: “once trained, the intermediate AveragePooling, Conv, AveragePooling, and Dense layers are combined into one Dense layer (i.e. all these layers can be expressed as fully connected matrices and then multiplied).” Although this step happens post training, the principle of being able to express different layer of different sizes as fully connected matrices lends credence to the idea that layers with few neurons are equivalent to dense layers with active corresponding neurons and remaining neurons set to zero.
The foregoing applies to all independent claims and their dependent claims.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-4, 6-8, 10-11, 13, 15, 17-18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Soceanu et al. (US 20220329407 A1) hereinafter known as Soceanu in view of Chen (“Faster Neural Networks on Encrypted Data with Intel HE Transformer and Tensorflow”) hereinafter known as Chen.
Regarding independent claim 1, Soceanu teaches:
A system, comprising one or more processors, one or more computer readable memories, one or more computer readable storage medium, and program instructions stored on at least one of the one or more computer readable storage medium for execution by at least one of the one or more processors, via at least one of the one or more computer readable memories, to cause at least one of the one or more processors to: receive a machine learning network and a selected homomorphic encryption (HE) packing framework; (Soceanu ¶ [0107]: “For example, such compilers exist that take a neural network inference engine … can use tile tensor shapes to decide how to pack each element of the network.” Soceanu teaches a processor that receives a machine learning network. Soceanu ¶ [0108]: “(i) the user can supply a ‘tile tensor shape’, which is meta data describing the packing scheme.” Soceanu teaches a HE processing framework.)
generate a respective list of HE packings for each layer of the machine learning network based on the selected HE packing framework; (Soceanu ¶ [0035]: “The output [of the packing process] is a tile tensor data structure containing a set of tiles filled with data copied from the tensor and arranged within the tiles according to the tile tensor shape.” Soceanu teaches generating a tile tensor data structure, which is a list of tiles. Soceanu ¶ [0104]: “layers will all be packed to tile tensors … for example, consider a neural network with 3 fully connected layers … first layer receives as input a vector of length 40, and outputs a vector of length 30. The second layer, input a vector of length 30, and outputs a vector of length 20. The third layer, input a vector of length 20, and outputs a vector of length 10” Soceanu teaches that the layers are packed into the tile tensors and gives an example of compressing each layer in the neural network, one layer at a time. Soceanu ¶ [0119]: “the tensors involved in the context of homomorphic encryption ciphertexts (computations under encryption) can be packed into tile tensors” Soceanu teaches that the packings are based on homomorphic encryption, making them HE packings.)
…
…
Soceanu does not teach:
…
…
extend one or more layers of the machine learning network to include additional neurons based on the analysis of their respective list of HE packings, wherein the analysis comprises detecting a list of HE packings having one or more empty slots and occupying each of the one or more empty slots with an additional neuron weight;
and train the extended machine learning network.
However, Chen teaches:
…
…
extend one or more layers of the machine learning network to include additional neurons based on the analysis of their respective list of HE packings, wherein the analysis comprises detecting a list of HE packings having one or more empty slots and occupying each of the one or more empty slots with an additional neuron weight; (Chen [Page 5, Paragraph 1]: “One of the most commonly used libraries for HE is Microsoft SEAL, which implements the CKKS LHE scheme. It’s possible to encrypt neural networks with Microsoft SEAL” Chen teaches that the neural network is encrypted using Microsoft SEAL, which uses HE. Chen [Page 8, Paragraph 4]: “We can do grid search to tune hyperparameters and also to adjust the architecture: namely the number of hidden nodes in the first fully connected layer.” Chen teaches running grid search to change the number of neurons of the encrypted layer, based on the accuracy of the HE packings. Chen [Page 9, Figure “Shallow Model Accuracy”]: Chen teaches that the change of the number of neurons is an increase (extending the layer), as this increases the accuracy of the HE packings. Chen [Page 6, Figure “Example CryptoNets neural network”]: On the last three lines of the code in this figure, there are two dense layers of different size (one has 100 neurons while the other has 10 neurons). Furthermore, there is an activation function that is created. Next, refer to Page 8, of Figure “Modified shallow model”, where Chen teaches code that creates a modified shallow model. Chen elaborates, “We can define a new model that takes out the convolutional and first activation and replaces it with a fully connected layer” (Chen, Page 8, Paragraph 2). If a layer with fewer neurons is analogous to one with many neurons but with many deactivated or set to zero, it follows that the operation that Chen teaches is one that detects a layer with empty slots (akin to layer with fewer neurons) and fill it with neurons (akin to fully connected layer).)
and train the extended machine learning network. (Chen [Page 9, Figure titled “Results of hyperparameter search”]: Chen teaches the training accuracy of the neural network as it is being extended.)
Soceanu and Chen are in the same field of endeavor as the present invention, as the
references are directed to improving the efficiency of operations regarding homomorphic encrypted neural networks. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine HE packing the neural network as taught in Soceanu with extend the neural network based on the HE packings as taught in Chen. Chen provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Soceanu to include teachings of Chen because the combination would allow for creating a HE encrypted neural network with additional nodes. This has the potential benefit of being able to increase the accuracy of the HE encrypted neural network while maintaining a similar runtime.
Regarding dependent claim 3, Soceanu and Chen teach:
The system of claim 1,
Soceanu teaches:
wherein the at least one of the one or more processors is to receive an optimization parameter and generate the list of HE packings based on the optimization parameter. (Soceanu ¶ [0108]: “(i) the user can supply a ‘tile tensor shape’, which is meta data describing the packing scheme.” Soceanu teaches receiving an optimization parameter, the tile tensor shape, which is used for the generation of HE packings.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 4, Soceanu and Chen teach:
The system of claim 1,
Soceanu teaches:
wherein at least one of the one or more processors is to generate a plurality of lists of HE packings corresponding to a plurality of received optimization parameters. (Soceanu ¶ [0109]: “Some embodiments of the present invention may include one, or more, of the following operations, features, characteristics and/or advantages: (i) a general technique to pack n dimensional array to n dimensional chunks, where the user can specify any dimension to be replicated; (ii) a single algorithm for summing over dimension i of an n-dimensional tile, and extending it to n-dimensional array of tiles, thus extending it to sum over a tile tensor; (iii) a general technique tensor contractions that extends to any number of dimensions (this is useful for example in the context of neural network inference); (iv) ability to efficiently compute matrix-matrix multiplication, and consecutive sequence of; (v) maintains the encrypted data alongside meta data describing the packing details, and having operators that act on both; (vi) human-readable notation for describing the meta data; and/or (vii) automatic optimizations.” Soceanu teaches a multitude of optimizations (advantages) that can be used to generate the plurality of lists of HE packings.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 6, Soceanu and Chen teach:
The system of claim 1,
Soceanu teaches:
wherein the received machine learning network is pretrained. (Soceanu ¶ [0107]: “For example, such compilers exist that take a neural network inference engine.” Soceanu teaches that the received machine learning network is an inference engine, meaning that it is necessarily trained already.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 7, Soceanu and Chen teach:
The system of claim 1,
wherein the received machine learning network comprises an HE-friendly deep neural network (DNN). (Soceanu ¶ [0006]: “The neural network is made of tensors.” Soceanu teaches that the received machine learning network is a deep neural network, as it has multiple tensors and each tensor is a multi-dimensional array. Soceanu ¶ [0065]: “Using this equivalence (“homomorphic”) property, computers can perform complicated sequences of operations on tile tensors, such as a sequence of matrix-vector multiplications, or neural network inference, and in the end retrieve the correct result as a tensor. Thus, for example neural network inference can be performed efficiently under encryption.” Soceanu teaches that the neural network inference is efficiently performed under encryption, or is HE-friendly.)
The reasons to combine are substantially similar to those of claim 1.
Independent claim 8, (method) is substantially similar to claim 1 (system), but has the following additional elements:
Soceanu teaches:
A computer-implemented method (Soceanu ¶ [0116]: “Computer: any device with significant data processing.” Soceanu teaches a computer.)
a selected homomorphic encryption (HE) packing framework (Soceanu ¶ [0108]: “(i) the user can supply a ‘tile tensor shape’, which is meta data describing the packing scheme.” Soceanu teaches a HE packing framework.)
The reasons to combine are substantially similar to those of claim 1.
Claims 10-11 are rejected on the same grounds under 35 U.S.C. 103 as claims 3-4, as they are
substantially similar, respectively. Mutatis mutandis.
Regarding dependent claim 13, Soceanu and Chen teach:
The computer-implemented method of claim 8,
Chen teaches:
wherein extending the machine learning network comprises adding neurons to a layer of the machine learning network. (Chen [Page 8, Paragraph 4]: “adjust the architecture: namely the number of hidden nodes in the first fully connected layer.” Chen teaches extending the network to increase the number of neurons (nodes) based on the accuracy of the HE packings.)
The reasons to combine are substantially similar to those of claim 1.
Independent claim 15, (product) is substantially similar to claim 8 (method), but has the following additional elements:
Soceanu teaches:
A computer program product … the computer program product comprising a computer-readable storage medium having program code embodied thereon, the program code executable by a processor to cause the processor to: (Soceanu ¶ [0116]: “Computer: any device with significant data processing.” Soceanu teaches a computer processor. Soceanu ¶ [0052]: “program 300 is stored in persistent storage.” Soceanu teaches computer readable storage with program.)
Chen teaches:
… for training machine learning networks, … (Chen [Page 5, Paragraph 4]: “In the training step, we define a model architecture and then train as usual using Tensorflow.” Chen teaches that the architecture trains neural networks.)
The reasons to combine are substantially similar to those of claim 1.
Claims 17, 18, 20 are rejected on the same grounds under 35 U.S.C. 103 as claims 3, 4, 13 as they are substantially similar, respectively. Mutatis mutandis.
Claims 2, 9, 14, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Soceanu in view of Chen in view of Nandakumar et al. (“Towards Deep Neural Network Training on Encrypted Data”) hereinafter known as Nandakumar.
Regarding dependent claim 2, Soceanu and Chen teach:
The system of claim 1,
Soceanu and Chen do not teach:
wherein the at least one of the one or more processors is further caused to: pack the trained extended machine learning network into HE ciphertexts;
and run the packed machine learning network via the HE packing framework.
However, Nandakumar teaches:
wherein the at least one of the one or more processors is further caused to: pack the trained extended machine learning network into HE ciphertexts; (Nandakumar [Page 46, Column 1, Paragraph 5]: “we packed different dimensions of a input sample/weight vector into a single ciphertext” Nandakumar teaches packing a neural network into HE ciphertexts. Nandakumar [Page 44, Column 2, Paragraph 3]: “Firstly, we present the full 784-dimensional (28 × 28) input to the neural network (denoted as NN1), which contained 128 and 32 neurons in the two hidden layers … we also consider a much smaller network (denoted as NN2) … containing 32 and 16 neurons in the two hidden layers.” Nandakumar teaches that this packing is done on an extended machine learning network with more neurons.)
and run the packed machine learning network via the HE packing framework. (Nandakumar [Page 45, Column 1, Paragraph 4]: “to estimate the classification accuracy of the proposed FHE-NIT solution, we quantize all the values into fixed-point signed integers.” Nandakumar teaches running the packed machine learning network via a HE framework that quantizes the values.)
Nandakumar is in the same field of endeavor as the present invention, since it is directed to packing an extended neural network into HE ciphertexts and running it. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine extending the neural network based on HE packings taught in Soceanu as modified by Chen with packing the neural network after extending as taught in Nandakumar. Nandakumar provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Soceanu as modified by Chen to include teachings of Nandakumar because the combination would make the resulting neural network more friendly to HE calculations. This has the potential benefit of increasing the accuracy of HE operations in the neural network without compromising significantly on speed.
Claim 9 is rejected on the same grounds under 35 U.S.C. 103 as claim 2, as they are
substantially similar. Mutatis mutandis.
Regarding dependent claim 14, Soceanu and Chen teach:
The computer-implemented method of claim 8,
Soceanu and Chen do not teach:
further comprising packing the extended machine learning network using an HE packing of the list of HE packings that corresponds to a received optimization parameter.
However, Nandakumar teaches:
further comprising packing the extended machine learning network using an HE packing of the list of HE packings that corresponds to a received optimization parameter. (Nandakumar [Page 46, Column 1, Paragraph 5]: “In the optimized approach, we packed the same dimension of multiple input samples into a single ciphertext and replicate each weight parameter into all the slots of a weight vector” Nandakumar teaches that the trained extended machine learning network is packed into HE ciphertexts. This packing is achieved by the packings of others with same dimension, to optimize based on the slots of a weight vector.)
The reasons to combine are substantially similar to those of claim 2.
Claim 16 is rejected on the same grounds under 35 U.S.C. 103 as claim 2 as they are substantially similar. Mutatis mutandis.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Soceanu in view of Chen in view of Minixhofer et al. (US 20220058472 A1) hereinafter known as Minixhofer.
Regarding dependent claim 5, Soceanu and Chen teach:
The system of claim 1,
wherein the at least one of the one or more processors is to extend the machine learning network by adding neurons to a layer of the machine learning network, … (Chen [Page 8, Paragraph 4]: “adjust the architecture: namely the number of hidden nodes in the first fully connected layer.” Chen teaches extending the network to increase the number of neurons (nodes) based on the accuracy of the HE packings.)
Soceanu and Chen do not teach:
… wherein the neurons are initially set to have weights of zero.
However, Minixhofer teaches:
… wherein the neurons are initially set to have weights of zero. (Minixhofer ¶ [0028]: “Any number of hidden neurons up to k in each layer (in case a layer needs fewer than n neurons, the corresponding weights can be set to zero)” Minixhofer teaches initializing the weights of the neurons in the neural network to zero.)
Minixhofer is in the same field of endeavor as the present invention, since it is directed to implementing structures of neural networks, namely neurons in layers. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine extending the neural network based on HE packings taught in Soceanu as modified by Chen with setting the weights of the additional neurons to zero as taught in Minixhofer. Minixhofer provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Soceanu as modified by Chen to include teachings of Minixhofer because the combination would allow for an extended neural network with additional neurons having weight zero. This has the potential benefit of being able to train the extended neural network with a clean slate to determine weights that create an accurate model.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYU HYUNG HAN whose telephone number is (703) 756-5529. The examiner can normally be reached on MF 9-5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Kyu Hyung Han/
Examiner
Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123