DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 07/19/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The disclosure is objected to because of the following informalities:
In paragraph [0031], the code text is blurry and illegible. The specification should be amended to ensure that the text is legible.
Appropriate correction is required.
Claim Objections
Claims 6-7 and 13-15 are objected to under 37 CFR 1.75(c) as being in improper form because a multiple dependent claim cannot depend from any other multiple dependent claim. See MPEP § 608.01(n). Accordingly, the claims have not been further treated on the merits.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4-5 and 11-12 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 4 recites the limitation “en = [x - sum(i=1..n-1,wiei) + b]/wn” in line 3. This limitation lacks clarity because “en = [x - sum(i=1..n-1,wiei) + b]/wn” renders the limitation uncertain regarding what “x” is.
Claim 5 recites the limitation “the order of the inputs” in lines 1-2. There is insufficient antecedent basis for this limitation in the claim. For examination purposes, “the order of the inputs” has been interpreted as “an order of the inputs”.
Claim 11 recites the limitation “en = [x - sum(i=1..n-1,wiei) + b]/wn” in line 3. This limitation lacks clarity because “en = [x - sum(i=1..n-1,wiei) + b]/wn” renders the limitation uncertain regarding what “x” is.
Claim 12 recites the limitation “the order of the inputs” in line 1. There is insufficient antecedent basis for this limitation in the claim. For examination purposes, “the order of the inputs” has been interpreted as “an order of the inputs”.
Claim 12 recites the limitation “the secured input layer” in line 2. There is insufficient antecedent basis for this limitation in the claim. For examination purposes, “the secured input layer” has been interpreted as “a secured input layer”.
Dependent claim 5 is rejected based on being directly or indirectly dependent on rejected claim 4.
Dependent claim 12 is rejected based on being directly or indirectly dependent on rejected claim 11.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 8-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claims do not fall within at least one of the four categories of patent eligible subject matter because, under their broadest reasonable interpretation in light of the specification, the claims are directed to software per se.
Regarding claims 8-12, the claims are directed to a “secured model implemented as a neural network”. The claims lack any structure at all and it appears that one of ordinary skill in the art could interpret the claims as software per se. Such language points to software per se when there is no language in the claim or specification by which the claim elements can be made functional and statutory and the specification provide any indication that the “secured model implemented as a neural network” is anything other than software.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3 and 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over Farhady Ghalaty et al. (US 20260/050786 A1) in view of Zhang et al. ("FracBNN: Accurate and FPGA-Efficient Binary Neural Networks with Fractional Activations") and further in view of Liu et al. ("Molecular and DNA Artificial Neural Networks via Fractional Coding").
Regarding Claim 1,
Farhady Ghalaty et al. teaches a method for securing a model implemented as a neural network (Fig. 6; [0087]: "FIG. 6 is an example flow chart of the process 600 for compiling an input neural network with obfuscating network structures" teaches a method for adding obfuscating network layers to a neural network model. [0008]: "The techniques described in this specification can enhance the security for neural networks implemented on hardware devices" teaches that the method is for securing the neural network model), the method comprising:
receiving an input specification data structure specifying inputs of an original input layer of an original neural network, wherein the original neural network is constructed to execute a model (Fig. 6; [0087]-[0088]: "FIG. 6 is an example flow chart of the process 600 for compiling an input neural network with obfuscating network structures … The system receives data representing a machine learning model (610). The machine learning model can include various types of machine learning models, for example, a neural network. Data representing a neural network can specify multiple inference operations. The neural network can include parameters specifying a sequence of multiple network layers, multiple nodes in each of the multiple layers, nodal weights and nodal operations for each node, and other structures associated with the neural network" teaches receiving data representing a machine learning model including a neural network (input specification data structure) specifying inputs (including variables and weights) of the different layers of the model (e.g. including the original input layer));
creating a protected input layer based on the fractionalized inputs and the weighting table whereby the protected input layer is operative to decode encoded inputs (Fig. 4; [0074]-[0076]: "FIG. 4 illustrates another example process 400 of obfuscating network structures associated with an input neural network. The obfuscating network structures can be determined by a secure machine learning model compiler when it compiles an input neural network … Similarly, the secure machine learning model compiler included in the described system can determine a critical layer 412 of an input neural network for compiling from a sequence of network layers … The critical layer 412 can include one or more nodes 406A-N, each having a nodal weight value and a corresponding nodal operation (e.g., nodal activation functions as described above) … The obfuscating network structures of the example process 400 include one or more obfuscating network layers 464 and/or 462. The secure machine learning model compiler is configured to modify the input neural network by adding the one or more obfuscating network layers 464 and/or 462 before and/or after the critical layer 412 according to a sequence" teaches creating obfuscating network layers (protected input layer) based on the input neural network data. Fig. 6; [0087]-[0092]: "FIG. 6 is an example flow chart of the process 600 for compiling an input neural network with obfuscating network structures … The system receives data representing a machine learning model (610). The machine learning model can include various types of machine learning models, for example, a neural network. Data representing a neural network can specify multiple inference operations. The neural network can include parameters specifying a sequence of multiple network layers, multiple nodes in each of the multiple layers, nodal weights and nodal operations for each node, and other structures associated with the neural network … Based on input data associated with the neural network, the system determines whether the neural network is security-sensitive, and/or has required a considerable amount of time and/or cost for training that satisfies a particular threshold. In response to determining that the neural network is security-sensitive, and/or has required a considerable amount of resources, the system determines to compile the neural network under a “secured mode,” as described above. In the “secured mode,” the system could obfuscate one or more measurable characteristics of the neural network by determining one or more obfuscating operations to be included in instructions in addition to inference operations specified by the neural network when compiling the neural network, and the one or more obfuscating operations, when performed with inference operations, could obfuscate one or more measurable characteristics of the neural network" teaches the input neural network data including inputs (including variables (e.g. fractionalized inputs) and weights (e.g. weighting table)) of the different layers of the model (e.g. including the original input layer) and determines to perform obfuscating network layer (protected input layer) operations (e.g. encoded inputs are decoded to perform obfuscating operations)); and
connecting the protected input layer to the original input layer to thereby create a secured model (Fig. 4; [0074]-[0076]: "FIG. 4 illustrates another example process 400 of obfuscating network structures associated with an input neural network. The obfuscating network structures can be determined by a secure machine learning model compiler when it compiles an input neural network … The obfuscating network structures of the example process 400 include one or more obfuscating network layers 464 and/or 462. The secure machine learning model compiler is configured to modify the input neural network by adding the one or more obfuscating network layers 464 and/or 462 before and/or after the critical layer 412 according to a sequence" teaches a secured neural network model that comprises obfuscating network layers (protected input layer) connected to the original input layer).
Farhady Ghalaty et al. does not appear to explicitly teach fractionalizing the inputs to thereby create fractionalized inputs; specifying weightings that define how much each fractionalized input contributes to neurons in the original input layer; storing the weightings in a weighting table.
However, Zhang et al. teaches fractionalizing the inputs to thereby create fractionalized inputs (Fig. 6; Section. 3.2, first-third paragraphs: "A binary input layer can reduce the resource consumption of an FPGA accelerator, since a separate floating- or fixed-point convolution engine is no longer required. The challenge of binarizing both weights and activations in the input layer is the lack of input channels … It is therefore necessary to split the images into more channels …
PNG
media_image1.png
156
328
media_image1.png
Greyscale
… As shown in Figure 6a, a natural way of enriching the channels is to treat each pixel as an 8-dimensional binary vector since pixels are 8-bit fixed-point numbers … we propose to use thermometer encoding to transform a pixel to a thermometer vector. Previous work has used thermometer encoding to resist adversarial attacks to neural networks [5]. There is also a study [15] that binarizes the input images but the dimension of the encoded vector must be a power of two. Here we use thermometer encoding to binarize the input layer in an end-to-end trainable BNN, and our method supports a flexible vector length" teaches encoding (fractionalizing) inputs into vectors of binary values (fractionalized inputs)).
Farhady Ghalaty et al. is analogous to the claimed invention because it is directed towards securing neural networks.
Zhang et al. is analogous to the claimed invention because it is directed towards neural network input encoding.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate fractionalizing the inputs to thereby create fractionalized inputs as taught by Zhang et al. to the disclosed invention of Farhady Ghalaty et al.
One of ordinary skill in the art would have been motivated to make this modification to "help with preserving the feature similarity, thus incurring minimal accuracy degradation." (Zhang et al. Section 1, last paragraph).
Farhady Ghalaty et al. in view of Zhang et al. does not appear to explicitly teach specifying weightings that define how much each fractionalized input contributes to neurons in the original input layer; storing the weightings in a weighting table.
However, Liu et al. teaches specifying weightings that define how much each fractionalized input contributes to neurons in the original input layer (Section II, first paragraph: "In fractional coding, each value is encoded using two molecules. For example, value X can be encoded as X1/(X1+X0) where X1 and X0, respectively, represent the molecules of type-1 and type-0. In stochastic logic, X1 and X0, respectively, represent the number of 1 and 0 bits in a unary bit stream" teaches using fractional coding to encode each input value X (e.g. encoding fractional inputs). Table III; Section VI. B, first paragraph: "The molecular ANN classifier is tested using an ANN with one hidden layer containing five neurons and four neurons for the input layer … The testing data contains 10422 samples and each sample contains a vector with 4 features (x=x1,x2,x3,x4) and a bias term, b. Since the range of the input features should be [−1,1] under the constraint of bipolar format representation, a linear mapping is performed on the input features. Consider the input data as a 10422×4 matrix X, where the number of rows (10422) and columns (4) represent the number of inputs data samples and the number of features, respectively. The linear mapping is performed for all samples as follows:
PNG
media_image2.png
88
532
media_image2.png
Greyscale
where max(Xi) and min(Xi) represent the maximum and minimum magnitudes of the ith feature among all 10422 samples. After this linear mapping, each element in a column of the input matrix X has mean 0 and the dynamic range of [−1,1]. The histogram of 41688 features from 10422 4-dimensional feature vectors after linear mapping is shown in Fig. 16 …
PNG
media_image3.png
162
496
media_image3.png
Greyscale
… Table III lists the weight matrices and bias vectors of the optimized ANN model, where wI represents the connection weight matrix of the input-hidden layer connection, wh represents the hidden layer-output connection, bh represents the bias column vector for the hidden neurons, and bo is the bias for the output neuron" teaches that the fractionalized inputs X have specified weights (weightings) (weights are shown in Table III) for contributions to neurons in the input layer);
storing the weightings in a weighting table (Table III; Section VI. B, first paragraph: "The molecular ANN classifier is tested using an ANN with one hidden layer containing five neurons and four neurons for the input layer … The testing data contains 10422 samples and each sample contains a vector with 4 features (x=x1,x2,x3,x4) and a bias term, b. Since the range of the input features should be [−1,1] under the constraint of bipolar format representation, a linear mapping is performed on the input features. Consider the input data as a 10422×4 matrix X, where the number of rows (10422) and columns (4) represent the number of inputs data samples and the number of features, respectively. The linear mapping is performed for all samples as follows:
PNG
media_image2.png
88
532
media_image2.png
Greyscale
where max(Xi) and min(Xi) represent the maximum and minimum magnitudes of the ith feature among all 10422 samples. After this linear mapping, each element in a column of the input matrix X has mean 0 and the dynamic range of [−1,1]. The histogram of 41688 features from 10422 4-dimensional feature vectors after linear mapping is shown in Fig. 16 …
PNG
media_image3.png
162
496
media_image3.png
Greyscale
… Table III lists the weight matrices and bias vectors of the optimized ANN model, where wI represents the connection weight matrix of the input-hidden layer connection, wh represents the hidden layer-output connection, bh represents the bias column vector for the hidden neurons, and bo is the bias for the output neuron" teaches that the weights (weightings) are stored in a weighting table (Table III)).
Farhady Ghalaty et al. is analogous to the claimed invention because it is directed towards securing neural networks.
Zhang et al. and Liu et al. are analogous to the claimed invention because they are directed towards neural network input encoding.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate specifying weightings that define how much each fractionalized input contributes to neurons in the original input layer; storing the weightings in a weighting table as taught by Liu et al. to the disclosed invention of Farhady Ghalaty et al. in view of Zhang et al.
One of ordinary skill in the art would have been motivated to make this modification because "the performance of molecular ANN using linear mapping for input data is close to the ideal results" (Liu et al. Section VI. B, last paragraph paragraph).
Regarding Claim 2,
Farhady Ghalaty et al. in view of Zhang et al. and further in view of Liu et al. teaches the method of claim 1.
In addition, Zhang et al. further teaches wherein the fractionalizing comprises selecting a number N of values of e1 to en to represent an input value (Fig. 6; Section. 3.2, first-third paragraphs: "A binary input layer can reduce the resource consumption of an FPGA accelerator, since a separate floating- or fixed-point convolution engine is no longer required. The challenge of binarizing both weights and activations in the input layer is the lack of input channels … It is therefore necessary to split the images into more channels …
PNG
media_image1.png
156
328
media_image1.png
Greyscale
… As shown in Figure 6a, a natural way of enriching the channels is to treat each pixel as an 8-dimensional binary vector since pixels are 8-bit fixed-point numbers … we propose to use thermometer encoding to transform a pixel to a thermometer vector. Previous work has used thermometer encoding to resist adversarial attacks to neural networks [5]. There is also a study [15] that binarizes the input images but the dimension of the encoded vector must be a power of two. Here we use thermometer encoding to binarize the input layer in an end-to-end trainable BNN, and our method supports a flexible vector length" teaches encoding (fractionalizing) inputs into vectors of binary values (fractionalized inputs e1 to en) to represent the input value (e.g. pixel 109 in Fig. 6) based on a selected number of bits (N number of values) (e.g. N=8 in Fig. 6)).
Farhady Ghalaty et al. is analogous to the claimed invention because it is directed towards securing neural networks.
Zhang et al. and Liu et al. are analogous to the claimed invention because they are directed towards neural network input encoding.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the fractionalizing comprises selecting a number N of values of e1 to en to represent an input value as taught by Zhang et al. to the disclosed invention of Farhady Ghalaty et al. in view of Liu et al.
One of ordinary skill in the art would have been motivated to make this modification to "help with preserving the feature similarity, thus incurring minimal accuracy degradation." (Zhang et al. Section 1, last paragraph).
Regarding Claim 3,
Farhady Ghalaty et al. in view of Zhang et al. and further in view of Liu et al. teaches the method of claim 2.
In addition, Liu et al. further teaches wherein the weighting table includes N+1 fields respectively storing weightings w1 to wn and a bias value b (Table III; Section VI. B, first paragraph: "The molecular ANN classifier is tested using an ANN with one hidden layer containing five neurons and four neurons for the input layer … The testing data contains 10422 samples and each sample contains a vector with 4 features (x=x1,x2,x3,x4) and a bias term, b. Since the range of the input features should be [−1,1] under the constraint of bipolar format representation, a linear mapping is performed on the input features. Consider the input data as a 10422×4 matrix X, where the number of rows (10422) and columns (4) represent the number of inputs data samples and the number of features, respectively. The linear mapping is performed for all samples as follows:
PNG
media_image2.png
88
532
media_image2.png
Greyscale
where max(Xi) and min(Xi) represent the maximum and minimum magnitudes of the ith feature among all 10422 samples. After this linear mapping, each element in a column of the input matrix X has mean 0 and the dynamic range of [−1,1]. The histogram of 41688 features from 10422 4-dimensional feature vectors after linear mapping is shown in Fig. 16 …
PNG
media_image3.png
162
496
media_image3.png
Greyscale
… Table III lists the weight matrices and bias vectors of the optimized ANN model, where wI represents the connection weight matrix of the input-hidden layer connection, wh represents the hidden layer-output connection, bh represents the bias column vector for the hidden neurons, and bo is the bias for the output neuron" teaches that the weights (weightings) are stored in a weighting table (Table III) with N+1 fields (N=4) that represent w1 to wn (w1 to w4 in Table III) and a bias (bh)).
Farhady Ghalaty et al. is analogous to the claimed invention because it is directed towards securing neural networks.
Zhang et al. and Liu et al. are analogous to the claimed invention because they are directed towards neural network input encoding.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the weighting table includes N+1 fields respectively storing weightings w1 to wn and a bias value b as taught by Liu et al. to the disclosed invention of Farhady Ghalaty et al. in view of Zhang et al.
One of ordinary skill in the art would have been motivated to make this modification because "the performance of molecular ANN using linear mapping for input data is close to the ideal results" (Liu et al. Section VI. B, last paragraph paragraph).
Regarding Claim 8,
Farhady Ghalaty et al. teaches a secured model implemented as a neural network (Fig. 4; [0074]: "FIG. 4 illustrates another example process 400 of obfuscating network structures associated with an input neural network. The obfuscating network structures can be determined by a secure machine learning model compiler when it compiles an input neural network. The secure machine learning model compiler can be equivalent to the secure machine learning model compiler 105 of FIG. 1 and/or the secure machine learning model compiler 200 of FIG. 2" teaches a secured neural network model with added obfuscating network layers (protected layer) to an original neural network by a secure machine learning model compiler), the secured model comprising:
an unsecured model implemented as a neural network having an original input layer, and output layer, and at least one hidden layer (Fig. 4; [0074]-[0075]: "FIG. 4 illustrates another example process 400 of obfuscating network structures associated with an input neural network … Similarly, the secure machine learning model compiler included in the described system can determine a critical layer 412 of an input neural network for compiling from a sequence of network layers (414, 412, and 416), as shown in FIG. 4. The one or more network layers 414 are preceding neural network layers that precede the critical layer 412 in the sequence, and the one or more network layers 416 are succeeding network layers 316 that succeed the critical layer 412 in the sequence. The critical layer 412 can include one or more nodes 406A-N, each having a nodal weight value and a corresponding nodal operation (e.g., nodal activation functions as described above)" teaches a secured neural network model that comprises an unsecured original neural network including an input layer (414), hidden layer (412), and output layer (416). [0003]: "Neural networks can employ one or more layers of nodes representing multiple operations, e.g., vector or matrix operations … Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with the current values of a respective set of network parameters" teaches that the original neural network model includes, an input layer, hidden layer, and output layer); and
a protected input layer connected to the original input layer (Fig. 4; [0074]-[0076]: "FIG. 4 illustrates another example process 400 of obfuscating network structures associated with an input neural network. The obfuscating network structures can be determined by a secure machine learning model compiler when it compiles an input neural network … Similarly, the secure machine learning model compiler included in the described system can determine a critical layer 412 of an input neural network for compiling from a sequence of network layers … The critical layer 412 can include one or more nodes 406A-N, each having a nodal weight value and a corresponding nodal operation (e.g., nodal activation functions as described above) … The obfuscating network structures of the example process 400 include one or more obfuscating network layers 464 and/or 462. The secure machine learning model compiler is configured to modify the input neural network by adding the one or more obfuscating network layers 464 and/or 462 before and/or after the critical layer 412 according to a sequence" teaches a secured neural network model that comprises obfuscating network layers (protected input layer) connected to the original input layer),
where the protected input layer is created by receiving an input specification data structure specifying inputs of the original input layer (Fig. 6; [0087]-[0088]: "FIG. 6 is an example flow chart of the process 600 for compiling an input neural network with obfuscating network structures … The system receives data representing a machine learning model (610). The machine learning model can include various types of machine learning models, for example, a neural network. Data representing a neural network can specify multiple inference operations. The neural network can include parameters specifying a sequence of multiple network layers, multiple nodes in each of the multiple layers, nodal weights and nodal operations for each node, and other structures associated with the neural network" teaches receiving data representing a machine learning model (input specification data structure) specifying inputs (including variables and weights) of the different layers of the model (e.g. including the original input layer)),
whereby the protected input layer is operative to decode encoded inputs (Fig. 2; [0058]: "As shown in FIG. 2, the secure machine learning model compiler 105 is configured to process input data and generate output data by processing the input data. As described above, the input data can include one or more machine learning models (e.g., neural networks) encoded in high-level programming languages. The output data can include compiled machine learning models encoded in a machine-readable low level programming language (e.g., binary code)" teaches that the machine learning model input data is encoded for input to the secure machine learning model compiler. Fig. 6; [0087]-[0092]: "FIG. 6 is an example flow chart of the process 600 for compiling an input neural network with obfuscating network structures … The system receives data representing a machine learning model (610). The machine learning model can include various types of machine learning models, for example, a neural network. Data representing a neural network can specify multiple inference operations. The neural network can include parameters specifying a sequence of multiple network layers, multiple nodes in each of the multiple layers, nodal weights and nodal operations for each node, and other structures associated with the neural network … Based on input data associated with the neural network, the system determines whether the neural network is security-sensitive, and/or has required a considerable amount of time and/or cost for training that satisfies a particular threshold. In response to determining that the neural network is security-sensitive, and/or has required a considerable amount of resources, the system determines to compile the neural network under a “secured mode,” as described above. In the “secured mode,” the system could obfuscate one or more measurable characteristics of the neural network by determining one or more obfuscating operations to be included in instructions in addition to inference operations specified by the neural network when compiling the neural network, and the one or more obfuscating operations, when performed with inference operations, could obfuscate one or more measurable characteristics of the neural network" teaches that the secure machine learning model compiler receives the input data representing a machine learning model specifying inputs (including variables and weights) and determines to perform obfuscating network layer (protected input layer) operations (e.g. encoded inputs are decoded to perform obfuscating operations)).
Farhady Ghalaty et al. does not appear to explicitly teach fractionalizing the inputs to thereby create fractionalized inputs, specifying weightings that define how much each fractionalized input contributes to neurons in the original input layer, storing the weightings in a weighting table.
However, Zhang et al. teaches fractionalizing the inputs to thereby create fractionalized inputs (Fig. 6; Section. 3.2, first-third paragraphs: "A binary input layer can reduce the resource consumption of an FPGA accelerator, since a separate floating- or fixed-point convolution engine is no longer required. The challenge of binarizing both weights and activations in the input layer is the lack of input channels … It is therefore necessary to split the images into more channels …
PNG
media_image1.png
156
328
media_image1.png
Greyscale
… As shown in Figure 6a, a natural way of enriching the channels is to treat each pixel as an 8-dimensional binary vector since pixels are 8-bit fixed-point numbers … we propose to use thermometer encoding to transform a pixel to a thermometer vector. Previous work has used thermometer encoding to resist adversarial attacks to neural networks [5]. There is also a study [15] that binarizes the input images but the dimension of the encoded vector must be a power of two. Here we use thermometer encoding to binarize the input layer in an end-to-end trainable BNN, and our method supports a flexible vector length" teaches encoding (fractionalizing) inputs into vectors of binary values (fractionalized inputs)).
Farhady Ghalaty et al. is analogous to the claimed invention because it is directed towards securing neural networks.
Zhang et al. is analogous to the claimed invention because it is directed towards neural network input encoding.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate fractionalizing the inputs to thereby create fractionalized inputs as taught by Zhang et al. to the disclosed invention of Farhady Ghalaty et al.
One of ordinary skill in the art would have been motivated to make this modification to "help with preserving the feature similarity, thus incurring minimal accuracy degradation." (Zhang et al. Section 1, last paragraph).
Farhady Ghalaty et al. in view of Zhang et al. does not appear to explicitly teach specifying weightings that define how much each fractionalized input contributes to neurons in the original input layer, storing the weightings in a weighting table.
However, Liu et al. teaches specifying weightings that define how much each fractionalized input contributes to neurons in the original input layer (Section II, first paragraph: "In fractional coding, each value is encoded using two molecules. For example, value X can be encoded as X1/(X1+X0) where X1 and X0, respectively, represent the molecules of type-1 and type-0. In stochastic logic, X1 and X0, respectively, represent the number of 1 and 0 bits in a unary bit stream" teaches using fractional coding to encode each input value X (e.g. encoding fractional inputs). Table III; Section VI. B, first paragraph: "The molecular ANN classifier is tested using an ANN with one hidden layer containing five neurons and four neurons for the input layer … The testing data contains 10422 samples and each sample contains a vector with 4 features (x=x1,x2,x3,x4) and a bias term, b. Since the range of the input features should be [−1,1] under the constraint of bipolar format representation, a linear mapping is performed on the input features. Consider the input data as a 10422×4 matrix X, where the number of rows (10422) and columns (4) represent the number of inputs data samples and the number of features, respectively. The linear mapping is performed for all samples as follows:
PNG
media_image2.png
88
532
media_image2.png
Greyscale
where max(Xi) and min(Xi) represent the maximum and minimum magnitudes of the ith feature among all 10422 samples. After this linear mapping, each element in a column of the input matrix X has mean 0 and the dynamic range of [−1,1]. The histogram of 41688 features from 10422 4-dimensional feature vectors after linear mapping is shown in Fig. 16 …
PNG
media_image3.png
162
496
media_image3.png
Greyscale
… Table III lists the weight matrices and bias vectors of the optimized ANN model, where wI represents the connection weight matrix of the input-hidden layer connection, wh represents the hidden layer-output connection, bh represents the bias column vector for the hidden neurons, and bo is the bias for the output neuron" teaches that the fractionalized inputs X have specified weights (weightings) (weights are shown in Table III) for contributions to neurons in the input layer),
storing the weightings in a weighting table (Table III; Section VI. B, first paragraph: "The molecular ANN classifier is tested using an ANN with one hidden layer containing five neurons and four neurons for the input layer … The testing data contains 10422 samples and each sample contains a vector with 4 features (x=x1,x2,x3,x4) and a bias term, b. Since the range of the input features should be [−1,1] under the constraint of bipolar format representation, a linear mapping is performed on the input features. Consider the input data as a 10422×4 matrix X, where the number of rows (10422) and columns (4) represent the number of inputs data samples and the number of features, respectively. The linear mapping is performed for all samples as follows:
PNG
media_image2.png
88
532
media_image2.png
Greyscale
where max(Xi) and min(Xi) represent the maximum and minimum magnitudes of the ith feature among all 10422 samples. After this linear mapping, each element in a column of the input matrix X has mean 0 and the dynamic range of [−1,1]. The histogram of 41688 features from 10422 4-dimensional feature vectors after linear mapping is shown in Fig. 16 …
PNG
media_image3.png
162
496
media_image3.png
Greyscale
… Table III lists the weight matrices and bias vectors of the optimized ANN model, where wI represents the connection weight matrix of the input-hidden layer connection, wh represents the hidden layer-output connection, bh represents the bias column vector for the hidden neurons, and bo is the bias for the output neuron" teaches that the weights (weightings) are stored in a weighting table (Table III)).
Farhady Ghalaty et al. is analogous to the claimed invention because it is directed towards securing neural networks.
Zhang et al. and Liu et al. are analogous to the claimed invention because they are directed towards neural network input encoding.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate specifying weightings that define how much each fractionalized input contributes to neurons in the original input layer, storing the weightings in a weighting table as taught by Liu et al. to the disclosed invention of Farhady Ghalaty et al. in view of Zhang et al.
One of ordinary skill in the art would have been motivated to make this modification because "the performance of molecular ANN using linear mapping for input data is close to the ideal results" (Liu et al. Section VI. B, last paragraph paragraph).
Regarding Claim 9,
Farhady Ghalaty et al. in view of Zhang et al. and further in view of Liu et al. teaches the secured model of claim 8.
In addition, Zhang et al. further teaches wherein the fractionalizing comprises selecting a number N of values of e1 to en to represent an input value (Fig. 6; Section. 3.2, first-third paragraphs: "A binary input layer can reduce the resource consumption of an FPGA accelerator, since a separate floating- or fixed-point convolution engine is no longer required. The challenge of binarizing both weights and activations in the input layer is the lack of input channels … It is therefore necessary to split the images into more channels …
PNG
media_image1.png
156
328
media_image1.png
Greyscale
… As shown in Figure 6a, a natural way of enriching the channels is to treat each pixel as an 8-dimensional binary vector since pixels are 8-bit fixed-point numbers … we propose to use thermometer encoding to transform a pixel to a thermometer vector. Previous work has used thermometer encoding to resist adversarial attacks to neural networks [5]. There is also a study [15] that binarizes the input images but the dimension of the encoded vector must be a power of two. Here we use thermometer encoding to binarize the input layer in an end-to-end trainable BNN, and our method supports a flexible vector length" teaches encoding (fractionalizing) inputs into vectors of binary values (fractionalized inputs e1 to en) to represent the input value (e.g. pixel 109 in Fig. 6) based on a selected number of bits (N number of values) (e.g. N=8 in Fig. 6)).
Farhady Ghalaty et al. is analogous to the claimed invention because it is directed towards securing neural networks.
Zhang et al. and Liu et al. are analogous to the claimed invention because they are directed towards neural network input encoding.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the fractionalizing comprises selecting a number N of values of e1 to en to represent an input value as taught by Zhang et al. to the disclosed invention of Farhady Ghalaty et al. in view of Liu et al.
One of ordinary skill in the art would have been motivated to make this modification to "help with preserving the feature similarity, thus incurring minimal accuracy degradation." (Zhang et al. Section 1, last paragraph).
Regarding Claim 10,
Farhady Ghalaty et al. in view of Zhang et al. and further in view of Liu et al. teaches the secured model of claim 9.
In addition, Liu et al. further teaches wherein the weighting table includes N+1 fields respectively storing weightings w1 to wn and a bias value b (Table III; Section VI. B, first paragraph: "The molecular ANN classifier is tested using an ANN with one hidden layer containing five neurons and four neurons for the input layer … The testing data contains 10422 samples and each sample contains a vector with 4 features (x=x1,x2,x3,x4) and a bias term, b. Since the range of the input features should be [−1,1] under the constraint of bipolar format representation, a linear mapping is performed on the input features. Consider the input data as a 10422×4 matrix X, where the number of rows (10422) and columns (4) represent the number of inputs data samples and the number of features, respectively. The linear mapping is performed for all samples as follows:
PNG
media_image2.png
88
532
media_image2.png
Greyscale
where max(Xi) and min(Xi) represent the maximum and minimum magnitudes of the ith feature among all 10422 samples. After this linear mapping, each element in a column of the input matrix X has mean 0 and the dynamic range of [−1,1]. The histogram of 41688 features from 10422 4-dimensional feature vectors after linear mapping is shown in Fig. 16 …
PNG
media_image3.png
162
496
media_image3.png
Greyscale
… Table III lists the weight matrices and bias vectors of the optimized ANN model, where wI represents the connection weight matrix of the input-hidden layer connection, wh represents the hidden layer-output connection, bh represents the bias column vector for the hidden neurons, and bo is the bias for the output neuron" teaches that the weights (weightings) are stored in a weighting table (Table III) with N+1 fields (N=4) that represent w1 to wn (w1 to w4 in Table III) and a bias (bh)).
Farhady Ghalaty et al. is analogous to the claimed invention because it is directed towards securing neural networks.
Zhang et al. and Liu et al. are analogous to the claimed invention because they are directed towards neural network input encoding.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the weighting table includes N+1 fields respectively storing weightings w1 to wn and a bias value b as taught by Liu et al. to the disclosed invention of Farhady Ghalaty et al. in view of Zhang et al.
One of ordinary skill in the art would have been motivated to make this modification because "the performance of molecular ANN using linear mapping for input data is close to the ideal results" (Liu et al. Section VI. B, last paragraph paragraph).
Claims 5 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Farhady Ghalaty et al. (US 20260/050786 A1) in view of Zhang et al. ("FracBNN: Accurate and FPGA-Efficient Binary Neural Networks with Fractional Activations") in view of Liu et al. ("Molecular and DNA Artificial Neural Networks via Fractional Coding") and further in view of Gardner et al. ("Encoding Spike Patterns in Multilayer Spiking Neural Networks").
Regarding Claim 5,
Farhady Ghalaty et al. in view of Zhang et al. and further in view of Liu et al. teaches the method of preceding claim 1, the method of preceding claim 2, and the method of preceding claim 3.
Farhady Ghalaty et al. in view of Zhang et al. and further in view of Liu et al. does not appear to explicitly teach further comprising randomizing the order of the inputs.
However, Gardner et al. teaches further comprising randomizing the order of the inputs (Page 10, fourth paragraph: "For the inputs, each binary value was encoded by a set of 50 Poisson spike trains with a mean firing rate of 6 Hz, predetermined at the start of each simulation run; hence, paired binary input values were represented by spike patterns over two groups of 50 neurons … Binary inputs were presented to the network episodically in a random order" teaches that the order of the binary encoded inputs (fractional inputs) is randomized).
Farhady Ghalaty et al. is analogous to the claimed invention because it is directed towards securing neural networks.
Zhang et al., Liu et al., and Gardner et al. are analogous to the claimed invention because they are directed towards neural network input encoding.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate further comprising randomizing the order of the inputs as taught by Gardner et al. to the disclosed invention of Farhady Ghalaty et al. in view of Zhang et al. and further in view of Liu et al.
One of ordinary skill in the art would have been motivated to make this modification to "highlight the advantages of using a fully temporal code based on multiple output spike-timings to reliably encode for input patterns" (Gardner et al. Page 22, fourth paragraph).
Regarding Claim 12,
Farhady Ghalaty et al. in view of Zhang et al. and further in view of Liu et al. teaches the secured model of claim 8, the secured model of claim 9, and the secured model of claim 10.
Farhady Ghalaty et al. in view of Zhang et al. and further in view of Liu et al. does not appear to explicitly teach wherein the order of the inputs is randomized in the secured input layer.
However, Gardner et al. teaches wherein the order of the inputs is randomized in the secured input layer (Page 10, fourth paragraph: "For the inputs, each binary value was encoded by a set of 50 Poisson spike trains with a mean firing rate of 6 Hz, predetermined at the start of each simulation run; hence, paired binary input values were represented by spike patterns over two groups of 50 neurons … Binary inputs were presented to the network episodically in a random order" teaches that the order of the binary encoded inputs (fractional inputs in the secured input layer) is randomized).
Farhady Ghalaty et al. is analogous to the claimed invention because it is directed towards securing neural networks.
Zhang et al., Liu et al., and Gardner et al. are analogous to the claimed invention because they are directed towards neural network input encoding.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the order of the inputs is randomized in the secured input layer as taught by Gardner et al. to the disclosed invention of Farhady Ghalaty et al. in view of Zhang et al. and further in view of Liu et al.
One of ordinary skill in the art would have been motivated to make this modification to "highlight the advantages of using a fully temporal code based on multiple output spike-timings to reliably encode for input patterns" (Gardner et al. Page 22, fourth paragraph).
Allowable Subject Matter
Claims 4 and 5 (when dependent on claim 4) would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIAN J HALES whose telephone number is (571)272-0878. The examiner can normally be reached M-F 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRIAN J HALES/Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights